This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in Web of Science Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Santhanaraman, G. Articles by Panda, D. K. Search for Related Content Social Bookmarking                         What's this?

Designing Zero-Copy Message Passing Interface Derived Datatype Communication Over Infiniband: Alternative Approaches and Performance Evaluation

Network-Based Computing Laboratory, Department of Computer Science and Engineering, The Ohio State University, Columbus, OH 43210, USA, SANTHANA{at}CSE.OHIO-STATE.EDU

Jiesheng Wu

Network-Based Computing Laboratory, Department of Computer Science and Engineering, The Ohio State University, Columbus, OH 43210, USA, WUJ{at}CSE.OHIO-STATE.EDU

Wei Huang

Network-Based Computing Laboratory, Department of Computer Science and Engineering, The Ohio State University, Columbus, OH 43210, USA, HUANWEI{at}CSE.OHIO-STATE.EDU

Dhabaleswar K. Panda

Network-Based Computing Laboratory, Department of Computer Science and Engineering, The Ohio State University, Columbus, OH 43210, USA, PANDA{at}CSE.OHIO-STATE.EDU

In this paper, we present a new scheme, Send Gather Receive Scatter (SGRS), to perform zero-copy datatype communication over InfiniBand. This scheme leverages the gather/scatter feature provided by InfiniBand channel semantics. It takes advantage of the capability of processing non-contiguity on both send and receive sides in the Send Gather and Receive Scatter operations. We have implemented this new design and evaluated the performance for Message Passing Interface level point-to-point microbenchmarks and collectives, on PCI-X and upcoming high performance PCI-Express systems. In our previous work we had come up with an alternate zero-copy approach using multiple RDMA Writes (Multi-W). Compared to the existing Multi-W zero-copy datatype scheme, the SGRS scheme can overcome the drawbacks of low network utilization and high startup cost. On PCI-X platforms, our experimental results show significant improvement in both point-to-point and collective datatype communication. The latency of a vector datatype can be reduced by up to 62% and the bandwidth shows improvement up to 400% as compared with the Multi-W scheme. The Alltoall collective shows up to 23% reduction in latency. Further, the SGRS scheme shows low CPU overhead with a potential promise for better computation and communication overlap. The experimental results on PCI-Express platforms demonstrate the relevance of zero-copy protocols to overcome memory bandwidth limitations. The trends we observe in PCI-X platform are magnified on PCI-Express platforms with even higher improvement for the microbenchmarks and collectives.

Key Words: MPI • Datatype • zero-copy • InfiniBand

International Journal of High Performance Computing Applications, Vol. 19, No. 2, 129-142 (2005)
DOI: 10.1177/1094342005054259


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?