| Sudeep Dutt | 7df20f2 | 2015-04-29 05:32:28 -0700 | [diff] [blame] | 1 | The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low |
| 2 | level communications API across PCIe currently implemented for MIC. Currently |
| 3 | SCIF provides inter-node communication within a single host platform, where a |
| 4 | node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of |
| 5 | communicating over the PCIe bus while providing an API that is symmetric |
| 6 | across all the nodes in the PCIe network. An important design objective for SCIF |
| 7 | is to deliver the maximum possible performance given the communication |
| 8 | abilities of the hardware. SCIF has been used to implement an offload compiler |
| 9 | runtime and OFED support for MPI implementations for MIC coprocessors. |
| 10 | |
| 11 | ==== SCIF API Components ==== |
| 12 | The SCIF API has the following parts: |
| 13 | 1. Connection establishment using a client server model |
| 14 | 2. Byte stream messaging intended for short messages |
| 15 | 3. Node enumeration to determine online nodes |
| 16 | 4. Poll semantics for detection of incoming connections and messages |
| 17 | 5. Memory registration to pin down pages |
| 18 | 6. Remote memory mapping for low latency CPU accesses via mmap |
| 19 | 7. Remote DMA (RDMA) for high bandwidth DMA transfers |
| 20 | 8. Fence APIs for RDMA synchronization |
| 21 | |
| 22 | SCIF exposes the notion of a connection which can be used by peer processes on |
| 23 | nodes in a SCIF PCIe "network" to share memory "windows" and to communicate. A |
| 24 | process in a SCIF node initiates a SCIF connection to a peer process on a |
| 25 | different node via a SCIF "endpoint". SCIF endpoints support messaging APIs |
| 26 | which are similar to connection oriented socket APIs. Connected SCIF endpoints |
| 27 | can also register local memory which is followed by data transfer using either |
| 28 | DMA, CPU copies or remote memory mapping via mmap. SCIF supports both user and |
| 29 | kernel mode clients which are functionally equivalent. |
| 30 | |
| 31 | ==== SCIF Performance for MIC ==== |
| 32 | DMA bandwidth comparison between the TCP (over ethernet over PCIe) stack versus |
| 33 | SCIF shows the performance advantages of SCIF for HPC applications and runtimes. |
| 34 | |
| 35 | Comparison of TCP and SCIF based BW |
| 36 | |
| 37 | Throughput (GB/sec) |
| 38 | 8 + PCIe Bandwidth ****** |
| 39 | + TCP ###### |
| 40 | 7 + ************************************** SCIF %%%%%% |
| 41 | | %%%%%%%%%%%%%%%%%%% |
| 42 | 6 + %%%% |
| 43 | | %% |
| 44 | | %%% |
| 45 | 5 + %% |
| 46 | | %% |
| 47 | 4 + %% |
| 48 | | %% |
| 49 | 3 + %% |
| 50 | | % |
| 51 | 2 + %% |
| 52 | | %% |
| 53 | | % |
| 54 | 1 + |
| 55 | + ###################################### |
| 56 | 0 +++---+++--+--+-+--+--+-++-+--+-++-+--+-++-+- |
| 57 | 1 10 100 1000 10000 100000 |
| 58 | Transfer Size (KBytes) |
| 59 | |
| 60 | SCIF allows memory sharing via mmap(..) between processes on different PCIe |
| 61 | nodes and thus provides bare-metal PCIe latency. The round trip SCIF mmap |
| 62 | latency from the host to an x100 MIC for an 8 byte message is 0.44 usecs. |
| 63 | |
| 64 | SCIF has a user space library which is a thin IOCTL wrapper providing a user |
| 65 | space API similar to the kernel API in scif.h. The SCIF user space library |
| 66 | is distributed @ https://software.intel.com/en-us/mic-developer |
| 67 | |
| 68 | Here is some pseudo code for an example of how two applications on two PCIe |
| 69 | nodes would typically use the SCIF API: |
| 70 | |
| 71 | Process A (on node A) Process B (on node B) |
| 72 | |
| 73 | /* get online node information */ |
| 74 | scif_get_node_ids(..) scif_get_node_ids(..) |
| 75 | scif_open(..) scif_open(..) |
| 76 | scif_bind(..) scif_bind(..) |
| 77 | scif_listen(..) |
| 78 | scif_accept(..) scif_connect(..) |
| 79 | /* SCIF connection established */ |
| 80 | |
| 81 | /* Send and receive short messages */ |
| 82 | scif_send(..)/scif_recv(..) scif_send(..)/scif_recv(..) |
| 83 | |
| 84 | /* Register memory */ |
| 85 | scif_register(..) scif_register(..) |
| 86 | |
| 87 | /* RDMA */ |
| 88 | scif_readfrom(..)/scif_writeto(..) scif_readfrom(..)/scif_writeto(..) |
| 89 | |
| 90 | /* Fence DMAs */ |
| 91 | scif_fence_signal(..) scif_fence_signal(..) |
| 92 | |
| 93 | mmap(..) mmap(..) |
| 94 | |
| 95 | /* Access remote registered memory */ |
| 96 | |
| 97 | /* Close the endpoints */ |
| 98 | scif_close(..) scif_close(..) |