blob: 074adbdf83a44205cf9378a953865df791d1e8fb [file] [log] [blame]
Caz Yokoyama8d497512013-09-05 16:42:39 -07001An Intel MIC X100 device is a PCIe form factor add-in coprocessor
2card based on the Intel Many Integrated Core (MIC) architecture
3that runs a Linux OS. It is a PCIe endpoint in a platform and therefore
4implements the three required standard address spaces i.e. configuration,
5memory and I/O. The host OS loads a device driver as is typical for
6PCIe devices. The card itself runs a bootstrap after reset that
Dasaratharaman Chandramouliaf190492013-10-03 18:06:23 -07007transfers control to the card OS downloaded from the host driver. The
8host driver supports OSPM suspend and resume operations. It shuts down
9the card during suspend and reboots the card OS during resume.
Caz Yokoyama8d497512013-09-05 16:42:39 -070010The card OS as shipped by Intel is a Linux kernel with modifications
11for the X100 devices.
12
13Since it is a PCIe card, it does not have the ability to host hardware
14devices for networking, storage and console. We provide these devices
Sudeep Duttc74c9312016-02-08 15:48:18 -080015on X100 coprocessors thus enabling a self-bootable equivalent
16environment for applications. A key benefit of our solution is that it
17leverages the standard virtio framework for network, disk and console
18devices, though in our case the virtio framework is used across a PCIe
19bus. A Virtio Over PCIe (VOP) driver allows creating user space
20backends or devices on the host which are used to probe virtio drivers
21for these devices on the MIC card. The existing VRINGH infrastructure
22in the kernel is used to access virtio rings from the host. The card
23VOP driver allows card virtio drivers to communicate with their user
24space backends on the host via a device page. Ring 3 apps on the host
25can add, remove and configure virtio devices. A thin MIC specific
26virtio_config_ops is implemented which is borrowed heavily from
27previous similar implementations in lguest and s390.
Caz Yokoyama8d497512013-09-05 16:42:39 -070028
Siva Yerramreddya8035842014-07-11 14:04:19 -070029MIC PCIe card has a dma controller with 8 channels. These channels are
30shared between the host s/w and the card s/w. 0 to 3 are used by host
31and 4 to 7 by card. As the dma device doesn't show up as PCIe device,
32a virtual bus called mic bus is created and virtual dma devices are
33created on it by the host/card drivers. On host the channels are private
34and used only by the host driver to transfer data for the virtio devices.
35
Sudeep Dutt7df20f22015-04-29 05:32:28 -070036The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a
37low level communications API across PCIe currently implemented for MIC.
38More details are available at scif_overview.txt.
39
Ashutosh Dixitf4a66c22015-09-29 18:14:30 -070040The Coprocessor State Management (COSM) driver on the host allows for
41boot, shutdown and reset of Intel MIC devices. It communicates with a COSM
42"client" driver on the MIC cards over SCIF to perform these functions.
43
Caz Yokoyama8d497512013-09-05 16:42:39 -070044Here is a block diagram of the various components described above. The
45virtio backends are situated on the host rather than the card given better
46single threaded performance for the host compared to MIC, the ability of
47the host to initiate DMA's to/from the card using the MIC DMA engine and
48the fact that the virtio block storage backend can only be on the host.
49
Siva Yerramreddya8035842014-07-11 14:04:19 -070050 +----------+ | +----------+
51 | Card OS | | | Host OS |
52 +----------+ | +----------+
53 |
54 +-------+ +--------+ +------+ | +---------+ +--------+ +--------+
55 | Virtio| |Virtio | |Virtio| | |Virtio | |Virtio | |Virtio |
56 | Net | |Console | |Block | | |Net | |Console | |Block |
57 | Driver| |Driver | |Driver| | |backend | |backend | |backend |
Sudeep Duttc74c9312016-02-08 15:48:18 -080058 +---+---+ +---+----+ +--+---+ | +---------+ +----+---+ +--------+
Siva Yerramreddya8035842014-07-11 14:04:19 -070059 | | | | | | |
60 | | | |User | | |
Sudeep Duttc74c9312016-02-08 15:48:18 -080061 | | | |------|------------|--+------|-------
62 +---------+---------+ |Kernel |
63 | | |
64 +---------+ +---+----+ +------+ | +------+ +------+ +--+---+ +-------+
65 |MIC DMA | | VOP | | SCIF | | | SCIF | | COSM | | VOP | |MIC DMA|
66 +---+-----+ +---+----+ +--+---+ | +--+---+ +--+---+ +------+ +----+--+
67 | | | | | | |
68 +---+-----+ +---+----+ +--+---+ | +--+---+ +--+---+ +------+ +----+--+
69 |MIC | | VOP | |SCIF | | |SCIF | | COSM | | VOP | | MIC |
70 |HW Bus | | HW Bus| |HW Bus| | |HW Bus| | Bus | |HW Bus| |HW Bus |
71 +---------+ +--------+ +--+---+ | +--+---+ +------+ +------+ +-------+
72 | | | | | | |
73 | +-----------+--+ | | | +---------------+ |
74 | |Intel MIC | | | | |Intel MIC | |
75 | |Card Driver | | | | |Host Driver | |
76 +---+--------------+------+ | +----+---------------+-----+
Ashutosh Dixitf4a66c22015-09-29 18:14:30 -070077 | | |
Siva Yerramreddya8035842014-07-11 14:04:19 -070078 +-------------------------------------------------------------+
79 | |
80 | PCIe Bus |
81 +-------------------------------------------------------------+