blob: d226c7a5ba8bee46aeded6bfd181b840259fd4bb [file] [log] [blame]
Kirti Wankhede8e1c5a42016-11-17 02:16:31 +05301/*
2 * VFIO Mediated devices
3 *
4 * Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
5 * Author: Neo Jia <cjia@nvidia.com>
6 * Kirti Wankhede <kwankhede@nvidia.com>
7 *
8 * This program is free software; you can redistribute it and/or modify
9 * it under the terms of the GNU General Public License version 2 as
10 * published by the Free Software Foundation.
11 */
12
13Virtual Function I/O (VFIO) Mediated devices[1]
14===============================================
15
16The number of use cases for virtualizing DMA devices that do not have built-in
17SR_IOV capability is increasing. Previously, to virtualize such devices,
18developers had to create their own management interfaces and APIs, and then
19integrate them with user space software. To simplify integration with user space
20software, we have identified common requirements and a unified management
21interface for such devices.
22
23The VFIO driver framework provides unified APIs for direct device access. It is
24an IOMMU/device-agnostic framework for exposing direct device access to user
25space in a secure, IOMMU-protected environment. This framework is used for
26multiple devices, such as GPUs, network adapters, and compute accelerators. With
27direct device access, virtual machines or user space applications have direct
28access to the physical device. This framework is reused for mediated devices.
29
30The mediated core driver provides a common interface for mediated device
31management that can be used by drivers of different devices. This module
32provides a generic interface to perform these operations:
33
34* Create and destroy a mediated device
35* Add a mediated device to and remove it from a mediated bus driver
36* Add a mediated device to and remove it from an IOMMU group
37
38The mediated core driver also provides an interface to register a bus driver.
39For example, the mediated VFIO mdev driver is designed for mediated devices and
40supports VFIO APIs. The mediated bus driver adds a mediated device to and
41removes it from a VFIO group.
42
43The following high-level block diagram shows the main components and interfaces
44in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM
45devices as examples, as these devices are the first devices to use this module.
46
47 +---------------+
48 | |
49 | +-----------+ | mdev_register_driver() +--------------+
50 | | | +<------------------------+ |
51 | | mdev | | | |
52 | | bus | +------------------------>+ vfio_mdev.ko |<-> VFIO user
53 | | driver | | probe()/remove() | | APIs
54 | | | | +--------------+
55 | +-----------+ |
56 | |
57 | MDEV CORE |
58 | MODULE |
59 | mdev.ko |
60 | +-----------+ | mdev_register_device() +--------------+
61 | | | +<------------------------+ |
62 | | | | | nvidia.ko |<-> physical
63 | | | +------------------------>+ | device
64 | | | | callbacks +--------------+
65 | | Physical | |
66 | | device | | mdev_register_device() +--------------+
67 | | interface | |<------------------------+ |
68 | | | | | i915.ko |<-> physical
69 | | | +------------------------>+ | device
70 | | | | callbacks +--------------+
71 | | | |
72 | | | | mdev_register_device() +--------------+
73 | | | +<------------------------+ |
74 | | | | | ccw_device.ko|<-> physical
75 | | | +------------------------>+ | device
76 | | | | callbacks +--------------+
77 | +-----------+ |
78 +---------------+
79
80
81Registration Interfaces
82=======================
83
84The mediated core driver provides the following types of registration
85interfaces:
86
87* Registration interface for a mediated bus driver
88* Physical device driver interface
89
90Registration Interface for a Mediated Bus Driver
91------------------------------------------------
92
93The registration interface for a mediated bus driver provides the following
94structure to represent a mediated device's driver:
95
96 /*
97 * struct mdev_driver [2] - Mediated device's driver
98 * @name: driver name
99 * @probe: called when new device created
100 * @remove: called when device removed
101 * @driver: device driver structure
102 */
103 struct mdev_driver {
104 const char *name;
105 int (*probe) (struct device *dev);
106 void (*remove) (struct device *dev);
107 struct device_driver driver;
108 };
109
110A mediated bus driver for mdev should use this structure in the function calls
111to register and unregister itself with the core driver:
112
113* Register:
114
115 extern int mdev_register_driver(struct mdev_driver *drv,
116 struct module *owner);
117
118* Unregister:
119
120 extern void mdev_unregister_driver(struct mdev_driver *drv);
121
122The mediated bus driver is responsible for adding mediated devices to the VFIO
123group when devices are bound to the driver and removing mediated devices from
124the VFIO when devices are unbound from the driver.
125
126
127Physical Device Driver Interface
128--------------------------------
129
Alex Williamson42930552016-12-30 08:13:38 -0700130The physical device driver interface provides the mdev_parent_ops[3] structure
131to define the APIs to manage work in the mediated core driver that is related
132to the physical device.
Kirti Wankhede8e1c5a42016-11-17 02:16:31 +0530133
Alex Williamson42930552016-12-30 08:13:38 -0700134The structures in the mdev_parent_ops structure are as follows:
Kirti Wankhede8e1c5a42016-11-17 02:16:31 +0530135
136* dev_attr_groups: attributes of the parent device
137* mdev_attr_groups: attributes of the mediated device
138* supported_config: attributes to define supported configurations
139
Alex Williamson42930552016-12-30 08:13:38 -0700140The functions in the mdev_parent_ops structure are as follows:
Kirti Wankhede8e1c5a42016-11-17 02:16:31 +0530141
142* create: allocate basic resources in a driver for a mediated device
143* remove: free resources in a driver when a mediated device is destroyed
144
Alex Williamson42930552016-12-30 08:13:38 -0700145The callbacks in the mdev_parent_ops structure are as follows:
Kirti Wankhede8e1c5a42016-11-17 02:16:31 +0530146
147* open: open callback of mediated device
148* close: close callback of mediated device
149* ioctl: ioctl callback of mediated device
150* read : read emulation callback
151* write: write emulation callback
152* mmap: mmap emulation callback
153
Alex Williamson42930552016-12-30 08:13:38 -0700154A driver should use the mdev_parent_ops structure in the function call to
155register itself with the mdev core driver:
Kirti Wankhede8e1c5a42016-11-17 02:16:31 +0530156
157extern int mdev_register_device(struct device *dev,
Alex Williamson42930552016-12-30 08:13:38 -0700158 const struct mdev_parent_ops *ops);
Kirti Wankhede8e1c5a42016-11-17 02:16:31 +0530159
Alex Williamson42930552016-12-30 08:13:38 -0700160However, the mdev_parent_ops structure is not required in the function call
161that a driver should use to unregister itself with the mdev core driver:
Kirti Wankhede8e1c5a42016-11-17 02:16:31 +0530162
163extern void mdev_unregister_device(struct device *dev);
164
165
166Mediated Device Management Interface Through sysfs
167==================================================
168
169The management interface through sysfs enables user space software, such as
170libvirt, to query and configure mediated devices in a hardware-agnostic fashion.
171This management interface provides flexibility to the underlying physical
172device's driver to support features such as:
173
174* Mediated device hot plug
175* Multiple mediated devices in a single virtual machine
176* Multiple mediated devices from different physical devices
177
178Links in the mdev_bus Class Directory
179-------------------------------------
180The /sys/class/mdev_bus/ directory contains links to devices that are registered
181with the mdev core driver.
182
183Directories and files under the sysfs for Each Physical Device
184--------------------------------------------------------------
185
186|- [parent physical device]
187|--- Vendor-specific-attributes [optional]
188|--- [mdev_supported_types]
189| |--- [<type-id>]
190| | |--- create
191| | |--- name
192| | |--- available_instances
193| | |--- device_api
194| | |--- description
195| | |--- [devices]
196| |--- [<type-id>]
197| | |--- create
198| | |--- name
199| | |--- available_instances
200| | |--- device_api
201| | |--- description
202| | |--- [devices]
203| |--- [<type-id>]
204| |--- create
205| |--- name
206| |--- available_instances
207| |--- device_api
208| |--- description
209| |--- [devices]
210
211* [mdev_supported_types]
212
213 The list of currently supported mediated device types and their details.
214
215 [<type-id>], device_api, and available_instances are mandatory attributes
216 that should be provided by vendor driver.
217
218* [<type-id>]
219
220 The [<type-id>] name is created by adding the the device driver string as a
221 prefix to the string provided by the vendor driver. This format of this name
222 is as follows:
223
224 sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name);
225
Alex Williamson9372e6fe2016-12-30 08:13:41 -0700226 (or using mdev_parent_dev(mdev) to arrive at the parent device outside
227 of the core mdev code)
228
Kirti Wankhede8e1c5a42016-11-17 02:16:31 +0530229* device_api
230
231 This attribute should show which device API is being created, for example,
232 "vfio-pci" for a PCI device.
233
234* available_instances
235
236 This attribute should show the number of devices of type <type-id> that can be
237 created.
238
239* [device]
240
241 This directory contains links to the devices of type <type-id> that have been
242created.
243
244* name
245
246 This attribute should show human readable name. This is optional attribute.
247
248* description
249
250 This attribute should show brief features/description of the type. This is
251 optional attribute.
252
253Directories and Files Under the sysfs for Each mdev Device
254----------------------------------------------------------
255
256|- [parent phy device]
257|--- [$MDEV_UUID]
258 |--- remove
259 |--- mdev_type {link to its type}
260 |--- vendor-specific-attributes [optional]
261
262* remove (write only)
263Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can
264fail the remove() callback if that device is active and the vendor driver
265doesn't support hot unplug.
266
267Example:
268 # echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove
269
270Mediated device Hot plug:
271------------------------
272
273Mediated devices can be created and assigned at runtime. The procedure to hot
274plug a mediated device is the same as the procedure to hot plug a PCI device.
275
276Translation APIs for Mediated Devices
277=====================================
278
279The following APIs are provided for translating user pfn to host pfn in a VFIO
280driver:
281
282extern int vfio_pin_pages(struct device *dev, unsigned long *user_pfn,
283 int npage, int prot, unsigned long *phys_pfn);
284
285extern int vfio_unpin_pages(struct device *dev, unsigned long *user_pfn,
286 int npage);
287
288These functions call back into the back-end IOMMU module by using the pin_pages
289and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently
290these callbacks are supported in the TYPE1 IOMMU module. To enable them for
291other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide
292these two callback functions.
293
Kirti Wankhede9d1a5462016-11-17 02:16:33 +0530294Using the Sample Code
295=====================
296
297mtty.c in samples/vfio-mdev/ directory is a sample driver program to
298demonstrate how to use the mediated device framework.
299
300The sample driver creates an mdev device that simulates a serial port over a PCI
301card.
302
3031. Build and load the mtty.ko module.
304
305 This step creates a dummy device, /sys/devices/virtual/mtty/mtty/
306
307 Files in this device directory in sysfs are similar to the following:
308
309 # tree /sys/devices/virtual/mtty/mtty/
310 /sys/devices/virtual/mtty/mtty/
311 |-- mdev_supported_types
312 | |-- mtty-1
313 | | |-- available_instances
314 | | |-- create
315 | | |-- device_api
316 | | |-- devices
317 | | `-- name
318 | `-- mtty-2
319 | |-- available_instances
320 | |-- create
321 | |-- device_api
322 | |-- devices
323 | `-- name
324 |-- mtty_dev
325 | `-- sample_mtty_dev
326 |-- power
327 | |-- autosuspend_delay_ms
328 | |-- control
329 | |-- runtime_active_time
330 | |-- runtime_status
331 | `-- runtime_suspended_time
332 |-- subsystem -> ../../../../class/mtty
333 `-- uevent
334
3352. Create a mediated device by using the dummy device that you created in the
336 previous step.
337
338 # echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" > \
339 /sys/devices/virtual/mtty/mtty/mdev_supported_types/mtty-2/create
340
3413. Add parameters to qemu-kvm.
342
343 -device vfio-pci,\
344 sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001
345
3464. Boot the VM.
347
348 In the Linux guest VM, with no hardware on the host, the device appears
349 as follows:
350
351 # lspci -s 00:05.0 -xxvv
352 00:05.0 Serial controller: Device 4348:3253 (rev 10) (prog-if 02 [16550])
353 Subsystem: Device 4348:3253
354 Physical Slot: 5
355 Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
356 Stepping- SERR- FastB2B- DisINTx-
357 Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
358 <TAbort- <MAbort- >SERR- <PERR- INTx-
359 Interrupt: pin A routed to IRQ 10
360 Region 0: I/O ports at c150 [size=8]
361 Region 1: I/O ports at c158 [size=8]
362 Kernel driver in use: serial
363 00: 48 43 53 32 01 00 00 02 10 02 00 07 00 00 00 00
364 10: 51 c1 00 00 59 c1 00 00 00 00 00 00 00 00 00 00
365 20: 00 00 00 00 00 00 00 00 00 00 00 00 48 43 53 32
366 30: 00 00 00 00 00 00 00 00 00 00 00 00 0a 01 00 00
367
368 In the Linux guest VM, dmesg output for the device is as follows:
369
370 serial 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ
37110
372 0000:00:05.0: ttyS1 at I/O 0xc150 (irq = 10) is a 16550A
373 0000:00:05.0: ttyS2 at I/O 0xc158 (irq = 10) is a 16550A
374
375
3765. In the Linux guest VM, check the serial ports.
377
378 # setserial -g /dev/ttyS*
379 /dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4
380 /dev/ttyS1, UART: 16550A, Port: 0xc150, IRQ: 10
381 /dev/ttyS2, UART: 16550A, Port: 0xc158, IRQ: 10
382
3836. Using a minicom or any terminal enulation program, open port /dev/ttyS1 or
384 /dev/ttyS2 with hardware flow control disabled.
385
3867. Type data on the minicom terminal or send data to the terminal emulation
387 program and read the data.
388
389 Data is loop backed from hosts mtty driver.
390
3918. Destroy the mediated device that you created.
392
393 # echo 1 > /sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001/remove
394
Kirti Wankhede8e1c5a42016-11-17 02:16:31 +0530395References
Kirti Wankhede9d1a5462016-11-17 02:16:33 +0530396==========
Kirti Wankhede8e1c5a42016-11-17 02:16:31 +0530397
398[1] See Documentation/vfio.txt for more information on VFIO.
399[2] struct mdev_driver in include/linux/mdev.h
Alex Williamson42930552016-12-30 08:13:38 -0700400[3] struct mdev_parent_ops in include/linux/mdev.h
Kirti Wankhede8e1c5a42016-11-17 02:16:31 +0530401[4] struct vfio_iommu_driver_ops in include/linux/vfio.h