blob: 476df0496686d50b16fa73126cc4fdf52ac004f7 [file] [log] [blame]
Scott Feldman4ceec222015-05-10 09:48:09 -07001Ethernet switch device driver model (switchdev)
2===============================================
3Copyright (c) 2014 Jiri Pirko <jiri@resnulli.us>
4Copyright (c) 2014-2015 Scott Feldman <sfeldma@gmail.com>
Jiri Pirko007f7902014-11-28 14:34:17 +01005
Jiri Pirko007f7902014-11-28 14:34:17 +01006
Scott Feldman4ceec222015-05-10 09:48:09 -07007The Ethernet switch device driver model (switchdev) is an in-kernel driver
8model for switch devices which offload the forwarding (data) plane from the
9kernel.
Jiri Pirko007f7902014-11-28 14:34:17 +010010
Scott Feldman4ceec222015-05-10 09:48:09 -070011Figure 1 is a block diagram showing the components of the switchdev model for
12an example setup using a data-center-class switch ASIC chip. Other setups
13with SR-IOV or soft switches, such as OVS, are possible.
Jiri Pirko007f7902014-11-28 14:34:17 +010014
Jiri Pirko007f7902014-11-28 14:34:17 +010015
Scott Feldman4ceec222015-05-10 09:48:09 -070016                             User-space tools                                 
17                                                                              
18       user space                   |                                         
19      +-------------------------------------------------------------------+   
20       kernel                       | Netlink                                 
21                                    |                                         
22                     +--------------+-------------------------------+         
23                     |         Network stack                        |         
24                     |           (Linux)                            |         
25                     |                                              |         
26                     +----------------------------------------------+         
27                                                                              
28 sw1p2 sw1p4 sw1p6
29                      sw1p1  + sw1p3 +  sw1p5 +         eth1             
30                        +    |    +    |    +    |            +               
31                        |    |    |    |    |    |            |               
32                     +--+----+----+----+-+--+----+---+  +-----+-----+         
33                     |         Switch driver         |  |    mgmt   |         
34                     |        (this document)        |  |   driver  |         
35                     |                               |  |           |         
36                     +--------------+----------------+  +-----------+         
37                                    |                                         
38       kernel                       | HW bus (eg PCI)                         
39      +-------------------------------------------------------------------+   
40       hardware                     |                                         
41                     +--------------+---+------------+                        
42                     |         Switch device (sw1)   |                        
43                     |  +----+                       +--------+               
44                     |  |    v offloaded data path   | mgmt port              
45                     |  |    |                       |                        
46                     +--|----|----+----+----+----+---+                        
47                        |    |    |    |    |    |                            
48                        +    +    +    +    +    +                            
49                       p1   p2   p3   p4   p5   p6
50                                       
51                             front-panel ports                                
52                                                                              
Jiri Pirko007f7902014-11-28 14:34:17 +010053
Scott Feldman4ceec222015-05-10 09:48:09 -070054 Fig 1.
Jiri Pirko007f7902014-11-28 14:34:17 +010055
Jiri Pirko007f7902014-11-28 14:34:17 +010056
Scott Feldman4ceec222015-05-10 09:48:09 -070057Include Files
58-------------
Jiri Pirko007f7902014-11-28 14:34:17 +010059
Scott Feldman4ceec222015-05-10 09:48:09 -070060#include <linux/netdevice.h>
61#include <net/switchdev.h>
62
63
64Configuration
65-------------
66
67Use "depends NET_SWITCHDEV" in driver's Kconfig to ensure switchdev model
68support is built for driver.
69
70
71Switch Ports
72------------
73
74On switchdev driver initialization, the driver will allocate and register a
75struct net_device (using register_netdev()) for each enumerated physical switch
76port, called the port netdev. A port netdev is the software representation of
77the physical port and provides a conduit for control traffic to/from the
78controller (the kernel) and the network, as well as an anchor point for higher
79level constructs such as bridges, bonds, VLANs, tunnels, and L3 routers. Using
80standard netdev tools (iproute2, ethtool, etc), the port netdev can also
81provide to the user access to the physical properties of the switch port such
82as PHY link state and I/O statistics.
83
84There is (currently) no higher-level kernel object for the switch beyond the
85port netdevs. All of the switchdev driver ops are netdev ops or switchdev ops.
86
87A switch management port is outside the scope of the switchdev driver model.
88Typically, the management port is not participating in offloaded data plane and
89is loaded with a different driver, such as a NIC driver, on the management port
90device.
91
92Port Netdev Naming
93^^^^^^^^^^^^^^^^^^
94
95Udev rules should be used for port netdev naming, using some unique attribute
96of the port as a key, for example the port MAC address or the port PHYS name.
97Hard-coding of kernel netdev names within the driver is discouraged; let the
98kernel pick the default netdev name, and let udev set the final name based on a
99port attribute.
100
101Using port PHYS name (ndo_get_phys_port_name) for the key is particularly
Scott Feldman1f5dc442015-05-12 23:03:54 -0700102useful for dynamically-named ports where the device names its ports based on
Scott Feldman4ceec222015-05-10 09:48:09 -0700103external configuration. For example, if a physical 40G port is split logically
104into 4 10G ports, resulting in 4 port netdevs, the device can give a unique
105name for each port using port PHYS name. The udev rule would be:
106
107SUBSYSTEM=="net", ACTION=="add", DRIVER="<driver>", ATTR{phys_port_name}!="", \
108 NAME="$attr{phys_port_name}"
109
110Suggested naming convention is "swXpYsZ", where X is the switch name or ID, Y
111is the port name or ID, and Z is the sub-port name or ID. For example, sw1p1s0
112would be sub-port 0 on port 1 on switch 1.
113
114Switch ID
115^^^^^^^^^
116
Scott Feldmanf5ed2fe2015-06-03 20:43:40 -0700117The switchdev driver must implement the switchdev op switchdev_port_attr_get
118for SWITCHDEV_ATTR_PORT_PARENT_ID for each port netdev, returning the same
119physical ID for each port of a switch. The ID must be unique between switches
120on the same system. The ID does not need to be unique between switches on
121different systems.
Scott Feldman4ceec222015-05-10 09:48:09 -0700122
123The switch ID is used to locate ports on a switch and to know if aggregated
124ports belong to the same switch.
125
126Port Features
127^^^^^^^^^^^^^
128
129NETIF_F_NETNS_LOCAL
130
131If the switchdev driver (and device) only supports offloading of the default
132network namespace (netns), the driver should set this feature flag to prevent
133the port netdev from being moved out of the default netns. A netns-aware
Scott Feldman1f5dc442015-05-12 23:03:54 -0700134driver/device would not set this flag and be responsible for partitioning
Scott Feldman4ceec222015-05-10 09:48:09 -0700135hardware to preserve netns containment. This means hardware cannot forward
136traffic from a port in one namespace to another port in another namespace.
137
138Port Topology
139^^^^^^^^^^^^^
140
141The port netdevs representing the physical switch ports can be organized into
142higher-level switching constructs. The default construct is a standalone
143router port, used to offload L3 forwarding. Two or more ports can be bonded
144together to form a LAG. Two or more ports (or LAGs) can be bridged to bridge
Scott Feldmand290f1f2015-06-03 20:43:41 -0700145L2 networks. VLANs can be applied to sub-divide L2 networks. L2-over-L3
Scott Feldman4ceec222015-05-10 09:48:09 -0700146tunnels can be built on ports. These constructs are built using standard Linux
147tools such as the bridge driver, the bonding/team drivers, and netlink-based
148tools such as iproute2.
149
150The switchdev driver can know a particular port's position in the topology by
151monitoring NETDEV_CHANGEUPPER notifications. For example, a port moved into a
152bond will see it's upper master change. If that bond is moved into a bridge,
153the bond's upper master will change. And so on. The driver will track such
154movements to know what position a port is in in the overall topology by
155registering for netdevice events and acting on NETDEV_CHANGEUPPER.
156
157L2 Forwarding Offload
158---------------------
159
160The idea is to offload the L2 data forwarding (switching) path from the kernel
161to the switchdev device by mirroring bridge FDB entries down to the device. An
162FDB entry is the {port, MAC, VLAN} tuple forwarding destination.
163
164To offloading L2 bridging, the switchdev driver/device should support:
165
166 - Static FDB entries installed on a bridge port
167 - Notification of learned/forgotten src mac/vlans from device
168 - STP state changes on the port
169 - VLAN flooding of multicast/broadcast and unknown unicast packets
170
171Static FDB Entries
172^^^^^^^^^^^^^^^^^^
173
174The switchdev driver should implement ndo_fdb_add, ndo_fdb_del and ndo_fdb_dump
175to support static FDB entries installed to the device. Static bridge FDB
176entries are installed, for example, using iproute2 bridge cmd:
177
178 bridge fdb add ADDR dev DEV [vlan VID] [self]
179
Scott Feldman4b5364f2015-06-03 20:43:42 -0700180The driver should use the helper switchdev_port_fdb_xxx ops for ndo_fdb_xxx
181ops, and handle add/delete/dump of SWITCHDEV_OBJ_PORT_FDB object using
182switchdev_port_obj_xxx ops.
183
Scott Feldman1f5dc442015-05-12 23:03:54 -0700184XXX: what should be done if offloading this rule to hardware fails (for
185example, due to full capacity in hardware tables) ?
186
Scott Feldman4ceec222015-05-10 09:48:09 -0700187Note: by default, the bridge does not filter on VLAN and only bridges untagged
188traffic. To enable VLAN support, turn on VLAN filtering:
189
190 echo 1 >/sys/class/net/<bridge>/bridge/vlan_filtering
191
192Notification of Learned/Forgotten Source MAC/VLANs
193^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
194
195The switch device will learn/forget source MAC address/VLAN on ingress packets
196and notify the switch driver of the mac/vlan/port tuples. The switch driver,
197in turn, will notify the bridge driver using the switchdev notifier call:
198
199 err = call_switchdev_notifiers(val, dev, info);
200
Scott Feldmanf5ed2fe2015-06-03 20:43:40 -0700201Where val is SWITCHDEV_FDB_ADD when learning and SWITCHDEV_FDB_DEL when
202forgetting, and info points to a struct switchdev_notifier_fdb_info. On
203SWITCHDEV_FDB_ADD, the bridge driver will install the FDB entry into the
204bridge's FDB and mark the entry as NTF_EXT_LEARNED. The iproute2 bridge
205command will label these entries "offload":
Scott Feldman4ceec222015-05-10 09:48:09 -0700206
207 $ bridge fdb
208 52:54:00:12:35:01 dev sw1p1 master br0 permanent
209 00:02:00:00:02:00 dev sw1p1 master br0 offload
210 00:02:00:00:02:00 dev sw1p1 self
211 52:54:00:12:35:02 dev sw1p2 master br0 permanent
212 00:02:00:00:03:00 dev sw1p2 master br0 offload
213 00:02:00:00:03:00 dev sw1p2 self
214 33:33:00:00:00:01 dev eth0 self permanent
215 01:00:5e:00:00:01 dev eth0 self permanent
216 33:33:ff:00:00:00 dev eth0 self permanent
217 01:80:c2:00:00:0e dev eth0 self permanent
218 33:33:00:00:00:01 dev br0 self permanent
219 01:00:5e:00:00:01 dev br0 self permanent
220 33:33:ff:12:35:01 dev br0 self permanent
221
222Learning on the port should be disabled on the bridge using the bridge command:
223
224 bridge link set dev DEV learning off
225
226Learning on the device port should be enabled, as well as learning_sync:
227
228 bridge link set dev DEV learning on self
229 bridge link set dev DEV learning_sync on self
230
231Learning_sync attribute enables syncing of the learned/forgotton FDB entry to
232the bridge's FDB. It's possible, but not optimal, to enable learning on the
233device port and on the bridge port, and disable learning_sync.
234
235To support learning and learning_sync port attributes, the driver implements
Scott Feldmanf5ed2fe2015-06-03 20:43:40 -0700236switchdev op switchdev_port_attr_get/set for SWITCHDEV_ATTR_PORT_BRIDGE_FLAGS.
237The driver should initialize the attributes to the hardware defaults.
Scott Feldman4ceec222015-05-10 09:48:09 -0700238
239FDB Ageing
240^^^^^^^^^^
241
242There are two FDB ageing models supported: 1) ageing by the device, and 2)
243ageing by the kernel. Ageing by the device is preferred if many FDB entries
Scott Feldmanf5ed2fe2015-06-03 20:43:40 -0700244are supported. The driver calls call_switchdev_notifiers(SWITCHDEV_FDB_DEL,
245...) to age out the FDB entry. In this model, ageing by the kernel should be
246turned off. XXX: how to turn off ageing in kernel on a per-port basis or
247otherwise prevent the kernel from ageing out the FDB entry?
Scott Feldman4ceec222015-05-10 09:48:09 -0700248
249In the kernel ageing model, the standard bridge ageing mechanism is used to age
250out stale FDB entries. To keep an FDB entry "alive", the driver should refresh
251the FDB entry by calling call_switchdev_notifiers(SWITCHDEV_FDB_ADD, ...). The
252notification will reset the FDB entry's last-used time to now. The driver
253should rate limit refresh notifications, for example, no more than once a
Scott Feldmanb4ad7ba2015-06-14 11:33:11 -0700254second. If the FDB entry expires, fdb_delete is called to remove entry from
255the device.
Scott Feldman4ceec222015-05-10 09:48:09 -0700256
257STP State Change on Port
258^^^^^^^^^^^^^^^^^^^^^^^^
259
260Internally or with a third-party STP protocol implementation (e.g. mstpd), the
261bridge driver maintains the STP state for ports, and will notify the switch
Scott Feldmanf5ed2fe2015-06-03 20:43:40 -0700262driver of STP state change on a port using the switchdev op
263switchdev_attr_port_set for SWITCHDEV_ATTR_PORT_STP_UPDATE.
Scott Feldman4ceec222015-05-10 09:48:09 -0700264
265State is one of BR_STATE_*. The switch driver can use STP state updates to
266update ingress packet filter list for the port. For example, if port is
267DISABLED, no packets should pass, but if port moves to BLOCKED, then STP BPDUs
268and other IEEE 01:80:c2:xx:xx:xx link-local multicast packets can pass.
269
270Note that STP BDPUs are untagged and STP state applies to all VLANs on the port
271so packet filters should be applied consistently across untagged and tagged
272VLANs on the port.
273
274Flooding L2 domain
275^^^^^^^^^^^^^^^^^^
276
277For a given L2 VLAN domain, the switch device should flood multicast/broadcast
278and unknown unicast packets to all ports in domain, if allowed by port's
279current STP state. The switch driver, knowing which ports are within which
280vlan L2 domain, can program the switch device for flooding. The packet should
281also be sent to the port netdev for processing by the bridge driver. The
Scott Feldmana48037e2015-07-18 18:24:52 -0700282bridge should not reflood the packet to the same ports the device flooded,
283otherwise there will be duplicate packets on the wire.
284
285To avoid duplicate packets, the device/driver should mark a packet as already
286forwarded using skb->offload_fwd_mark. The same mark is set on the device
287ports in the domain using dev->offload_fwd_mark. If the skb->offload_fwd_mark
288is non-zero and matches the forwarding egress port's dev->skb_mark, the kernel
289will drop the skb right before transmit on the egress port, with the
290understanding that the device already forwarded the packet on same egress port.
291The driver can use switchdev_port_fwd_mark_set() to set a globally unique mark
292for port's dev->offload_fwd_mark, based on the port's parent ID (switch ID) and
293a group ifindex.
Scott Feldman4ceec222015-05-10 09:48:09 -0700294
295It is possible for the switch device to not handle flooding and push the
296packets up to the bridge driver for flooding. This is not ideal as the number
297of ports scale in the L2 domain as the device is much more efficient at
298flooding packets that software.
299
300IGMP Snooping
301^^^^^^^^^^^^^
302
303XXX: complete this section
304
305
Scott Feldman7616dcb2015-06-03 20:43:43 -0700306L3 Routing Offload
307------------------
Scott Feldman4ceec222015-05-10 09:48:09 -0700308
309Offloading L3 routing requires that device be programmed with FIB entries from
310the kernel, with the device doing the FIB lookup and forwarding. The device
311does a longest prefix match (LPM) on FIB entries matching route prefix and
Scott Feldman7616dcb2015-06-03 20:43:43 -0700312forwards the packet to the matching FIB entry's nexthop(s) egress ports.
Scott Feldman4ceec222015-05-10 09:48:09 -0700313
Scott Feldman7616dcb2015-06-03 20:43:43 -0700314To program the device, the driver implements support for
315SWITCHDEV_OBJ_IPV[4|6]_FIB object using switchdev_port_obj_xxx ops.
316switchdev_port_obj_add is used for both adding a new FIB entry to the device,
317or modifying an existing entry on the device.
Scott Feldman4ceec222015-05-10 09:48:09 -0700318
Scott Feldman7616dcb2015-06-03 20:43:43 -0700319XXX: Currently, only SWITCHDEV_OBJ_IPV4_FIB objects are supported.
Scott Feldman4ceec222015-05-10 09:48:09 -0700320
Scott Feldman7616dcb2015-06-03 20:43:43 -0700321SWITCHDEV_OBJ_IPV4_FIB object passes:
322
323 struct switchdev_obj_ipv4_fib { /* IPV4_FIB */
324 u32 dst;
325 int dst_len;
326 struct fib_info *fi;
327 u8 tos;
328 u8 type;
329 u32 nlflags;
330 u32 tb_id;
331 } ipv4_fib;
332
333to add/modify/delete IPv4 dst/dest_len prefix on table tb_id. The *fi
334structure holds details on the route and route's nexthops. *dev is one of the
335port netdevs mentioned in the routes next hop list. If the output port netdevs
336referenced in the route's nexthop list don't all have the same switch ID, the
337driver is not called to add/modify/delete the FIB entry.
Scott Feldman4ceec222015-05-10 09:48:09 -0700338
339Routes offloaded to the device are labeled with "offload" in the ip route
340listing:
341
342 $ ip route show
343 default via 192.168.0.2 dev eth0
344 11.0.0.0/30 dev sw1p1 proto kernel scope link src 11.0.0.2 offload
345 11.0.0.4/30 via 11.0.0.1 dev sw1p1 proto zebra metric 20 offload
346 11.0.0.8/30 dev sw1p2 proto kernel scope link src 11.0.0.10 offload
347 11.0.0.12/30 via 11.0.0.9 dev sw1p2 proto zebra metric 20 offload
348 12.0.0.2 proto zebra metric 30 offload
349 nexthop via 11.0.0.1 dev sw1p1 weight 1
350 nexthop via 11.0.0.9 dev sw1p2 weight 1
351 12.0.0.3 via 11.0.0.1 dev sw1p1 proto zebra metric 20 offload
352 12.0.0.4 via 11.0.0.9 dev sw1p2 proto zebra metric 20 offload
353 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.15
354
Scott Feldman7616dcb2015-06-03 20:43:43 -0700355XXX: add/mod/del IPv6 FIB API
Scott Feldman4ceec222015-05-10 09:48:09 -0700356
357Nexthop Resolution
358^^^^^^^^^^^^^^^^^^
359
360The FIB entry's nexthop list contains the nexthop tuple (gateway, dev), but for
361the switch device to forward the packet with the correct dst mac address, the
362nexthop gateways must be resolved to the neighbor's mac address. Neighbor mac
363address discovery comes via the ARP (or ND) process and is available via the
364arp_tbl neighbor table. To resolve the routes nexthop gateways, the driver
365should trigger the kernel's neighbor resolution process. See the rocker
366driver's rocker_port_ipv4_resolve() for an example.
367
368The driver can monitor for updates to arp_tbl using the netevent notifier
369NETEVENT_NEIGH_UPDATE. The device can be programmed with resolved nexthops
Scott Feldmandd19f832015-08-12 18:45:25 -0700370for the routes as arp_tbl updates. The driver implements ndo_neigh_destroy
371to know when arp_tbl neighbor entries are purged from the port.