blob: b26122973525f81e691f2ef05e7069647d6b7e19 [file] [log] [blame]
Patrick McHardy56832642013-04-17 06:47:07 +00001This file documents how to use memory mapped I/O with netlink.
2
3Author: Patrick McHardy <kaber@trash.net>
4
5Overview
6--------
7
8Memory mapped netlink I/O can be used to increase throughput and decrease
9overhead of unicast receive and transmit operations. Some netlink subsystems
10require high throughput, these are mainly the netfilter subsystems
11nfnetlink_queue and nfnetlink_log, but it can also help speed up large
12dump operations of f.i. the routing database.
13
14Memory mapped netlink I/O used two circular ring buffers for RX and TX which
15are mapped into the processes address space.
16
17The RX ring is used by the kernel to directly construct netlink messages into
18user-space memory without copying them as done with regular socket I/O,
19additionally as long as the ring contains messages no recvmsg() or poll()
20syscalls have to be issued by user-space to get more message.
21
22The TX ring is used to process messages directly from user-space memory, the
23kernel processes all messages contained in the ring using a single sendmsg()
24call.
25
26Usage overview
27--------------
28
29In order to use memory mapped netlink I/O, user-space needs three main changes:
30
31- ring setup
32- conversion of the RX path to get messages from the ring instead of recvmsg()
33- conversion of the TX path to construct messages into the ring
34
35Ring setup is done using setsockopt() to provide the ring parameters to the
36kernel, then a call to mmap() to map the ring into the processes address space:
37
38- setsockopt(fd, SOL_NETLINK, NETLINK_RX_RING, &params, sizeof(params));
39- setsockopt(fd, SOL_NETLINK, NETLINK_TX_RING, &params, sizeof(params));
40- ring = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0)
41
42Usage of either ring is optional, but even if only the RX ring is used the
43mapping still needs to be writable in order to update the frame status after
44processing.
45
46Conversion of the reception path involves calling poll() on the file
47descriptor, once the socket is readable the frames from the ring are
Masanari Iidac17cb8b2013-10-30 16:46:15 +090048processed in order until no more messages are available, as indicated by
Patrick McHardy56832642013-04-17 06:47:07 +000049a status word in the frame header.
50
51On kernel side, in order to make use of memory mapped I/O on receive, the
52originating netlink subsystem needs to support memory mapped I/O, otherwise
53it will use an allocated socket buffer as usual and the contents will be
54 copied to the ring on transmission, nullifying most of the performance gains.
55Dumps of kernel databases automatically support memory mapped I/O.
56
Anatol Pomozovf884ab12013-05-08 16:56:16 -070057Conversion of the transmit path involves changing message construction to
Patrick McHardy56832642013-04-17 06:47:07 +000058use memory from the TX ring instead of (usually) a buffer declared on the
Masanari Iidac17cb8b2013-10-30 16:46:15 +090059stack and setting up the frame header appropriately. Optionally poll() can
Patrick McHardy56832642013-04-17 06:47:07 +000060be used to wait for free frames in the TX ring.
61
62Structured and definitions for using memory mapped I/O are contained in
63<linux/netlink.h>.
64
65RX and TX rings
66----------------
67
Anatol Pomozovf884ab12013-05-08 16:56:16 -070068Each ring contains a number of continuous memory blocks, containing frames of
69fixed size dependent on the parameters used for ring setup.
Patrick McHardy56832642013-04-17 06:47:07 +000070
71Ring: [ block 0 ]
72 [ frame 0 ]
73 [ frame 1 ]
74 [ block 1 ]
75 [ frame 2 ]
76 [ frame 3 ]
77 ...
78 [ block n ]
79 [ frame 2 * n ]
80 [ frame 2 * n + 1 ]
81
82The blocks are only visible to the kernel, from the point of view of user-space
Anatol Pomozovf884ab12013-05-08 16:56:16 -070083the ring just contains the frames in a continuous memory zone.
Patrick McHardy56832642013-04-17 06:47:07 +000084
85The ring parameters used for setting up the ring are defined as follows:
86
87struct nl_mmap_req {
88 unsigned int nm_block_size;
89 unsigned int nm_block_nr;
90 unsigned int nm_frame_size;
91 unsigned int nm_frame_nr;
92};
93
Anatol Pomozovf884ab12013-05-08 16:56:16 -070094Frames are grouped into blocks, where each block is a continuous region of memory
Patrick McHardy56832642013-04-17 06:47:07 +000095and holds nm_block_size / nm_frame_size frames. The total number of frames in
96the ring is nm_frame_nr. The following invariants hold:
97
98- frames_per_block = nm_block_size / nm_frame_size
99
100- nm_frame_nr = frames_per_block * nm_block_nr
101
102Some parameters are constrained, specifically:
103
104- nm_block_size must be a multiple of the architectures memory page size.
105 The getpagesize() function can be used to get the page size.
106
107- nm_frame_size must be equal or larger to NL_MMAP_HDRLEN, IOW a frame must be
108 able to hold at least the frame header
109
110- nm_frame_size must be smaller or equal to nm_block_size
111
112- nm_frame_size must be a multiple of NL_MMAP_MSG_ALIGNMENT
113
114- nm_frame_nr must equal the actual number of frames as specified above.
115
Anatol Pomozovf884ab12013-05-08 16:56:16 -0700116When the kernel can't allocate physically continuous memory for a ring block,
Masanari Iida3dd17ed2013-05-24 07:05:59 +0000117it will fall back to use physically discontinuous memory. This might affect
Patrick McHardy56832642013-04-17 06:47:07 +0000118performance negatively, in order to avoid this the nm_frame_size parameter
119should be chosen to be as small as possible for the required frame size and
120the number of blocks should be increased instead.
121
122Ring frames
123------------
124
125Each frames contain a frame header, consisting of a synchronization word and some
126meta-data, and the message itself.
127
128Frame: [ header message ]
129
130The frame header is defined as follows:
131
132struct nl_mmap_hdr {
133 unsigned int nm_status;
134 unsigned int nm_len;
135 __u32 nm_group;
136 /* credentials */
137 __u32 nm_pid;
138 __u32 nm_uid;
139 __u32 nm_gid;
140};
141
142- nm_status is used for synchronizing processing between the kernel and user-
143 space and specifies ownership of the frame as well as the operation to perform
144
145- nm_len contains the length of the message contained in the data area
146
147- nm_group specified the destination multicast group of message
148
149- nm_pid, nm_uid and nm_gid contain the netlink pid, UID and GID of the sending
150 process. These values correspond to the data available using SOCK_PASSCRED in
151 the SCM_CREDENTIALS cmsg.
152
153The possible values in the status word are:
154
155- NL_MMAP_STATUS_UNUSED:
156 RX ring: frame belongs to the kernel and contains no message
157 for user-space. Approriate action is to invoke poll()
158 to wait for new messages.
159
160 TX ring: frame belongs to user-space and can be used for
161 message construction.
162
163- NL_MMAP_STATUS_RESERVED:
164 RX ring only: frame is currently used by the kernel for message
165 construction and contains no valid message yet.
166 Appropriate action is to invoke poll() to wait for
167 new messages.
168
169- NL_MMAP_STATUS_VALID:
170 RX ring: frame contains a valid message. Approriate action is
171 to process the message and release the frame back to
172 the kernel by setting the status to
173 NL_MMAP_STATUS_UNUSED or queue the frame by setting the
174 status to NL_MMAP_STATUS_SKIP.
175
176 TX ring: the frame contains a valid message from user-space to
177 be processed by the kernel. After completing processing
178 the kernel will release the frame back to user-space by
179 setting the status to NL_MMAP_STATUS_UNUSED.
180
181- NL_MMAP_STATUS_COPY:
182 RX ring only: a message is ready to be processed but could not be
183 stored in the ring, either because it exceeded the
184 frame size or because the originating subsystem does
185 not support memory mapped I/O. Appropriate action is
186 to invoke recvmsg() to receive the message and release
187 the frame back to the kernel by setting the status to
188 NL_MMAP_STATUS_UNUSED.
189
190- NL_MMAP_STATUS_SKIP:
191 RX ring only: user-space queued the message for later processing, but
192 processed some messages following it in the ring. The
193 kernel should skip this frame when looking for unused
194 frames.
195
196The data area of a frame begins at a offset of NL_MMAP_HDRLEN relative to the
197frame header.
198
199TX limitations
200--------------
201
202Kernel processing usually involves validation of the message received by
203user-space, then processing its contents. The kernel must assure that
204userspace is not able to modify the message contents after they have been
205validated. In order to do so, the message is copied from the ring frame
206to an allocated buffer if either of these conditions is false:
207
208- only a single mapping of the ring exists
209- the file descriptor is not shared between processes
210
211This means that for threaded programs, the kernel will fall back to copying.
212
213Example
214-------
215
216Ring setup:
217
218 unsigned int block_size = 16 * getpagesize();
219 struct nl_mmap_req req = {
220 .nm_block_size = block_size,
221 .nm_block_nr = 64,
222 .nm_frame_size = 16384,
223 .nm_frame_nr = 64 * block_size / 16384,
224 };
225 unsigned int ring_size;
226 void *rx_ring, *tx_ring;
227
228 /* Configure ring parameters */
229 if (setsockopt(fd, NETLINK_RX_RING, &req, sizeof(req)) < 0)
230 exit(1);
231 if (setsockopt(fd, NETLINK_TX_RING, &req, sizeof(req)) < 0)
232 exit(1)
233
Masanari Iidac17cb8b2013-10-30 16:46:15 +0900234 /* Calculate size of each individual ring */
Patrick McHardy56832642013-04-17 06:47:07 +0000235 ring_size = req.nm_block_nr * req.nm_block_size;
236
237 /* Map RX/TX rings. The TX ring is located after the RX ring */
238 rx_ring = mmap(NULL, 2 * ring_size, PROT_READ | PROT_WRITE,
239 MAP_SHARED, fd, 0);
240 if ((long)rx_ring == -1L)
241 exit(1);
242 tx_ring = rx_ring + ring_size:
243
244Message reception:
245
246This example assumes some ring parameters of the ring setup are available.
247
248 unsigned int frame_offset = 0;
249 struct nl_mmap_hdr *hdr;
250 struct nlmsghdr *nlh;
251 unsigned char buf[16384];
252 ssize_t len;
253
254 while (1) {
255 struct pollfd pfds[1];
256
257 pfds[0].fd = fd;
258 pfds[0].events = POLLIN | POLLERR;
259 pfds[0].revents = 0;
260
261 if (poll(pfds, 1, -1) < 0 && errno != -EINTR)
262 exit(1);
263
264 /* Check for errors. Error handling omitted */
265 if (pfds[0].revents & POLLERR)
266 <handle error>
267
268 /* If no new messages, poll again */
269 if (!(pfds[0].revents & POLLIN))
270 continue;
271
272 /* Process all frames */
273 while (1) {
274 /* Get next frame header */
275 hdr = rx_ring + frame_offset;
276
Cong Wang76237572013-06-24 19:46:54 +0800277 if (hdr->nm_status == NL_MMAP_STATUS_VALID) {
Patrick McHardy56832642013-04-17 06:47:07 +0000278 /* Regular memory mapped frame */
Cong Wang76237572013-06-24 19:46:54 +0800279 nlh = (void *)hdr + NL_MMAP_HDRLEN;
Patrick McHardy56832642013-04-17 06:47:07 +0000280 len = hdr->nm_len;
281
282 /* Release empty message immediately. May happen
283 * on error during message construction.
284 */
285 if (len == 0)
286 goto release;
287 } else if (hdr->nm_status == NL_MMAP_STATUS_COPY) {
288 /* Frame queued to socket receive queue */
289 len = recv(fd, buf, sizeof(buf), MSG_DONTWAIT);
290 if (len <= 0)
291 break;
292 nlh = buf;
293 } else
294 /* No more messages to process, continue polling */
295 break;
296
297 process_msg(nlh);
298release:
299 /* Release frame back to the kernel */
300 hdr->nm_status = NL_MMAP_STATUS_UNUSED;
301
302 /* Advance frame offset to next frame */
303 frame_offset = (frame_offset + frame_size) % ring_size;
304 }
305 }
306
307Message transmission:
308
309This example assumes some ring parameters of the ring setup are available.
310A single message is constructed and transmitted, to send multiple messages
311at once they would be constructed in consecutive frames before a final call
312to sendto().
313
314 unsigned int frame_offset = 0;
315 struct nl_mmap_hdr *hdr;
316 struct nlmsghdr *nlh;
317 struct sockaddr_nl addr = {
318 .nl_family = AF_NETLINK,
319 };
320
321 hdr = tx_ring + frame_offset;
322 if (hdr->nm_status != NL_MMAP_STATUS_UNUSED)
323 /* No frame available. Use poll() to avoid. */
324 exit(1);
325
326 nlh = (void *)hdr + NL_MMAP_HDRLEN;
327
328 /* Build message */
329 build_message(nlh);
330
331 /* Fill frame header: length and status need to be set */
332 hdr->nm_len = nlh->nlmsg_len;
333 hdr->nm_status = NL_MMAP_STATUS_VALID;
334
335 if (sendto(fd, NULL, 0, 0, &addr, sizeof(addr)) < 0)
336 exit(1);
337
338 /* Advance frame offset to next frame */
339 frame_offset = (frame_offset + frame_size) % ring_size;