blob: 5f0922613f1a8db7963fe3c44d21280d7290d84e [file] [log] [blame]
Willem de Bruijn8fe2f762014-08-31 21:27:47 -04001
21. Control Interfaces
3
4The interfaces for receiving network packages timestamps are:
Patrick Ohlycb9eff02009-02-12 05:03:36 +00005
6* SO_TIMESTAMP
Willem de Bruijn8fe2f762014-08-31 21:27:47 -04007 Generates a timestamp for each incoming packet in (not necessarily
8 monotonic) system time. Reports the timestamp via recvmsg() in a
9 control message as struct timeval (usec resolution).
Patrick Ohlycb9eff02009-02-12 05:03:36 +000010
11* SO_TIMESTAMPNS
Willem de Bruijn8fe2f762014-08-31 21:27:47 -040012 Same timestamping mechanism as SO_TIMESTAMP, but reports the
13 timestamp as struct timespec (nsec resolution).
Patrick Ohlycb9eff02009-02-12 05:03:36 +000014
15* IP_MULTICAST_LOOP + SO_TIMESTAMP[NS]
Willem de Bruijn8fe2f762014-08-31 21:27:47 -040016 Only for multicast:approximate transmit timestamp obtained by
17 reading the looped packet receive timestamp.
Patrick Ohlycb9eff02009-02-12 05:03:36 +000018
Willem de Bruijn8fe2f762014-08-31 21:27:47 -040019* SO_TIMESTAMPING
20 Generates timestamps on reception, transmission or both. Supports
21 multiple timestamp sources, including hardware. Supports generating
22 timestamps for stream sockets.
Patrick Ohlycb9eff02009-02-12 05:03:36 +000023
Patrick Ohlycb9eff02009-02-12 05:03:36 +000024
Willem de Bruijn8fe2f762014-08-31 21:27:47 -0400251.1 SO_TIMESTAMP:
Patrick Ohlycb9eff02009-02-12 05:03:36 +000026
Willem de Bruijn8fe2f762014-08-31 21:27:47 -040027This socket option enables timestamping of datagrams on the reception
28path. Because the destination socket, if any, is not known early in
29the network stack, the feature has to be enabled for all packets. The
30same is true for all early receive timestamp options.
Patrick Ohlycb9eff02009-02-12 05:03:36 +000031
Willem de Bruijn8fe2f762014-08-31 21:27:47 -040032For interface details, see `man 7 socket`.
33
34
351.2 SO_TIMESTAMPNS:
36
37This option is identical to SO_TIMESTAMP except for the returned data type.
38Its struct timespec allows for higher resolution (ns) timestamps than the
39timeval of SO_TIMESTAMP (ms).
40
41
421.3 SO_TIMESTAMPING:
43
44Supports multiple types of timestamp requests. As a result, this
45socket option takes a bitmap of flags, not a boolean. In
46
47 err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, (void *) val, &val);
48
49val is an integer with any of the following bits set. Setting other
50bit returns EINVAL and does not change the current state.
51
52
531.3.1 Timestamp Generation
54
55Some bits are requests to the stack to try to generate timestamps. Any
56combination of them is valid. Changes to these bits apply to newly
57created packets, not to packets already in the stack. As a result, it
58is possible to selectively request timestamps for a subset of packets
59(e.g., for sampling) by embedding an send() call within two setsockopt
60calls, one to enable timestamp generation and one to disable it.
61Timestamps may also be generated for reasons other than being
62requested by a particular socket, such as when receive timestamping is
63enabled system wide, as explained earlier.
64
65SOF_TIMESTAMPING_RX_HARDWARE:
66 Request rx timestamps generated by the network adapter.
67
68SOF_TIMESTAMPING_RX_SOFTWARE:
69 Request rx timestamps when data enters the kernel. These timestamps
70 are generated just after a device driver hands a packet to the
71 kernel receive stack.
72
73SOF_TIMESTAMPING_TX_HARDWARE:
74 Request tx timestamps generated by the network adapter.
75
76SOF_TIMESTAMPING_TX_SOFTWARE:
77 Request tx timestamps when data leaves the kernel. These timestamps
78 are generated in the device driver as close as possible, but always
79 prior to, passing the packet to the network interface. Hence, they
80 require driver support and may not be available for all devices.
81
82SOF_TIMESTAMPING_TX_SCHED:
83 Request tx timestamps prior to entering the packet scheduler. Kernel
84 transmit latency is, if long, often dominated by queuing delay. The
85 difference between this timestamp and one taken at
86 SOF_TIMESTAMPING_TX_SOFTWARE will expose this latency independent
87 of protocol processing. The latency incurred in protocol
88 processing, if any, can be computed by subtracting a userspace
89 timestamp taken immediately before send() from this timestamp. On
90 machines with virtual devices where a transmitted packet travels
91 through multiple devices and, hence, multiple packet schedulers,
92 a timestamp is generated at each layer. This allows for fine
93 grained measurement of queuing delay.
94
95SOF_TIMESTAMPING_TX_ACK:
96 Request tx timestamps when all data in the send buffer has been
97 acknowledged. This only makes sense for reliable protocols. It is
98 currently only implemented for TCP. For that protocol, it may
99 over-report measurement, because the timestamp is generated when all
100 data up to and including the buffer at send() was acknowledged: the
101 cumulative acknowledgment. The mechanism ignores SACK and FACK.
102
103
1041.3.2 Timestamp Reporting
Andrew Lutomirskiadca4762014-03-04 17:24:10 -0800105
106The other three bits control which timestamps will be reported in a
Willem de Bruijn8fe2f762014-08-31 21:27:47 -0400107generated control message. Changes to the bits take immediate
108effect at the timestamp reporting locations in the stack. Timestamps
109are only reported for packets that also have the relevant timestamp
110generation request set.
Andrew Lutomirskiadca4762014-03-04 17:24:10 -0800111
Willem de Bruijn8fe2f762014-08-31 21:27:47 -0400112SOF_TIMESTAMPING_SOFTWARE:
113 Report any software timestamps when available.
Andrew Lutomirskiadca4762014-03-04 17:24:10 -0800114
Willem de Bruijn8fe2f762014-08-31 21:27:47 -0400115SOF_TIMESTAMPING_SYS_HARDWARE:
116 This option is deprecated and ignored.
Andrew Lutomirskiadca4762014-03-04 17:24:10 -0800117
Willem de Bruijn8fe2f762014-08-31 21:27:47 -0400118SOF_TIMESTAMPING_RAW_HARDWARE:
119 Report hardware timestamps as generated by
120 SOF_TIMESTAMPING_TX_HARDWARE when available.
121
122
1231.3.3 Timestamp Options
124
Willem de Bruijn829ae9d2014-11-30 22:22:34 -0500125The interface supports the options
Willem de Bruijn8fe2f762014-08-31 21:27:47 -0400126
127SOF_TIMESTAMPING_OPT_ID:
128
129 Generate a unique identifier along with each packet. A process can
130 have multiple concurrent timestamping requests outstanding. Packets
131 can be reordered in the transmit path, for instance in the packet
132 scheduler. In that case timestamps will be queued onto the error
Willem de Bruijncbd3aad2014-11-30 22:22:35 -0500133 queue out of order from the original send() calls. It is not always
134 possible to uniquely match timestamps to the original send() calls
135 based on timestamp order or payload inspection alone, then.
136
137 This option associates each packet at send() with a unique
138 identifier and returns that along with the timestamp. The identifier
139 is derived from a per-socket u32 counter (that wraps). For datagram
140 sockets, the counter increments with each sent packet. For stream
141 sockets, it increments with every byte.
142
143 The counter starts at zero. It is initialized the first time that
144 the socket option is enabled. It is reset each time the option is
145 enabled after having been disabled. Resetting the counter does not
146 change the identifiers of existing packets in the system.
Willem de Bruijn8fe2f762014-08-31 21:27:47 -0400147
148 This option is implemented only for transmit timestamps. There, the
149 timestamp is always looped along with a struct sock_extended_err.
Andrew Lutomirski138a7f42014-11-24 12:02:29 -0800150 The option modifies field ee_data to pass an id that is unique
Willem de Bruijn8fe2f762014-08-31 21:27:47 -0400151 among all possibly concurrently outstanding timestamp requests for
Willem de Bruijncbd3aad2014-11-30 22:22:35 -0500152 that socket.
Willem de Bruijn8fe2f762014-08-31 21:27:47 -0400153
154
Willem de Bruijn829ae9d2014-11-30 22:22:34 -0500155SOF_TIMESTAMPING_OPT_CMSG:
156
157 Support recv() cmsg for all timestamped packets. Control messages
158 are already supported unconditionally on all packets with receive
159 timestamps and on IPv6 packets with transmit timestamp. This option
160 extends them to IPv4 packets with transmit timestamp. One use case
161 is to correlate packets with their egress device, by enabling socket
162 option IP_PKTINFO simultaneously.
163
164
Willem de Bruijn49ca0d82015-01-30 13:29:31 -0500165SOF_TIMESTAMPING_OPT_TSONLY:
166
167 Applies to transmit timestamps only. Makes the kernel return the
168 timestamp as a cmsg alongside an empty packet, as opposed to
169 alongside the original packet. This reduces the amount of memory
170 charged to the socket's receive budget (SO_RCVBUF) and delivers
171 the timestamp even if sysctl net.core.tstamp_allow_data is 0.
172 This option disables SOF_TIMESTAMPING_OPT_CMSG.
173
174
175New applications are encouraged to pass SOF_TIMESTAMPING_OPT_ID to
176disambiguate timestamps and SOF_TIMESTAMPING_OPT_TSONLY to operate
177regardless of the setting of sysctl net.core.tstamp_allow_data.
178
179An exception is when a process needs additional cmsg data, for
180instance SOL_IP/IP_PKTINFO to detect the egress network interface.
181Then pass option SOF_TIMESTAMPING_OPT_CMSG. This option depends on
182having access to the contents of the original packet, so cannot be
183combined with SOF_TIMESTAMPING_OPT_TSONLY.
184
185
Willem de Bruijn8fe2f762014-08-31 21:27:47 -04001861.4 Bytestream Timestamps
187
188The SO_TIMESTAMPING interface supports timestamping of bytes in a
189bytestream. Each request is interpreted as a request for when the
190entire contents of the buffer has passed a timestamping point. That
191is, for streams option SOF_TIMESTAMPING_TX_SOFTWARE will record
192when all bytes have reached the device driver, regardless of how
193many packets the data has been converted into.
194
195In general, bytestreams have no natural delimiters and therefore
196correlating a timestamp with data is non-trivial. A range of bytes
197may be split across segments, any segments may be merged (possibly
198coalescing sections of previously segmented buffers associated with
199independent send() calls). Segments can be reordered and the same
200byte range can coexist in multiple segments for protocols that
201implement retransmissions.
202
203It is essential that all timestamps implement the same semantics,
204regardless of these possible transformations, as otherwise they are
205incomparable. Handling "rare" corner cases differently from the
206simple case (a 1:1 mapping from buffer to skb) is insufficient
207because performance debugging often needs to focus on such outliers.
208
209In practice, timestamps can be correlated with segments of a
210bytestream consistently, if both semantics of the timestamp and the
211timing of measurement are chosen correctly. This challenge is no
212different from deciding on a strategy for IP fragmentation. There, the
213definition is that only the first fragment is timestamped. For
214bytestreams, we chose that a timestamp is generated only when all
215bytes have passed a point. SOF_TIMESTAMPING_TX_ACK as defined is easy to
216implement and reason about. An implementation that has to take into
217account SACK would be more complex due to possible transmission holes
218and out of order arrival.
219
220On the host, TCP can also break the simple 1:1 mapping from buffer to
221skbuff as a result of Nagle, cork, autocork, segmentation and GSO. The
222implementation ensures correctness in all cases by tracking the
223individual last byte passed to send(), even if it is no longer the
224last byte after an skbuff extend or merge operation. It stores the
225relevant sequence number in skb_shinfo(skb)->tskey. Because an skbuff
226has only one such field, only one timestamp can be generated.
227
228In rare cases, a timestamp request can be missed if two requests are
229collapsed onto the same skb. A process can detect this situation by
230enabling SOF_TIMESTAMPING_OPT_ID and comparing the byte offset at
231send time with the value returned for each timestamp. It can prevent
232the situation by always flushing the TCP stack in between requests,
233for instance by enabling TCP_NODELAY and disabling TCP_CORK and
234autocork.
235
236These precautions ensure that the timestamp is generated only when all
237bytes have passed a timestamp point, assuming that the network stack
238itself does not reorder the segments. The stack indeed tries to avoid
239reordering. The one exception is under administrator control: it is
240possible to construct a packet scheduler configuration that delays
241segments from the same stream differently. Such a setup would be
242unusual.
243
244
2452 Data Interfaces
246
247Timestamps are read using the ancillary data feature of recvmsg().
248See `man 3 cmsg` for details of this interface. The socket manual
249page (`man 7 socket`) describes how timestamps generated with
250SO_TIMESTAMP and SO_TIMESTAMPNS records can be retrieved.
251
252
2532.1 SCM_TIMESTAMPING records
254
255These timestamps are returned in a control message with cmsg_level
256SOL_SOCKET, cmsg_type SCM_TIMESTAMPING, and payload of type
Patrick Loschmidt69298692010-04-07 21:52:07 -0700257
258struct scm_timestamping {
Willem de Bruijn8fe2f762014-08-31 21:27:47 -0400259 struct timespec ts[3];
Patrick Loschmidt69298692010-04-07 21:52:07 -0700260};
Patrick Ohlycb9eff02009-02-12 05:03:36 +0000261
Willem de Bruijn8fe2f762014-08-31 21:27:47 -0400262The structure can return up to three timestamps. This is a legacy
263feature. Only one field is non-zero at any time. Most timestamps
264are passed in ts[0]. Hardware timestamps are passed in ts[2].
Patrick Ohlycb9eff02009-02-12 05:03:36 +0000265
Willem de Bruijn8fe2f762014-08-31 21:27:47 -0400266ts[1] used to hold hardware timestamps converted to system time.
267Instead, expose the hardware clock device on the NIC directly as
268a HW PTP clock source, to allow time conversion in userspace and
269optionally synchronize system time with a userspace PTP stack such
270as linuxptp. For the PTP clock API, see Documentation/ptp/ptp.txt.
Patrick Ohlycb9eff02009-02-12 05:03:36 +0000271
Willem de Bruijn8fe2f762014-08-31 21:27:47 -04002722.1.1 Transmit timestamps with MSG_ERRQUEUE
Patrick Ohlycb9eff02009-02-12 05:03:36 +0000273
Willem de Bruijn8fe2f762014-08-31 21:27:47 -0400274For transmit timestamps the outgoing packet is looped back to the
275socket's error queue with the send timestamp(s) attached. A process
276receives the timestamps by calling recvmsg() with flag MSG_ERRQUEUE
277set and with a msg_control buffer sufficiently large to receive the
278relevant metadata structures. The recvmsg call returns the original
279outgoing data packet with two ancillary messages attached.
Patrick Ohlycb9eff02009-02-12 05:03:36 +0000280
Willem de Bruijn8fe2f762014-08-31 21:27:47 -0400281A message of cm_level SOL_IP(V6) and cm_type IP(V6)_RECVERR
282embeds a struct sock_extended_err. This defines the error type. For
283timestamps, the ee_errno field is ENOMSG. The other ancillary message
284will have cm_level SOL_SOCKET and cm_type SCM_TIMESTAMPING. This
285embeds the struct scm_timestamping.
Patrick Ohlycb9eff02009-02-12 05:03:36 +0000286
287
Willem de Bruijn8fe2f762014-08-31 21:27:47 -04002882.1.1.2 Timestamp types
289
290The semantics of the three struct timespec are defined by field
291ee_info in the extended error structure. It contains a value of
292type SCM_TSTAMP_* to define the actual timestamp passed in
293scm_timestamping.
294
295The SCM_TSTAMP_* types are 1:1 matches to the SOF_TIMESTAMPING_*
296control fields discussed previously, with one exception. For legacy
297reasons, SCM_TSTAMP_SND is equal to zero and can be set for both
298SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE. It
299is the first if ts[2] is non-zero, the second otherwise, in which
300case the timestamp is stored in ts[0].
301
302
3032.1.1.3 Fragmentation
304
305Fragmentation of outgoing datagrams is rare, but is possible, e.g., by
306explicitly disabling PMTU discovery. If an outgoing packet is fragmented,
307then only the first fragment is timestamped and returned to the sending
308socket.
309
310
3112.1.1.4 Packet Payload
312
313The calling application is often not interested in receiving the whole
314packet payload that it passed to the stack originally: the socket
315error queue mechanism is just a method to piggyback the timestamp on.
316In this case, the application can choose to read datagrams with a
317smaller buffer, possibly even of length 0. The payload is truncated
318accordingly. Until the process calls recvmsg() on the error queue,
319however, the full packet is queued, taking up budget from SO_RCVBUF.
320
321
3222.1.1.5 Blocking Read
323
324Reading from the error queue is always a non-blocking operation. To
325block waiting on a timestamp, use poll or select. poll() will return
326POLLERR in pollfd.revents if any data is ready on the error queue.
327There is no need to pass this flag in pollfd.events. This flag is
328ignored on request. See also `man 2 poll`.
329
330
3312.1.2 Receive timestamps
332
333On reception, there is no reason to read from the socket error queue.
334The SCM_TIMESTAMPING ancillary data is sent along with the packet data
335on a normal recvmsg(). Since this is not a socket error, it is not
336accompanied by a message SOL_IP(V6)/IP(V6)_RECVERROR. In this case,
337the meaning of the three fields in struct scm_timestamping is
338implicitly defined. ts[0] holds a software timestamp if set, ts[1]
339is again deprecated and ts[2] holds a hardware timestamp if set.
340
341
3423. Hardware Timestamping configuration: SIOCSHWTSTAMP and SIOCGHWTSTAMP
Patrick Ohlycb9eff02009-02-12 05:03:36 +0000343
344Hardware time stamping must also be initialized for each device driver
Patrick Loschmidt69298692010-04-07 21:52:07 -0700345that is expected to do hardware time stamping. The parameter is defined in
346/include/linux/net_tstamp.h as:
Patrick Ohlycb9eff02009-02-12 05:03:36 +0000347
348struct hwtstamp_config {
Patrick Loschmidt69298692010-04-07 21:52:07 -0700349 int flags; /* no flags defined right now, must be zero */
350 int tx_type; /* HWTSTAMP_TX_* */
351 int rx_filter; /* HWTSTAMP_FILTER_* */
Patrick Ohlycb9eff02009-02-12 05:03:36 +0000352};
353
354Desired behavior is passed into the kernel and to a specific device by
355calling ioctl(SIOCSHWTSTAMP) with a pointer to a struct ifreq whose
356ifr_data points to a struct hwtstamp_config. The tx_type and
357rx_filter are hints to the driver what it is expected to do. If
358the requested fine-grained filtering for incoming packets is not
359supported, the driver may time stamp more than just the requested types
360of packets.
361
362A driver which supports hardware time stamping shall update the struct
363with the actual, possibly more permissive configuration. If the
364requested packets cannot be time stamped, then nothing should be
365changed and ERANGE shall be returned (in contrast to EINVAL, which
366indicates that SIOCSHWTSTAMP is not supported at all).
367
368Only a processes with admin rights may change the configuration. User
369space is responsible to ensure that multiple processes don't interfere
370with each other and that the settings are reset.
371
Ben Hutchingsfd468c72013-11-14 01:19:29 +0000372Any process can read the actual configuration by passing this
373structure to ioctl(SIOCGHWTSTAMP) in the same way. However, this has
374not been implemented in all drivers.
375
Patrick Ohlycb9eff02009-02-12 05:03:36 +0000376/* possible values for hwtstamp_config->tx_type */
377enum {
378 /*
379 * no outgoing packet will need hardware time stamping;
380 * should a packet arrive which asks for it, no hardware
381 * time stamping will be done
382 */
383 HWTSTAMP_TX_OFF,
384
385 /*
386 * enables hardware time stamping for outgoing packets;
387 * the sender of the packet decides which are to be
388 * time stamped by setting SOF_TIMESTAMPING_TX_SOFTWARE
389 * before sending the packet
390 */
391 HWTSTAMP_TX_ON,
392};
393
394/* possible values for hwtstamp_config->rx_filter */
395enum {
396 /* time stamp no incoming packet at all */
397 HWTSTAMP_FILTER_NONE,
398
399 /* time stamp any incoming packet */
400 HWTSTAMP_FILTER_ALL,
401
Patrick Loschmidt69298692010-04-07 21:52:07 -0700402 /* return value: time stamp all packets requested plus some others */
403 HWTSTAMP_FILTER_SOME,
Patrick Ohlycb9eff02009-02-12 05:03:36 +0000404
405 /* PTP v1, UDP, any kind of event packet */
406 HWTSTAMP_FILTER_PTP_V1_L4_EVENT,
407
Patrick Loschmidt69298692010-04-07 21:52:07 -0700408 /* for the complete list of values, please check
409 * the include file /include/linux/net_tstamp.h
410 */
Patrick Ohlycb9eff02009-02-12 05:03:36 +0000411};
412
Willem de Bruijn8fe2f762014-08-31 21:27:47 -04004133.1 Hardware Timestamping Implementation: Device Drivers
Patrick Ohlycb9eff02009-02-12 05:03:36 +0000414
415A driver which supports hardware time stamping must support the
Patrick Loschmidt69298692010-04-07 21:52:07 -0700416SIOCSHWTSTAMP ioctl and update the supplied struct hwtstamp_config with
Ben Hutchingsfd468c72013-11-14 01:19:29 +0000417the actual values as described in the section on SIOCSHWTSTAMP. It
418should also support SIOCGHWTSTAMP.
Patrick Loschmidt69298692010-04-07 21:52:07 -0700419
420Time stamps for received packets must be stored in the skb. To get a pointer
421to the shared time stamp structure of the skb call skb_hwtstamps(). Then
422set the time stamps in the structure:
423
424struct skb_shared_hwtstamps {
425 /* hardware time stamp transformed into duration
426 * since arbitrary point in time
427 */
428 ktime_t hwtstamp;
Patrick Loschmidt69298692010-04-07 21:52:07 -0700429};
Patrick Ohlycb9eff02009-02-12 05:03:36 +0000430
431Time stamps for outgoing packets are to be generated as follows:
Oliver Hartkopp2244d072010-08-17 08:59:14 +0000432- In hard_start_xmit(), check if (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)
433 is set no-zero. If yes, then the driver is expected to do hardware time
434 stamping.
Patrick Ohlycb9eff02009-02-12 05:03:36 +0000435- If this is possible for the skb and requested, then declare
Oliver Hartkopp2244d072010-08-17 08:59:14 +0000436 that the driver is doing the time stamping by setting the flag
437 SKBTX_IN_PROGRESS in skb_shinfo(skb)->tx_flags , e.g. with
438
439 skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
440
441 You might want to keep a pointer to the associated skb for the next step
442 and not free the skb. A driver not supporting hardware time stamping doesn't
443 do that. A driver must never touch sk_buff::tstamp! It is used to store
444 software generated time stamps by the network subsystem.
Jakub Kicinski59cb89e2014-03-16 20:32:48 +0100445- Driver should call skb_tx_timestamp() as close to passing sk_buff to hardware
446 as possible. skb_tx_timestamp() provides a software time stamp if requested
447 and hardware timestamping is not possible (SKBTX_IN_PROGRESS not set).
Patrick Ohlycb9eff02009-02-12 05:03:36 +0000448- As soon as the driver has sent the packet and/or obtained a
449 hardware time stamp for it, it passes the time stamp back by
450 calling skb_hwtstamp_tx() with the original skb, the raw
Patrick Loschmidt69298692010-04-07 21:52:07 -0700451 hardware time stamp. skb_hwtstamp_tx() clones the original skb and
452 adds the timestamps, therefore the original skb has to be freed now.
453 If obtaining the hardware time stamp somehow fails, then the driver
454 should not fall back to software time stamping. The rationale is that
455 this would occur at a later time in the processing pipeline than other
456 software time stamping and therefore could lead to unexpected deltas
457 between time stamps.