| Brendan Gregg | 4f13d88 | 2018-05-30 17:09:04 -0400 | [diff] [blame] | 1 | Demonstrations of tcpdrop, the Linux BPF/bcc version. |
| 2 | |
| 3 | |
| 4 | tcpdrop prints details of TCP packets or segments that were dropped by the |
| 5 | kernel, including the kernel stack trace that led to the drop: |
| 6 | |
| 7 | # ./tcpdrop.py |
| 8 | TIME PID IP SADDR:SPORT > DADDR:DPORT STATE (FLAGS) |
| 9 | 20:49:06 0 4 10.32.119.56:443 > 10.66.65.252:22912 CLOSE (ACK) |
| 10 | tcp_drop+0x1 |
| 11 | tcp_v4_do_rcv+0x135 |
| 12 | tcp_v4_rcv+0x9c7 |
| 13 | ip_local_deliver_finish+0x62 |
| 14 | ip_local_deliver+0x6f |
| 15 | ip_rcv_finish+0x129 |
| 16 | ip_rcv+0x28f |
| 17 | __netif_receive_skb_core+0x432 |
| 18 | __netif_receive_skb+0x18 |
| 19 | netif_receive_skb_internal+0x37 |
| 20 | napi_gro_receive+0xc5 |
| 21 | ena_clean_rx_irq+0x3c3 |
| 22 | ena_io_poll+0x33f |
| 23 | net_rx_action+0x140 |
| 24 | __softirqentry_text_start+0xdf |
| 25 | irq_exit+0xb6 |
| 26 | do_IRQ+0x82 |
| 27 | ret_from_intr+0x0 |
| 28 | native_safe_halt+0x6 |
| 29 | default_idle+0x20 |
| 30 | arch_cpu_idle+0x15 |
| 31 | default_idle_call+0x23 |
| 32 | do_idle+0x17f |
| 33 | cpu_startup_entry+0x73 |
| 34 | rest_init+0xae |
| 35 | start_kernel+0x4dc |
| 36 | x86_64_start_reservations+0x24 |
| 37 | x86_64_start_kernel+0x74 |
| 38 | secondary_startup_64+0xa5 |
| 39 | |
| 40 | 20:49:50 12431 4 127.0.0.1:8198 > 127.0.0.1:48280 CLOSE (RST|ACK) |
| 41 | tcp_drop+0x1 |
| 42 | tcp_v4_do_rcv+0x135 |
| 43 | __release_sock+0x88 |
| 44 | release_sock+0x30 |
| 45 | inet_stream_connect+0x47 |
| 46 | SYSC_connect+0x9e |
| 47 | sys_connect+0xe |
| 48 | do_syscall_64+0x73 |
| 49 | entry_SYSCALL_64_after_hwframe+0x3d |
| 50 | |
| 51 | [...] |
| 52 | |
| 53 | The last two columns show the state of the TCP session, and the TCP flags. |
| 54 | These two examples show packets arriving for a session in the closed state, |
| 55 | that were dropped by the kernel. |
| 56 | |
| 57 | This tool is useful for debugging high rates of drops, which can cause the |
| 58 | remote end to do timer-based retransmits, hurting performance. |
| 59 | |
| 60 | |
| 61 | USAGE: |
| 62 | |
| 63 | # ./tcpdrop.py -h |
| 64 | usage: tcpdrop.py [-h] |
| 65 | |
| 66 | Trace TCP drops by the kernel |
| 67 | |
| 68 | optional arguments: |
| 69 | -h, --help show this help message and exit |
| 70 | |
| 71 | examples: |
| 72 | ./tcpdrop # trace kernel TCP drops |