1 Demonstrations of tcpretrans, the Linux eBPF/bcc version. 2 3 4 This tool traces the kernel TCP retransmit function to show details of these 5 retransmits. For example: 6 7 # ./tcpretrans 8 TIME PID IP LADDR:LPORT T> RADDR:RPORT STATE 9 01:55:05 0 4 10.153.223.157:22 R> 69.53.245.40:34619 ESTABLISHED 10 01:55:05 0 4 10.153.223.157:22 R> 69.53.245.40:34619 ESTABLISHED 11 01:55:17 0 4 10.153.223.157:22 R> 69.53.245.40:22957 ESTABLISHED 12 [...] 13 14 This output shows three TCP retransmits, the first two were for an IPv4 15 connection from 10.153.223.157 port 22 to 69.53.245.40 port 34619. The TCP 16 state was "ESTABLISHED" at the time of the retransmit. The on-CPU PID at the 17 time of the retransmit is printed, in this case 0 (the kernel, which will 18 be the case most of the time). 19 20 Retransmits are usually a sign of poor network health, and this tool is 21 useful for their investigation. Unlike using tcpdump, this tool has very 22 low overhead, as it only traces the retransmit function. It also prints 23 additional kernel details: the state of the TCP session at the time of the 24 retransmit. 25 26 27 A -l option will include TCP tail loss probe attempts: 28 29 # ./tcpretrans -l 30 TIME PID IP LADDR:LPORT T> RADDR:RPORT STATE 31 01:55:45 0 4 10.153.223.157:22 R> 69.53.245.40:51601 ESTABLISHED 32 01:55:46 0 4 10.153.223.157:22 R> 69.53.245.40:51601 ESTABLISHED 33 01:55:46 0 4 10.153.223.157:22 R> 69.53.245.40:51601 ESTABLISHED 34 01:55:53 0 4 10.153.223.157:22 L> 69.53.245.40:46444 ESTABLISHED 35 01:56:06 0 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED 36 01:56:06 0 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED 37 01:56:08 0 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED 38 01:56:08 0 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED 39 01:56:08 1938 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED 40 01:56:08 0 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED 41 01:56:08 0 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED 42 [...] 43 44 See the "L>" in the "T>" column. These are attempts: the kernel probably 45 sent a TLP, but in some cases it might not have been ultimately sent. 46 47 To spot heavily retransmitting flows quickly one can use the -c flag. It will 48 count occurring retransmits per flow. 49 50 # ./tcpretrans.py -c 51 Tracing retransmits ... Hit Ctrl-C to end 52 ^C 53 LADDR:LPORT RADDR:RPORT RETRANSMITS 54 192.168.10.50:60366 <-> 172.217.21.194:443 700 55 192.168.10.50:666 <-> 172.213.11.195:443 345 56 192.168.10.50:366 <-> 172.212.22.194:443 211 57 [...] 58 59 This can ease to quickly isolate congested or otherwise awry network paths 60 responsible for clamping tcp performance. 61 62 USAGE message: 63 64 # ./tcpretrans -h 65 usage: tcpretrans [-h] [-l] 66 67 Trace TCP retransmits 68 69 optional arguments: 70 -h, --help show this help message and exit 71 -l, --lossprobe include tail loss probe attempts 72 -c, --count count occurred retransmits per flow 73 74 examples: 75 ./tcpretrans # trace TCP retransmits 76 ./tcpretrans -l # include TLP attempts 77
