Blame - tools/tcprtt_example.txt - platform/external/bcc

2020-08-23 19:17:52 +0800

[diff] [blame]

1

Demonstrations of tcprtt, the Linux eBPF/bcc version.

2

3

4

This program traces TCP RTT(round-trip time) to analyze the quality of

5

network, then help us to distinguish the network latency trouble is from

6

user process or physical network.

7

8

For example, wrk show the http request latency distribution:

9

# wrk -d 30 -c 10 --latency http://192.168.122.100/index.html

10

Running 30s test @ http://192.168.122.100/index.html

11

2 threads and 10 connections

12

Thread Stats Avg Stdev Max +/- Stdev

13

Latency 86.75ms 153.76ms 1.54s 90.85%

14

Req/Sec 160.91 76.07 424.00 67.06%

Latency Distribution

50% 14.55ms

75% 119.21ms

90% 230.22ms

99% 726.90ms

9523 requests in 30.02s, 69.62MB read

21

Socket errors: connect 0, read 0, write 0, timeout 1

22

23

During wrk testing, run tcprtt:

24

# ./tcprtt -i 1 -d 10 -m

25

Tracing TCP RTT... Hit Ctrl-C to end.

26

msecs : count distribution

27

0 -> 1 : 4 | |

28

2 -> 3 : 0 | |

29

4 -> 7 : 1055 |****************************************|

8 -> 15 : 26 | |

16 -> 31 : 0 | |

32 -> 63 : 0 | |

64 -> 127 : 18 | |

128 -> 255 : 14 | |

256 -> 511 : 14 | |

512 -> 1023 : 12 | |

The wrk output shows that the latency of web service is not stable, and tcprtt

39

also shows unstable TCP RTT. So in this situation, we need to make sure the

40

quality of network is good or not firstly.

41

42

zhenwei pi

2020-09-29 00:07:38 +0800

[diff] [blame]

43

Use filter for address and(or) port. Ex, only collect local address 192.168.122.200

44

and remote address 192.168.122.100 and remote port 80.

zhenwei pi

2020-08-23 19:17:52 +0800

[diff] [blame]

45

# ./tcprtt -i 1 -d 10 -m -a 192.168.122.200 -A 192.168.122.100 -P 80

46

47

zhenwei pi

2020-09-29 00:07:38 +0800

[diff] [blame]

48

Tracing at server side, show each clients with its own histogram.

49

For example, run tcprtt on a storage node to show initiators' rtt histogram:

zhenwei pi

2021-06-21 15:06:58 +0800

[diff] [blame]

50

# ./tcprtt -i 1 --lport 3260 --byraddr -e

zhenwei pi

2020-09-29 00:07:38 +0800

[diff] [blame]

51

Tracing TCP RTT... Hit Ctrl-C to end.

52

zhenwei pi

2021-06-21 15:06:58 +0800

[diff] [blame]

53

54

Remote Addres = 10.194.87.206 [AVG 170]

55

usecs : count distribution

zhenwei pi

2020-09-29 00:07:38 +0800

[diff] [blame]

56

0 -> 1 : 0 | |

57

2 -> 3 : 0 | |

58

4 -> 7 : 0 | |

zhenwei pi

2021-06-21 15:06:58 +0800

[diff] [blame]

8 -> 15 : 0 | |

16 -> 31 : 0 | |

32 -> 63 : 31 | |

64 -> 127 : 5150 |******************* |

63

128 -> 255 : 10327 |****************************************|

64

256 -> 511 : 1014 |*** |

65

512 -> 1023 : 10 | |

66

1024 -> 2047 : 7 | |

67

2048 -> 4095 : 14 | |

68

4096 -> 8191 : 10 | |

zhenwei pi

2020-09-29 00:07:38 +0800

[diff] [blame]

69

zhenwei pi

2021-06-21 15:06:58 +0800

[diff] [blame]

70

Remote Addres = 10.194.87.197 [AVG 4293]

71

usecs : count distribution

zhenwei pi

2020-09-29 00:07:38 +0800

[diff] [blame]

72

0 -> 1 : 0 | |

73

2 -> 3 : 0 | |

zhenwei pi

2021-06-21 15:06:58 +0800

[diff] [blame]

4 -> 7 : 0 | |

8 -> 15 : 0 | |

16 -> 31 : 0 | |

32 -> 63 : 0 | |

64 -> 127 : 0 | |

128 -> 255 : 0 | |

256 -> 511 : 0 | |

512 -> 1023 : 0 | |

1024 -> 2047 : 3 |******** |

83

2048 -> 4095 : 12 |********************************** |

84

4096 -> 8191 : 14 |****************************************|

zhenwei pi

2020-09-29 00:07:38 +0800

[diff] [blame]

85

zhenwei pi

2021-06-21 15:06:58 +0800

[diff] [blame]

86

Remote Addres = 10.194.88.148 [AVG 6215]

87

usecs : count distribution

zhenwei pi

2020-09-29 00:07:38 +0800

[diff] [blame]

88

0 -> 1 : 0 | |

89

2 -> 3 : 0 | |

zhenwei pi

2021-06-21 15:06:58 +0800

[diff] [blame]

4 -> 7 : 0 | |

8 -> 15 : 0 | |

16 -> 31 : 0 | |

32 -> 63 : 0 | |

64 -> 127 : 0 | |

128 -> 255 : 0 | |

256 -> 511 : 0 | |

512 -> 1023 : 0 | |

1024 -> 2047 : 0 | |

2048 -> 4095 : 0 | |

4096 -> 8191 : 2 |****************************************|

101

102

Remote Addres = 10.194.87.90 [AVG 2188]

103

usecs : count distribution

0 -> 1 : 0 | |

2 -> 3 : 0 | |

4 -> 7 : 0 | |

8 -> 15 : 0 | |

16 -> 31 : 0 | |

32 -> 63 : 0 | |

64 -> 127 : 0 | |

128 -> 255 : 0 | |

256 -> 511 : 15 |********* |

113

512 -> 1023 : 30 |****************** |

114

1024 -> 2047 : 50 |****************************** |

115

2048 -> 4095 : 65 |****************************************|

116

4096 -> 8191 : 22 |************* |

zhenwei pi

2020-09-29 00:07:38 +0800

[diff] [blame]

....

zhenwei pi

2021-06-21 15:06:58 +0800

[diff] [blame]

121

Use -e(--extension) to show extension RTT:

122

# ./tcprtt -i 1 -e

123

124

All Addresses = ******* [AVG 324]

125

usecs : count distribution

0 -> 1 : 0 | |

2 -> 3 : 0 | |

4 -> 7 : 0 | |

8 -> 15 : 0 | |

16 -> 31 : 0 | |

32 -> 63 : 0 | |

64 -> 127 : 5360 |******** |

133

128 -> 255 : 23834 |****************************************|

134

256 -> 511 : 11276 |****************** |

135

512 -> 1023 : 700 |* |

136

1024 -> 2047 : 434 | |

137

2048 -> 4095 : 356 | |

138

4096 -> 8191 : 328 | |

139

8192 -> 16383 : 91 | |

140

141

zhenwei pi

2020-08-23 19:17:52 +0800

[diff] [blame]

142

Full USAGE:

143

144

# ./tcprtt -h

zhenwei pi

2021-06-21 15:06:58 +0800

[diff] [blame]

145

usage: tcprtt [-h] [-i INTERVAL] [-d DURATION] [-T] [-m] [-p LPORT]

146

[-P RPORT] [-a LADDR] [-A RADDR] [-b] [-B] [-e] [-D]

Hariharan Ananthakrishnan

04893e3

2021-08-12 05:55:21 -0700

[diff] [blame]

147

[-4 | -6]

zhenwei pi

2020-08-23 19:17:52 +0800

[diff] [blame]

148

149

Summarize TCP RTT as a histogram

150

151

optional arguments:

152

-h, --help show this help message and exit

153

-i INTERVAL, --interval INTERVAL

154

summary interval, seconds

155

-d DURATION, --duration DURATION

156

total duration of trace, seconds

157

-T, --timestamp include timestamp on output

158

-m, --milliseconds millisecond histogram

zhenwei pi

2020-09-29 00:07:38 +0800

[diff] [blame]

159

-p LPORT, --lport LPORT

160

filter for local port

161

-P RPORT, --rport RPORT

162

filter for remote port

163

-a LADDR, --laddr LADDR

164

filter for local address

165

-A RADDR, --raddr RADDR

166

filter for remote address

167

-b, --byladdr show sockets histogram by local address

168

-B, --byraddr show sockets histogram by remote address

zhenwei pi

2021-06-21 15:06:58 +0800

[diff] [blame]

169

-e, --extension show extension summary(average)

zhenwei pi

2020-08-23 19:17:52 +0800

[diff] [blame]

170

-D, --debug print BPF program before starting (for debugging

171

purposes)

Hariharan Ananthakrishnan

04893e3

2021-08-12 05:55:21 -0700

[diff] [blame]

172

-4, --ipv4 trace IPv4 family only

173

-6, --ipv6 trace IPv6 family only

zhenwei pi

2020-08-23 19:17:52 +0800

[diff] [blame]

174

175

examples:

176

./tcprtt # summarize TCP RTT

177

./tcprtt -i 1 -d 10 # print 1 second summaries, 10 times

178

./tcprtt -m -T # summarize in millisecond, and timestamps

zhenwei pi

2020-09-29 00:07:38 +0800

[diff] [blame]

179

./tcprtt -p # filter for local port

180

./tcprtt -P # filter for remote port

181

./tcprtt -a # filter for local address

182

./tcprtt -A # filter for remote address

183

./tcprtt -b # show sockets histogram by local address

184

./tcprtt -B # show sockets histogram by remote address

zhenwei pi

2020-08-23 19:17:52 +0800

[diff] [blame]

185

./tcprtt -D # show debug bpf text

zhenwei pi