1 Demonstrations of tplist. 2 3 4 tplist displays kernel tracepoints and USDT probes, including their 5 format. It can be used to discover probe points for use with the trace 6 and argdist tools. Kernel tracepoints are scattered around the kernel 7 and provide valuable static tracing on block and network I/O, scheduling, 8 power events, and many other subjects. USDT probes are placed in libraries 9 (such as libc) and executables (such as node) and provide static tracing 10 information that can (optionally) be turned on and off at runtime. 11 12 For example, suppose you want to discover which USDT probes a particular 13 executable contains. Just run tplist on that executable (or library): 14 15 $ tplist -l basic_usdt 16 /home/vagrant/basic_usdt basic_usdt:start_main 17 /home/vagrant/basic_usdt basic_usdt:loop_iter 18 /home/vagrant/basic_usdt basic_usdt:end_main 19 20 The loop_iter probe sounds interesting. How many arguments are available? 21 22 $ tplist '*loop_iter' -l basic_usdt -v 23 basic_usdt:loop_iter [sema 0x601036] 24 2 location(s) 25 2 argument(s) 26 27 This output indicates that the loop_iter probe is used in two locations 28 in the basic_usdt executable, and that it has two arguments. Fortunately, 29 the argdist and trace tools understand the probe format and can print out 30 the arguments automatically -- you can refer to them as arg1, arg2, and 31 so on. 32 33 Try to explore with some common libraries on your system and see if they 34 contain UDST probes. Here are two examples you might find interesting: 35 36 $ tplist -l pthread # list probes in libpthread 37 /lib64/libpthread.so.0 libpthread:pthread_start 38 /lib64/libpthread.so.0 libpthread:pthread_create 39 /lib64/libpthread.so.0 libpthread:pthread_join 40 /lib64/libpthread.so.0 libpthread:pthread_join_ret 41 /lib64/libpthread.so.0 libpthread:mutex_init 42 ... more output truncated 43 44 $ tplist -l c # list probes in libc 45 /lib64/libc.so.6 libc:setjmp 46 /lib64/libc.so.6 libc:longjmp 47 /lib64/libc.so.6 libc:longjmp_target 48 /lib64/libc.so.6 libc:memory_arena_reuse_free_list 49 /lib64/libc.so.6 libc:memory_heap_new 50 ... more output truncated 51 52 tplist also understands kernel tracepoints, and can list their format 53 as well. For example, let's look for all block I/O-related tracepoints: 54 55 # tplist 'block*' 56 block:block_touch_buffer 57 block:block_dirty_buffer 58 block:block_rq_abort 59 block:block_rq_requeue 60 block:block_rq_complete 61 block:block_rq_insert 62 block:block_rq_issue 63 block:block_bio_bounce 64 block:block_bio_complete 65 block:block_bio_backmerge 66 block:block_bio_frontmerge 67 block:block_bio_queue 68 block:block_getrq 69 block:block_sleeprq 70 block:block_plug 71 block:block_unplug 72 block:block_split 73 block:block_bio_remap 74 block:block_rq_remap 75 76 The block:block_rq_complete tracepoints sounds interesting. Let's print 77 its format to see what we can trace with argdist and trace: 78 79 $ tplist -v block:block_rq_complete 80 block:block_rq_complete 81 dev_t dev; 82 sector_t sector; 83 unsigned int nr_sector; 84 int errors; 85 char rwbs[8]; 86 87 The dev, sector, nr_sector, etc. variables can now all be used in probes 88 you specify with argdist or trace. 89 90 91 For debugging USDT probes, it is sometimes useful to see the exact locations 92 and arguments of the probes, including the registers or global variables from 93 which their values are coming from. In super-verbose mode, tplist will print 94 this information (note the -vv): 95 96 $ tplist -vv -l c *alloc* 97 libc:memory_malloc_retry [sema 0x0] 98 location #0 /lib64/libc.so.6 0x835c0 99 argument #0 8 unsigned bytes @ bp 100 location #1 /lib64/libc.so.6 0x83778 101 argument #0 8 unsigned bytes @ bp 102 location #2 /lib64/libc.so.6 0x85a50 103 argument #0 8 unsigned bytes @ bp 104 libc:memory_realloc_retry [sema 0x0] 105 location #0 /lib64/libc.so.6 0x84b90 106 argument #0 8 unsigned bytes @ r13 107 argument #1 8 unsigned bytes @ bp 108 location #1 /lib64/libc.so.6 0x85cf0 109 argument #0 8 unsigned bytes @ r13 110 argument #1 8 unsigned bytes @ bp 111 libc:memory_calloc_retry [sema 0x0] 112 location #0 /lib64/libc.so.6 0x850f0 113 argument #0 8 unsigned bytes @ bp 114 115 116 USAGE message: 117 118 $ tplist -h 119 usage: tplist.py [-h] [-p PID] [-l LIB] [-v] [filter] 120 121 Display kernel tracepoints or USDT probes and their formats. 122 123 positional arguments: 124 filter A filter that specifies which probes/tracepoints to print 125 126 optional arguments: 127 -h, --help show this help message and exit 128 -p PID, --pid PID List USDT probes in the specified process 129 -l LIB, --lib LIB List USDT probes in the specified library or executable 130 -v Increase verbosity level (print variables, arguments, etc.) 131 132
