Pekka Paalanen | c6c67c1 | 2008-05-12 21:20:59 +0200 | [diff] [blame^] | 1 | In-kernel memory-mapped I/O tracing |
| 2 | |
| 3 | |
| 4 | Home page and links to optional user space tools: |
| 5 | |
| 6 | http://nouveau.freedesktop.org/wiki/MmioTrace |
| 7 | |
| 8 | MMIO tracing was originally developed by Intel around 2003 for their Fault |
| 9 | Injection Test Harness. In Dec 2006 - Jan 2007, using the code from Intel, |
| 10 | Jeff Muizelaar created a tool for tracing MMIO accesses with the Nouveau |
| 11 | project in mind. Since then many people have contributed. |
| 12 | |
| 13 | Mmiotrace was built for reverse engineering any memory-mapped IO device with |
| 14 | the Nouveau project as the first real user. Only x86 and x86_64 architectures |
| 15 | are supported. |
| 16 | |
| 17 | Out-of-tree mmiotrace was originally modified for mainline inclusion and |
| 18 | ftrace framework by Pekka Paalanen <pq@iki.fi>. |
| 19 | |
| 20 | |
| 21 | Preparation |
| 22 | ----------- |
| 23 | |
| 24 | Mmiotrace feature is compiled in by the CONFIG_MMIOTRACE option. Tracing is |
| 25 | disabled by default, so it is safe to have this set to yes. SMP systems are |
| 26 | supported, but tracing is unreliable and may miss events if more than one CPU |
| 27 | is on-line, therefore mmiotrace takes all but one CPU off-line during run-time |
| 28 | activation [not implemented]. |
| 29 | |
| 30 | |
| 31 | Usage Quick Reference |
| 32 | --------------------- |
| 33 | |
| 34 | $ mount -t debugfs debugfs /debug |
| 35 | $ echo mmiotrace > /debug/tracing/current_tracer |
| 36 | $ cat /debug/tracing/trace_pipe > mydump.txt & |
| 37 | Start X or whatever. |
| 38 | $ echo "X is up" > /debug/tracing/marker |
| 39 | $ echo none > /debug/tracing/current_tracer |
| 40 | Check kernel log. |
| 41 | |
| 42 | |
| 43 | Usage |
| 44 | ----- |
| 45 | |
| 46 | Make sure debugfs is mounted to /debug. If not, (requires root privileges) |
| 47 | $ mount -t debugfs debugfs /debug |
| 48 | |
| 49 | Check that the driver you are about to trace is not loaded. |
| 50 | |
| 51 | Activate mmiotrace (requires root privileges): |
| 52 | $ echo mmiotrace > /debug/tracing/current_tracer |
| 53 | |
| 54 | Start storing the trace: |
| 55 | $ cat /debug/tracing/trace_pipe > mydump.txt & |
| 56 | The 'cat' process should stay running (sleeping) in the background. |
| 57 | |
| 58 | Load the driver you want to trace and use it. Mmiotrace will only catch MMIO |
| 59 | accesses to areas that are ioremapped while mmiotrace is active. |
| 60 | |
| 61 | [Unimplemented feature:] |
| 62 | During tracing you can place comments (markers) into the trace by |
| 63 | $ echo "X is up" > /debug/tracing/marker |
| 64 | This makes it easier to see which part of the (huge) trace corresponds to |
| 65 | which action. It is recommended to place descriptive markers about what you |
| 66 | do. |
| 67 | |
| 68 | Shut down mmiotrace (requires root privileges): |
| 69 | $ echo none > /debug/tracing/current_tracer |
| 70 | The 'cat' process exits. If it does not, kill it by 'fg' and pressing ctrl+c. |
| 71 | |
| 72 | [This feature is not implemented yet!] |
| 73 | Check your kernel log. If there are messages about mmiotrace losing events, |
| 74 | this is due to buffer overrun, and the trace is incomplete. You should enlarge |
| 75 | the buffers and try again. [How?] |
| 76 | |
| 77 | If you are doing a trace for a driver project, e.g. Nouveau, you should also |
| 78 | do the following before sending your results: |
| 79 | $ lspci -vvv > lspci.txt |
| 80 | $ dmesg > dmesg.txt |
| 81 | $ tar zcf pciid-nick-mmiotrace.tar.gz mydump.txt lspci.txt dmesg.txt |
| 82 | and then send the .tar.gz file. The trace compresses considerably. Replace |
| 83 | "pciid" and "nick" with the PCI ID or model name of your piece of hardware |
| 84 | under investigation and your nick name. |
| 85 | |
| 86 | |
| 87 | How Mmiotrace Works |
| 88 | ------------------- |
| 89 | |
| 90 | Access to hardware IO-memory is gained by mapping addresses from PCI bus by |
| 91 | calling one of the ioremap_*() functions. Mmiotrace is hooked into the |
| 92 | __ioremap() function and gets called whenever a mapping is created. Mapping is |
| 93 | an event that is recorded into the trace log. Note, that ISA range mappings |
| 94 | are not caught, since the mapping always exists and is returned directly. |
| 95 | |
| 96 | MMIO accesses are recorded via page faults. Just before __ioremap() returns, |
| 97 | the mapped pages are marked as not present. Any access to the pages causes a |
| 98 | fault. The page fault handler calls mmiotrace to handle the fault. Mmiotrace |
| 99 | marks the page present, sets TF flag to achieve single stepping and exits the |
| 100 | fault handler. The instruction that faulted is executed and debug trap is |
| 101 | entered. Here mmiotrace again marks the page as not present. The instruction |
| 102 | is decoded to get the type of operation (read/write), data width and the value |
| 103 | read or written. These are stored to the trace log. |
| 104 | |
| 105 | Setting the page present in the page fault handler has a race condition on SMP |
| 106 | machines. During the single stepping other CPUs may run freely on that page |
| 107 | and events can be missed without a notice. Re-enabling other CPUs during |
| 108 | tracing is discouraged. |
| 109 | |
| 110 | |
| 111 | Trace Log Format |
| 112 | ---------------- |
| 113 | |
| 114 | The raw log is text and easily filtered with e.g. grep and awk. One record is |
| 115 | one line in the log. A record starts with a keyword, followed by keyword |
| 116 | dependant arguments. Arguments are separated by a space, or continue until the |
| 117 | end of line. The format for version 20070824 is as follows: |
| 118 | |
| 119 | Explanation Keyword Space separated arguments |
| 120 | --------------------------------------------------------------------------- |
| 121 | |
| 122 | read event R width, timestamp, map id, physical, value, PC, PID |
| 123 | write event W width, timestamp, map id, physical, value, PC, PID |
| 124 | ioremap event MAP timestamp, map id, physical, virtual, length, PC, PID |
| 125 | iounmap event UNMAP timestamp, map id, PC, PID |
| 126 | marker MARK timestamp, text |
| 127 | version VERSION the string "20070824" |
| 128 | info for reader LSPCI one line from lspci -v |
| 129 | PCI address map PCIDEV space separated /proc/bus/pci/devices data |
| 130 | unk. opcode UNKNOWN timestamp, map id, physical, data, PC, PID |
| 131 | |
| 132 | Timestamp is in seconds with decimals. Physical is a PCI bus address, virtual |
| 133 | is a kernel virtual address. Width is the data width in bytes and value is the |
| 134 | data value. Map id is an arbitrary id number identifying the mapping that was |
| 135 | used in an operation. PC is the program counter and PID is process id. PC is |
| 136 | zero if it is not recorded. PID is always zero as tracing MMIO accesses |
| 137 | originating in user space memory is not yet supported. |
| 138 | |
| 139 | For instance, the following awk filter will pass all 32-bit writes that target |
| 140 | physical addresses in the range [0xfb73ce40, 0xfb800000[ |
| 141 | |
| 142 | $ awk '/W 4 / { adr=strtonum($5); if (adr >= 0xfb73ce40 && |
| 143 | adr < 0xfb800000) print; }' |
| 144 | |
| 145 | |
| 146 | Tools for Developers |
| 147 | -------------------- |
| 148 | |
| 149 | The user space tools include utilities for: |
| 150 | - replacing numeric addresses and values with hardware register names |
| 151 | - replaying MMIO logs, i.e., re-executing the recorded writes |
| 152 | |
| 153 | |