Home | History | Annotate | Download | only in debug
      1 <html devsite>
      2   <head>
      3     <title>Diagnosing Native Crashes</title>
      4     <meta name="project_path" value="/_project.yaml" />
      5     <meta name="book_path" value="/_book.yaml" />
      6   </head>
      7   <body>
      8   <!--
      9       Copyright 2017 The Android Open Source Project
     10 
     11       Licensed under the Apache License, Version 2.0 (the "License");
     12       you may not use this file except in compliance with the License.
     13       You may obtain a copy of the License at
     14 
     15           http://www.apache.org/licenses/LICENSE-2.0
     16 
     17       Unless required by applicable law or agreed to in writing, software
     18       distributed under the License is distributed on an "AS IS" BASIS,
     19       WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
     20       See the License for the specific language governing permissions and
     21       limitations under the License.
     22   -->
     23 
     24 
     25 
     26 <p>
     27 If you've never seen a native crash before, start with
     28 <a href="/devices/tech/debug/index.html">Debugging Native Android
     29 Platform Code</a>.
     30 </p>
     31 
     32 <h2 id=crashtypes>Types of native crash</h2>
     33 <p>
     34 The sections below detail the most common kinds of native crash. Each includes
     35 an example chunk of <code>debuggerd</code> output, with the key evidence that helps you
     36 distinguish that specific kind of crash highlighted in orange italic text.
     37 </p>
     38 <h3 id=abort>Abort</h3>
     39 <p>
     40 Aborts are interesting because they're deliberate. There are many different ways
     41 to abort (including calling <code><a
     42 href="http://man7.org/linux/man-pages/man3/abort.3.html">abort(3)</a></code>,
     43 failing an <code><a
     44 href="http://man7.org/linux/man-pages/man3/assert.3.html">assert(3)</a></code>,
     45 using one of the Android-specific fatal logging types), but they all involve
     46 calling <code>abort</code>. A call to <code>abort</code> basically signals the
     47 calling thread with SIGABRT, so a frame showing "abort" in <code>libc.so</code>
     48 plus SIGABRT are the things to look for in the <code>debuggerd</code> output to
     49 recognize this case.</p>
     50 
     51 <p>
     52 As mentioned above, there may be an explicit "abort message" line. But you
     53 should also look in the <code>logcat</code> output to see what this thread logged before
     54 deliberately killing itself, because the basic abort primitive doesn't accept a
     55 message.
     56 </p>
     57 <p>
     58 Older versions of Android (especially on 32-bit ARM) followed a convoluted path
     59 between the original abort call (frame 4 here) and the actual sending of the
     60 signal (frame 0 here):
     61 </p>
     62 <pre class="devsite-click-to-copy">
     63 pid: 1656, tid: 1656, name: crasher  >>> crasher <<<
     64 signal 6 (<i style="color:Orange">SIGABRT</i>), code -6 (SI_TKILL), fault addr --------
     65 <i style="color:Orange">Abort message</i>: 'some_file.c:123: some_function: assertion "false" failed'
     66     r0 00000000  r1 00000678  r2 00000006  r3 f70b6dc8
     67     r4 f70b6dd0  r5 f70b6d80  r6 00000002  r7 0000010c
     68     r8 ffffffed  r9 00000000  sl 00000000  fp ff96ae1c
     69     ip 00000006  sp ff96ad18  lr f700ced5  pc f700dc98  cpsr 400b0010
     70 backtrace:
     71     #00 pc 00042c98  /system/lib/libc.so (tgkill+12)
     72     #01 pc 00041ed1  /system/lib/libc.so (pthread_kill+32)
     73     #02 pc 0001bb87  /system/lib/libc.so (raise+10)
     74     #03 pc 00018cad  /system/lib/libc.so (__libc_android_abort+34)
     75     #04 pc 000168e8  /system/lib/<i style="color:Orange">libc.so</i> (<i style="color:Orange">abort</i>+4)
     76     #05 pc 0001a78f  /system/lib/libc.so (__libc_fatal+16)
     77     #06 pc 00018d35  /system/lib/libc.so (__assert2+20)
     78     #07 pc 00000f21  /system/xbin/crasher
     79     #08 pc 00016795  /system/lib/libc.so (__libc_init+44)
     80     #09 pc 00000abc  /system/xbin/crasher
     81 </pre>
     82 <p>
     83 More recent versions call <code><a
     84 href="http://man7.org/linux/man-pages/man2/tgkill.2.html">tgkill(2)</a></code>
     85 directly from <code>abort</code>, so there are fewer stack frames for you to
     86 skip over before you get to the interesting frames:</p>
     87 
     88 <pre class="devsite-click-to-copy">
     89 pid: 25301, tid: 25301, name: crasher  >>> crasher <<<
     90 signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
     91     r0 00000000  r1 000062d5  r2 00000006  r3 00000008
     92     r4 ffa09dd8  r5 000062d5  r6 000062d5  r7 0000010c
     93     r8 00000000  r9 00000000  sl 00000000  fp ffa09f0c
     94     ip 00000000  sp ffa09dc8  lr eac63ce3  pc eac93f0c  cpsr 000d0010
     95 backtrace:
     96     #00 pc 00049f0c  /system/lib/libc.so (tgkill+12)
     97     #01 pc 00019cdf  /system/lib/libc.so (abort+50)
     98     #02 pc 000012db  /system/xbin/crasher (maybe_abort+26)
     99     #03 pc 000015b7  /system/xbin/crasher (do_action+414)
    100     #04 pc 000020d5  /system/xbin/crasher (main+100)
    101     #05 pc 000177a1  /system/lib/libc.so (__libc_init+48)
    102     #06 pc 000010e4  /system/xbin/crasher (_start+96)
    103 </pre>
    104 <p>
    105 You can reproduce an instance of this type of crash using: <code>crasher
    106 abort</code>
    107 </p>
    108 <h3 id=nullpointer>Pure null pointer dereference</h3>
    109 <p>
    110 This is the classic native crash, and although it's just a special case of the
    111 next crash type, it's worth mentioning separately because it usually requires
    112 the least thought.
    113 </p>
    114 <p>
    115 In the example below, even though the crashing function is in
    116 <code>libc.so</code>, because the string functions just operate on the pointers
    117 they're given, you can infer that <code><a
    118 href="http://man7.org/linux/man-pages/man3/strlen.3.html">strlen(3)</a></code>
    119 was called with a null pointer; and this crash should go straight to the author
    120 of the calling code. In this case, frame #01 is the bad caller.
    121 </p>
    122 
    123 <pre class="devsite-click-to-copy">
    124 pid: 25326, tid: 25326, name: crasher  >>> crasher <<<
    125 signal 11 (<i style="color:Orange">SIGSEGV</i>), code 1 (SEGV_MAPERR), <i style="color:Orange">fault addr 0x0</i>
    126     r0 00000000  r1 00000000  r2 00004c00  r3 00000000
    127     r4 ab088071  r5 fff92b34  r6 00000002  r7 fff92b40
    128     r8 00000000  r9 00000000  sl 00000000  fp fff92b2c
    129     ip ab08cfc4  sp fff92a08  lr ab087a93  pc efb78988  cpsr 600d0030
    130 
    131 backtrace:
    132     #00 pc 00019988  /system/lib/libc.so (strlen+71)
    133     #01 pc 00001a8f  /system/xbin/crasher (strlen_null+22)
    134     #02 pc 000017cd  /system/xbin/crasher (do_action+948)
    135     #03 pc 000020d5  /system/xbin/crasher (main+100)
    136     #04 pc 000177a1  /system/lib/libc.so (__libc_init+48)
    137     #05 pc 000010e4  /system/xbin/crasher (_start+96)
    138 </pre>
    139 <p>
    140 You can reproduce an instance of this type of crash using: <code>crasher
    141 strlen-NULL</code>
    142 </p>
    143 <h3 id=lowaddress>Low-address null pointer dereference</h3>
    144 <p>
    145 In many cases the fault address won't be 0, but some other low number. Two- or
    146 three-digit addresses in particular are very common, whereas a six-digit address
    147 is almost certainly not a null pointer dereference&#8212that would require a 1MiB
    148 offset. This usually occurs when you have code that dereferences a null pointer
    149 as if it was a valid struct. Common functions are <code><a
    150 href="http://man7.org/linux/man-pages/man3/fprintf.3.html">fprintf(3)</a></code>
    151 (or any other function taking a FILE*) and <code><a
    152 href="http://man7.org/linux/man-pages/man3/readdir.3.html">readdir(3)</a></code>,
    153 because code often fails to check that the <code><a
    154 href="http://man7.org/linux/man-pages/man3/fopen.3.html">fopen(3)</a></code> or
    155 <code><a
    156 href="http://man7.org/linux/man-pages/man3/opendir.3.html">opendir(3)</a></code>
    157 call actually succeeded first.
    158 </p>
    159 
    160 <p>
    161 Here's an example of <code>readdir</code>:
    162 </p>
    163 
    164 <pre class="devsite-click-to-copy">
    165 pid: 25405, tid: 25405, name: crasher  >>> crasher <<<
    166 signal 11 (<i style="color:Orange">SIGSEGV</i>), code 1 (SEGV_MAPERR), <i style="color:Orange">fault addr 0xc</i>
    167     r0 0000000c  r1 00000000  r2 00000000  r3 3d5f0000
    168     r4 00000000  r5 0000000c  r6 00000002  r7 ff8618f0
    169     r8 00000000  r9 00000000  sl 00000000  fp ff8618dc
    170     ip edaa6834  sp ff8617a8  lr eda34a1f  pc eda618f6  cpsr 600d0030
    171 
    172 backtrace:
    173     #00 pc 000478f6  /system/lib/libc.so (pthread_mutex_lock+1)
    174     #01 pc 0001aa1b  /system/lib/libc.so (readdir+10)
    175     #02 pc 00001b35  /system/xbin/crasher (readdir_null+20)
    176     #03 pc 00001815  /system/xbin/crasher (do_action+976)
    177     #04 pc 000021e5  /system/xbin/crasher (main+100)
    178     #05 pc 000177a1  /system/lib/libc.so (__libc_init+48)
    179     #06 pc 00001110  /system/xbin/crasher (_start+96)
    180 </pre>
    181 <p>
    182 Here the direct cause of the crash is that <code><a
    183 href="http://man7.org/linux/man-pages/man3/pthread_mutex_lock.3p.html">pthread_mutex_lock(3)</a></code>
    184 has tried to access address 0xc (frame 0). But the first thing
    185 <code>pthread_mutex_lock</code> does is dereference the <code>state</code>
    186 element of the <code>pthread_mutex_t*</code> it was given. If you look at the
    187 source, you can see that element is at offset 0 in the struct, which tells you
    188 that <code>pthread_mutex_lock</code> was given the invalid pointer 0xc. From the
    189 frame 1 you can see that it was given that pointer by <code>readdir</code>,
    190 which extracts the <code>mutex_</code> field from the <code>DIR*</code> it's
    191 given. Looking at that structure, you can see that <code>mutex_</code> is at
    192 offset <code>sizeof(int) + sizeof(size_t) + sizeof(dirent*)</code> into
    193 <code>struct DIR</code>, which on a 32-bit device is 4 + 4 + 4 = 12 = 0xc, so
    194 you found the bug: <code>readdir</code> was passed a null pointer by the caller.
    195 At this point you can paste the stack into the stack tool to find out
    196 <em>where</em> in logcat this happened.</p>
    197 
    198 <pre class="prettyprint">
    199   struct DIR {
    200     int fd_;
    201     size_t available_bytes_;
    202     dirent* next_;
    203     pthread_mutex_t mutex_;
    204     dirent buff_[15];
    205     long current_pos_;
    206   };
    207 </pre>
    208 <p>
    209 In most cases you can actually skip this analysis. A sufficiently low fault
    210 address usually means you can just skip any <code>libc.so</code> frames in the
    211 stack and directly accuse the calling code. But not always, and this is how you
    212 would present a compelling case.
    213 </p>
    214 <p>
    215 You can reproduce instances of this kind of crash using: <code>crasher
    216 fprintf-NULL</code> or <code>crasher readdir-NULL</code>
    217 </p>
    218 <h3 id=fortify>FORTIFY failure</h3>
    219 <p>
    220 A FORTIFY failure is a special case of an abort that occurs when the C library
    221 detects a problem that might lead to a security vulnerability. Many C library
    222 functions are <em>fortified</em>; they take an extra argument that tells them how large
    223 a buffer actually is and check at run time whether the operation you're trying
    224 to perform actually fits. Here's an example where the code tries to
    225 <code>read(fd, buf, 32)</code> into a buffer that's actually only 10 bytes
    226 long...
    227 </p>
    228 <pre class="devsite-click-to-copy">
    229 pid: 25579, tid: 25579, name: crasher  >>> crasher <<<
    230 signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
    231 Abort message: 'FORTIFY: read: prevented 32-byte write into 10-byte buffer'</i>
    232     r0 00000000  r1 000063eb  r2 00000006  r3 00000008
    233     r4 ff96f350  r5 000063eb  r6 000063eb  r7 0000010c
    234     r8 00000000  r9 00000000  sl 00000000  fp ff96f49c
    235     ip 00000000  sp ff96f340  lr ee83ece3  pc ee86ef0c  cpsr 000d0010
    236 
    237 backtrace:
    238     #00 pc 00049f0c  /system/lib/libc.so (tgkill+12)
    239     #01 pc 00019cdf  /system/lib/libc.so (abort+50)
    240     #02 pc 0001e197  /system/lib/libc.so (<i style="color:Orange">__fortify_fatal</i>+30)
    241     #03 pc 0001baf9  /system/lib/libc.so (__read_chk+48)
    242     #04 pc 0000165b  /system/xbin/crasher (do_action+534)
    243     #05 pc 000021e5  /system/xbin/crasher (main+100)
    244     #06 pc 000177a1  /system/lib/libc.so (__libc_init+48)
    245     #07 pc 00001110  /system/xbin/crasher (_start+96)
    246 </pre>
    247 <p>
    248 You can reproduce an instance of this type of crash using: <code>crasher
    249 fortify</code>
    250 </p>
    251 <h3 id=stackcorruption>Stack corruption detected by -fstack-protector</h3>
    252 <p>
    253 The compiler's <code>-fstack-protector</code> option inserts checks into
    254 functions with on-stack buffers to guard against buffer overruns. This option is
    255 on by default for platform code but not for apps. When this option is enabled,
    256 the compiler adds instructions to the <a
    257 href="https://en.wikipedia.org/wiki/Function_prologue">function prologue</a> to
    258 write a random value just past the last local on the stack and to the function
    259 epilogue to read it back and check that it's not changed. If that value changed,
    260 it was overwritten by a buffer overrun, so the epilogue calls
    261 <code>__stack_chk_fail</code> to log a message and abort.
    262 </p>
    263 <pre class="devsite-click-to-copy">
    264 pid: 26717, tid: 26717, name: crasher  >>> crasher <<<
    265 signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
    266 <i style="color:Orange">Abort message: 'stack corruption detected'</i>
    267     r0 00000000  r1 0000685d  r2 00000006  r3 00000008
    268     r4 ffd516d8  r5 0000685d  r6 0000685d  r7 0000010c
    269     r8 00000000  r9 00000000  sl 00000000  fp ffd518bc
    270     ip 00000000  sp ffd516c8  lr ee63ece3  pc ee66ef0c  cpsr 000e0010
    271 
    272 backtrace:
    273     #00 pc 00049f0c  /system/lib/libc.so (tgkill+12)
    274     #01 pc 00019cdf  /system/lib/libc.so (abort+50)
    275     #02 pc 0001e07d  /system/lib/libc.so (__libc_fatal+24)
    276     #03 pc 0004863f  /system/lib/libc.so (<i style="color:Orange">__stack_chk_fail</i>+6)
    277     #04 pc 000013ed  /system/xbin/crasher (smash_stack+76)
    278     #05 pc 00001591  /system/xbin/crasher (do_action+280)
    279     #06 pc 00002219  /system/xbin/crasher (main+100)
    280     #07 pc 000177a1  /system/lib/libc.so (__libc_init+48)
    281     #08 pc 00001144  /system/xbin/crasher (_start+96)
    282 </pre>
    283 <p>
    284 You can distinguish this from other kinds of abort by the presence of
    285 <code>__stack_chk_fail</code> in the backtrace and the specific abort message.
    286 </p>
    287 <p>
    288 You can reproduce an instance of this type of crash using: <code>crasher
    289 smash-stack</code>
    290 </p>
    291 
    292 <h2 id=crashdump>Crash dumps</h2>
    293 
    294 <p>If you don't have a specific crash that you're investigating right now,
    295 the platform source includes a tool for testing <code>debuggerd</code> called crasher. If
    296 you <code>mm</code> in <code>system/core/debuggerd/</code> you'll get both a <code>crasher</code>
    297 and a <code>crasher64</code> on your path (the latter allowing you to test
    298 64-bit crashes). Crasher can crash in a large number of interesting ways based
    299 on the command line arguments you provide. Use <code>crasher --help</code>
    300 to see the currently supported selection.</p>
    301 
    302 <p>To introduce the different pieces in a crash dump, let's work through this example crash dump:</p>
    303 
    304 <pre class="devsite-click-to-copy">
    305 *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
    306 Build fingerprint: 'Android/aosp_flounder/flounder:5.1.51/AOSP/enh08201009:eng/test-keys'
    307 Revision: '0'
    308 ABI: 'arm'
    309 pid: 1656, tid: 1656, name: crasher  &gt;&gt;&gt; crasher &lt;&lt;&lt;
    310 signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
    311 Abort message: 'some_file.c:123: some_function: assertion "false" failed'
    312     r0 00000000  r1 00000678  r2 00000006  r3 f70b6dc8
    313     r4 f70b6dd0  r5 f70b6d80  r6 00000002  r7 0000010c
    314     r8 ffffffed  r9 00000000  sl 00000000  fp ff96ae1c
    315     ip 00000006  sp ff96ad18  lr f700ced5  pc f700dc98  cpsr 400b0010
    316 backtrace:
    317     #00 pc 00042c98  /system/lib/libc.so (tgkill+12)
    318     #01 pc 00041ed1  /system/lib/libc.so (pthread_kill+32)
    319     #02 pc 0001bb87  /system/lib/libc.so (raise+10)
    320     #03 pc 00018cad  /system/lib/libc.so (__libc_android_abort+34)
    321     #04 pc 000168e8  /system/lib/libc.so (abort+4)
    322     #05 pc 0001a78f  /system/lib/libc.so (__libc_fatal+16)
    323     #06 pc 00018d35  /system/lib/libc.so (__assert2+20)
    324     #07 pc 00000f21  /system/xbin/crasher
    325     #08 pc 00016795  /system/lib/libc.so (__libc_init+44)
    326     #09 pc 00000abc  /system/xbin/crasher
    327 Tombstone written to: /data/tombstones/tombstone_06
    328 *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
    329 </pre>
    330 
    331 <p>The line of asterisks with spaces is helpful if you're searching a log
    332 for native crashes. The string "*** ***" rarely shows up in logs other than
    333 at the beginning of a native crash.</p>
    334 
    335 <pre class="devsite-click-to-copy">
    336 Build fingerprint:
    337 'Android/aosp_flounder/flounder:5.1.51/AOSP/enh08201009:eng/test-keys'
    338 </pre>
    339 
    340 <p>The fingerprint lets you identify exactly which build the crash occurred
    341 on. This is exactly the same as the <code>ro.build.fingerprint</code> system property.</p>
    342 
    343 <pre class="devsite-click-to-copy">
    344 Revision: '0'
    345 </pre>
    346 
    347 <p>The revision refers to the hardware rather than the software. This is
    348 usually unused but can be useful to help you automatically ignore bugs known
    349 to be caused by bad hardware. This is exactly the same as the <code>ro.revision</code>
    350 system property.</p>
    351 
    352 <pre class="devsite-click-to-copy">
    353 ABI: 'arm'
    354 </pre>
    355 
    356 <p>The ABI is one of arm, arm64, mips, mips64, x86, or x86-64. This is
    357 mostly useful for the <code>stack</code> script mentioned above, so that it knows
    358 what toolchain to use.</p>
    359 
    360 <pre class="devsite-click-to-copy">
    361 pid: 1656, tid: 1656, name: crasher &gt;&gt;&gt; crasher &lt;&lt;&lt;
    362 </pre>
    363 
    364 <p>This line identifies the specific thread in the process that crashed. In
    365 this case, it was the process' main thread, so the process ID and thread
    366 ID match. The first name is the thread name, and the name surrounded by
    367 &gt;&gt;&gt; and &lt;&lt;&lt; is the process name. For an app, the process name
    368 is typically the fully-qualified package name (such as com.facebook.katana),
    369 which is useful when filing bugs or trying to find the app in Google Play. The
    370 pid and tid can also be useful in finding the relevant log lines preceding
    371 the crash.</p>
    372 
    373 <pre class="devsite-click-to-copy">
    374 signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
    375 </pre>
    376 
    377 <p>This line tells you which signal (SIGABRT) was received, and more about
    378 how it was received (SI_TKILL). The signals reported by <code>debuggerd</code> are SIGABRT,
    379 SIGBUS, SIGFPE, SIGILL, SIGSEGV, and SIGTRAP. The signal-specific codes vary
    380 based on the specific signal.</p>
    381 
    382 <pre class="devsite-click-to-copy">
    383 Abort message: 'some_file.c:123: some_function: assertion "false" failed'
    384 </pre>
    385 
    386 <p>Not all crashes will have an abort message line, but aborts will. This
    387 is automatically gathered from the last line of fatal logcat output for
    388 this pid/tid, and in the case of a deliberate abort is likely to give an
    389 explanation of why the program killed itself.</p>
    390 
    391 <pre class="devsite-click-to-copy">
    392 r0 00000000 r1 00000678 r2 00000006 r3 f70b6dc8
    393 r4 f70b6dd0 r5 f70b6d80 r6 00000002 r7 0000010c
    394 r8 ffffffed r9 00000000 sl 00000000 fp ff96ae1c
    395 ip 00000006 sp ff96ad18 lr f700ced5 pc f700dc98 cpsr 400b0010
    396 </pre>
    397 
    398 <p>The register dump shows the content of the CPU registers at the time the
    399 signal was received. (This section varies wildly between ABIs.) How useful
    400 these are will depend on the exact crash.</p>
    401 
    402 <pre class="devsite-click-to-copy">
    403 backtrace:
    404     #00 pc 00042c98 /system/lib/libc.so (tgkill+12)
    405     #01 pc 00041ed1 /system/lib/libc.so (pthread_kill+32)
    406     #02 pc 0001bb87 /system/lib/libc.so (raise+10)
    407     #03 pc 00018cad /system/lib/libc.so (__libc_android_abort+34)
    408     #04 pc 000168e8 /system/lib/libc.so (abort+4)
    409     #05 pc 0001a78f /system/lib/libc.so (__libc_fatal+16)
    410     #06 pc 00018d35 /system/lib/libc.so (__assert2+20)
    411     #07 pc 00000f21 /system/xbin/crasher
    412     #08 pc 00016795 /system/lib/libc.so (__libc_init+44)
    413     #09 pc 00000abc /system/xbin/crasher
    414 </pre>
    415 
    416 <p>The backtrace shows you where in the code we were at the time of
    417 crash. The first column is the frame number (matching gdb's style where
    418 the deepest frame is 0). The PC values are relative to the location of the
    419 shared library rather than absolute addresses. The next column is the name
    420 of the mapped region (which is usually a shared library or executable, but
    421 might not be for, say, JIT-compiled code). Finally, if symbols are available,
    422 the symbol that the PC value corresponds to is shown, along with the offset
    423 into that symbol in bytes. You can use this in conjunction with <code>objdump(1)</code>
    424 to find the corresponding assembler instruction.</p>
    425 
    426 <h2 id=tombstones>Tombstones</h2>
    427 
    428 <pre class="devsite-click-to-copy">
    429 Tombstone written to: /data/tombstones/tombstone_06
    430 </pre>
    431 
    432 <p>This tells you where <code>debuggerd</code> wrote extra information.
    433 <code>debuggerd</code> will keep up to 10 tombstones, cycling through
    434 the numbers 00 to 09 and overwriting existing tombstones as necessary.</p>
    435 
    436 <p>The tombstone contains the same information as the crash dump, plus a
    437 few extras. For example, it includes backtraces for <i>all</i> threads (not
    438 just the crashing thread), the floating point registers, raw stack dumps,
    439 and memory dumps around the addresses in registers. Most usefully it also
    440 includes a full memory map (similar to <code>/proc/<i>pid</i>/maps</code>). Here's an
    441 annotated example from a 32-bit ARM process crash:</p>
    442 
    443 <pre class="devsite-click-to-copy">
    444 memory map: (fault address prefixed with ---&gt;)
    445 ---&gt;ab15f000-ab162fff r-x 0 4000 /system/xbin/crasher (BuildId:
    446 b9527db01b5cf8f5402f899f64b9b121)
    447 </pre>
    448 
    449 <p>There are two things to note here. The first is that this line is prefixed
    450 with "---&gt;". The maps are most useful when your crash isn't just a null
    451 pointer dereference. If the fault address is small, it's probably some variant
    452 of a null pointer dereference. Otherwise looking at the maps around the fault
    453 address can often give you a clue as to what happened. Some possible issues
    454 that can be recognized by looking at the maps include:</p>
    455 
    456 <ul>
    457 <li>Reads/writes past the end of a block of memory.</li>
    458 <li>Reads/writes before the beginning of a block of memory.</li>
    459 <li>Attempts to execute non-code.</li>
    460 <li>Running off the end of a stack.</li>
    461 <li>Attempts to write to code (as in the example above).</li>
    462 </ul>
    463 
    464 <p>The second thing to note is that executables and shared libraries files
    465 will show the BuildId (if present) in Android M and later, so you can see
    466 exactly which version of your code crashed. (Platform binaries include a
    467 BuildId by default since Android M. NDK r12 and later automatically pass
    468 <code>-Wl,--build-id</code> to the linker too.)</p>
    469 
    470 <pre class="devsite-click-to-copy">
    471 ab163000-ab163fff r--      3000      1000  /system/xbin/crasher
    472 ab164000-ab164fff rw-         0      1000
    473 f6c80000-f6d7ffff rw-         0    100000  [anon:libc_malloc]
    474 </pre>
    475 
    476 <p>On Android the heap isn't necessarily a single region. Heap regions will
    477 be labeled <code>[anon:libc_malloc]</code>.</p>
    478 
    479 <pre class="devsite-click-to-copy">
    480 f6d82000-f6da1fff r--         0     20000  /dev/__properties__/u:object_r:logd_prop:s0
    481 f6da2000-f6dc1fff r--         0     20000  /dev/__properties__/u:object_r:default_prop:s0
    482 f6dc2000-f6de1fff r--         0     20000  /dev/__properties__/u:object_r:logd_prop:s0
    483 f6de2000-f6de5fff r-x         0      4000  /system/lib/libnetd_client.so (BuildId: 08020aa06ed48cf9f6971861abf06c9d)
    484 f6de6000-f6de6fff r--      3000      1000  /system/lib/libnetd_client.so
    485 f6de7000-f6de7fff rw-      4000      1000  /system/lib/libnetd_client.so
    486 f6dec000-f6e74fff r-x         0     89000  /system/lib/libc++.so (BuildId: 8f1f2be4b37d7067d366543fafececa2) (load base 0x2000)
    487 f6e75000-f6e75fff ---         0      1000
    488 f6e76000-f6e79fff r--     89000      4000  /system/lib/libc++.so
    489 f6e7a000-f6e7afff rw-     8d000      1000  /system/lib/libc++.so
    490 f6e7b000-f6e7bfff rw-         0      1000  [anon:.bss]
    491 f6e7c000-f6efdfff r-x         0     82000  /system/lib/libc.so (BuildId: d189b369d1aafe11feb7014d411bb9c3)
    492 f6efe000-f6f01fff r--     81000      4000  /system/lib/libc.so
    493 f6f02000-f6f03fff rw-     85000      2000  /system/lib/libc.so
    494 f6f04000-f6f04fff rw-         0      1000  [anon:.bss]
    495 f6f05000-f6f05fff r--         0      1000  [anon:.bss]
    496 f6f06000-f6f0bfff rw-         0      6000  [anon:.bss]
    497 f6f0c000-f6f21fff r-x         0     16000  /system/lib/libcutils.so (BuildId: d6d68a419dadd645ca852cd339f89741)
    498 f6f22000-f6f22fff r--     15000      1000  /system/lib/libcutils.so
    499 f6f23000-f6f23fff rw-     16000      1000  /system/lib/libcutils.so
    500 f6f24000-f6f31fff r-x         0      e000  /system/lib/liblog.so (BuildId: e4d30918d1b1028a1ba23d2ab72536fc)
    501 f6f32000-f6f32fff r--      d000      1000  /system/lib/liblog.so
    502 f6f33000-f6f33fff rw-      e000      1000  /system/lib/liblog.so
    503 </pre>
    504 
    505 <p>Typically a shared library will have three adjacent entries. One will be
    506 readable and executable (code), one will be read-only (read-only
    507 data), and one will be read-write (mutable data). The first column
    508 shows the address ranges for the mapping, the second column the permissions
    509 (in the usual Unix <code>ls(1)</code> style), the third column the offset into the file
    510 (in hex), the fourth column the size of the region (in hex), and the fifth
    511 column the file (or other region name).</p>
    512 
    513 <pre class="devsite-click-to-copy">
    514 f6f34000-f6f53fff r-x         0     20000  /system/lib/libm.so (BuildId: 76ba45dcd9247e60227200976a02c69b)
    515 f6f54000-f6f54fff ---         0      1000
    516 f6f55000-f6f55fff r--     20000      1000  /system/lib/libm.so
    517 f6f56000-f6f56fff rw-     21000      1000  /system/lib/libm.so
    518 f6f58000-f6f58fff rw-         0      1000
    519 f6f59000-f6f78fff r--         0     20000  /dev/__properties__/u:object_r:default_prop:s0
    520 f6f79000-f6f98fff r--         0     20000  /dev/__properties__/properties_serial
    521 f6f99000-f6f99fff rw-         0      1000  [anon:linker_alloc_vector]
    522 f6f9a000-f6f9afff r--         0      1000  [anon:atexit handlers]
    523 f6f9b000-f6fbafff r--         0     20000  /dev/__properties__/properties_serial
    524 f6fbb000-f6fbbfff rw-         0      1000  [anon:linker_alloc_vector]
    525 f6fbc000-f6fbcfff rw-         0      1000  [anon:linker_alloc_small_objects]
    526 f6fbd000-f6fbdfff rw-         0      1000  [anon:linker_alloc_vector]
    527 f6fbe000-f6fbffff rw-         0      2000  [anon:linker_alloc]
    528 f6fc0000-f6fc0fff r--         0      1000  [anon:linker_alloc]
    529 f6fc1000-f6fc1fff rw-         0      1000  [anon:linker_alloc_lob]
    530 f6fc2000-f6fc2fff r--         0      1000  [anon:linker_alloc]
    531 f6fc3000-f6fc3fff rw-         0      1000  [anon:linker_alloc_vector]
    532 f6fc4000-f6fc4fff rw-         0      1000  [anon:linker_alloc_small_objects]
    533 f6fc5000-f6fc5fff rw-         0      1000  [anon:linker_alloc_vector]
    534 f6fc6000-f6fc6fff rw-         0      1000  [anon:linker_alloc_small_objects]
    535 f6fc7000-f6fc7fff rw-         0      1000  [anon:arc4random _rsx structure]
    536 f6fc8000-f6fc8fff rw-         0      1000  [anon:arc4random _rs structure]
    537 f6fc9000-f6fc9fff r--         0      1000  [anon:atexit handlers]
    538 f6fca000-f6fcafff ---         0      1000  [anon:thread signal stack guard page]
    539 </pre>
    540 
    541 <p>
    542 Note that since Android 5.0 (Lollipop), the C library names most of its anonymous mapped
    543 regions so there are fewer mystery regions.
    544 </p>
    545 
    546 <pre class="devsite-click-to-copy">
    547 f6fcb000-f6fccfff rw- 0 2000 [stack:5081]
    548 </pre>
    549 
    550 <p>
    551 Regions named <code>[stack:<i>tid</i>]</code> are the stacks for the given threads.
    552 </p>
    553 
    554 <pre class="devsite-click-to-copy">
    555 f6fcd000-f702afff r-x         0     5e000  /system/bin/linker (BuildId: 84f1316198deee0591c8ac7f158f28b7)
    556 f702b000-f702cfff r--     5d000      2000  /system/bin/linker
    557 f702d000-f702dfff rw-     5f000      1000  /system/bin/linker
    558 f702e000-f702ffff rw-         0      2000
    559 f7030000-f7030fff r--         0      1000
    560 f7031000-f7032fff rw-         0      2000
    561 ffcd7000-ffcf7fff rw-         0     21000
    562 ffff0000-ffff0fff r-x         0      1000  [vectors]
    563 </pre>
    564 
    565 <p>Whether you see <code>[vector]</code> or <code>[vdso]</code> depends on the architecture. ARM uses [vector], while all other architectures use <a href="http://man7.org/linux/man-pages/man7/vdso.7.html">[vdso].</a></p>
    566   </body>
    567 </html>
    568