How to identify expected and unexpected crashes in tinderbox logs

I’ve seen this come up in several bugs recently, and it’s time to disseminate some knowledge. Here is what an unexpected crash usually looks like:

TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_seek.html | application timed out after 330 seconds with no output
PROCESS-CRASH | /tests/content/media/test/test_seek.html | application crashed (minidump found)

You’re looking at a test harness timeout because of a crash. Simple, easy to recognize. Even shows the crashing test for you!

PROCESS-CRASH | Main app process exited normally | application crashed (minidump found)

Here’s a sly one. This is a crash, but the lack of a test name and the “Main app process exited normally” actually means that a subprocess crashed intentionally. We’ve got tests that run scripts that cause crashes in child processes so that we can test the recovery behaviour in the parent, but unfortunately we display that information very well right now.

If you’re in doubt as to what kind of crash you’re seeing in the log, there’s another heuristic you can apply by looking at the crashing stack:

Crash reason: SIGSEGV
Crash address: 0x8

Thread 0 (crashed)
0 libxul.so!js::ctypes::ConvertToJS [typedefs.h:a538db9ab619 : 113 + 0x5]
rbx = 0x00000008 r12 = 0xa7833800 r13 = 0x00000000 r14 = 0x00000000
r15 = 0x00000000 rip = 0xb69142f5 rsp = 0xf25f7490 rbp = 0xa87a4690
Found by: given as instruction pointer in context
1 libxul.so!js::ctypes::PointerType::ContentsGetter [CTypes.cpp:a538db9ab619 : 3393 + 0x1b]
rbx = 0xa7833800 r12 = 0xa87a4690 r13 = 0xf25f74e8 r14 = 0xffffffff
r15 = 0xf25f7ae0 rip = 0xb69173cf rsp = 0xf25f74e0 rbp = 0xa877e750
Found by: call frame info

This is an intentional crash. We use jsctypes to dereference 0x8, an invalid address, and this is what it looks like every single time. If you don’t see this stack, you’re looking at a crash that should be filed.

So, to summarize: not every crash is unexpected. Keep your wits about you; know your crash stacks.

One Response to “How to identify expected and unexpected crashes in tinderbox logs”

  1. Sounds like https://bugzilla.mozilla.org/show_bug.cgi?id=539823 is confusing even more developers now that some of the stacks don’t have “nptest.dll!IntentionalCrash” at the top. We really should fix it.

Leave a Reply