print - example stack trace c++




How to automatically generate a stacktrace when my program crashes (19)

Linux

While the use of the backtrace() functions in execinfo.h to print a stacktrace and exit gracefully when you get a segmentation fault has already been suggested, I see no mention of the intricacies necessary to ensure the resulting backtrace points to the actual location of the fault (at least for some architectures - x86 & ARM).

The first two entries in the stack frame chain when you get into the signal handler contain a return address inside the signal handler and one inside sigaction() in libc. The stack frame of the last function called before the signal (which is the location of the fault) is lost.

Code

#ifndef _GNU_SOURCE
#define _GNU_SOURCE
#endif
#ifndef __USE_GNU
#define __USE_GNU
#endif

#include <execinfo.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <ucontext.h>
#include <unistd.h>

/* This structure mirrors the one found in /usr/include/asm/ucontext.h */
typedef struct _sig_ucontext {
 unsigned long     uc_flags;
 struct ucontext   *uc_link;
 stack_t           uc_stack;
 struct sigcontext uc_mcontext;
 sigset_t          uc_sigmask;
} sig_ucontext_t;

void crit_err_hdlr(int sig_num, siginfo_t * info, void * ucontext)
{
 void *             array[50];
 void *             caller_address;
 char **            messages;
 int                size, i;
 sig_ucontext_t *   uc;

 uc = (sig_ucontext_t *)ucontext;

 /* Get the address at the time the signal was raised */
#if defined(__i386__) // gcc specific
 caller_address = (void *) uc->uc_mcontext.eip; // EIP: x86 specific
#elif defined(__x86_64__) // gcc specific
 caller_address = (void *) uc->uc_mcontext.rip; // RIP: x86_64 specific
#else
#error Unsupported architecture. // TODO: Add support for other arch.
#endif

 fprintf(stderr, "signal %d (%s), address is %p from %p\n", 
  sig_num, strsignal(sig_num), info->si_addr, 
  (void *)caller_address);

 size = backtrace(array, 50);

 /* overwrite sigaction with caller's address */
 array[1] = caller_address;

 messages = backtrace_symbols(array, size);

 /* skip first stack frame (points here) */
 for (i = 1; i < size && messages != NULL; ++i)
 {
  fprintf(stderr, "[bt]: (%d) %s\n", i, messages[i]);
 }

 free(messages);

 exit(EXIT_FAILURE);
}

int crash()
{
 char * p = NULL;
 *p = 0;
 return 0;
}

int foo4()
{
 crash();
 return 0;
}

int foo3()
{
 foo4();
 return 0;
}

int foo2()
{
 foo3();
 return 0;
}

int foo1()
{
 foo2();
 return 0;
}

int main(int argc, char ** argv)
{
 struct sigaction sigact;

 sigact.sa_sigaction = crit_err_hdlr;
 sigact.sa_flags = SA_RESTART | SA_SIGINFO;

 if (sigaction(SIGSEGV, &sigact, (struct sigaction *)NULL) != 0)
 {
  fprintf(stderr, "error setting signal handler for %d (%s)\n",
    SIGSEGV, strsignal(SIGSEGV));

  exit(EXIT_FAILURE);
 }

 foo1();

 exit(EXIT_SUCCESS);
}

Output

signal 11 (Segmentation fault), address is (nil) from 0x8c50
[bt]: (1) ./test(crash+0x24) [0x8c50]
[bt]: (2) ./test(foo4+0x10) [0x8c70]
[bt]: (3) ./test(foo3+0x10) [0x8c8c]
[bt]: (4) ./test(foo2+0x10) [0x8ca8]
[bt]: (5) ./test(foo1+0x10) [0x8cc4]
[bt]: (6) ./test(main+0x74) [0x8d44]
[bt]: (7) /lib/libc.so.6(__libc_start_main+0xa8) [0x40032e44]

All the hazards of calling the backtrace() functions in a signal handler still exist and should not be overlooked, but I find the functionality I described here quite helpful in debugging crashes.

It is important to note that the example I provided is developed/tested on Linux for x86. I have also successfully implemented this on ARM using uc_mcontext.arm_pc instead of uc_mcontext.eip.

Here's a link to the article where I learned the details for this implementation: http://www.linuxjournal.com/article/6391

I am working on Linux with the GCC compiler. When my C++ program crashes I would like it to automatically generate a stacktrace.

My program is being run by many different users and it also runs on Linux, Windows and Macintosh (all versions are compiled using gcc).

I would like my program to be able to generate a stack trace when it crashes and the next time the user runs it, it will ask them if it is ok to send the stack trace to me so I can track down the problem. I can handle the sending the info to me but I don't know how to generate the trace string. Any ideas?


*nix: you can intercept SIGSEGV (usualy this signal is raised before crashing) and keep the info into a file. (besides the core file which you can use to debug using gdb for example).

win: Check this from msdn.

You can also look at the google's chrome code to see how it handles crashes. It has a nice exception handling mechanism.


As a Windows-only solution, you can get the equivalent of a stack trace (with much, much more information) using Windows Error Reporting. With just a few registry entries, it can be set up to collect user-mode dumps:

Starting with Windows Server 2008 and Windows Vista with Service Pack 1 (SP1), Windows Error Reporting (WER) can be configured so that full user-mode dumps are collected and stored locally after a user-mode application crashes. [...]

This feature is not enabled by default. Enabling the feature requires administrator privileges. To enable and configure the feature, use the following registry values under the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps key.

You can set the registry entries from your installer, which has the required privileges.

Creating a user-mode dump has the following advantages over generating a stack trace on the client:

  • It's already implemented in the system. You can either use WER as outlined above, or call MiniDumpWriteDump yourself, if you need more fine-grained control over the amount of information to dump. (Make sure to call it from a different process.)
  • Way more complete than a stack trace. Among others it can contain local variables, function arguments, stacks for other threads, loaded modules, and so on. The amount of data (and consequently size) is highly customizable.
  • No need to ship debug symbols. This both drastically decreases the size of your deployment, as well as makes it harder to reverse-engineer your application.
  • Largely independent of the compiler you use. Using WER does not even require any code. Either way, having a way to get a symbol database (PDB) is very useful for offline analysis. I believe GCC can either generate PDB's, or there are tools to convert the symbol database to the PDB format.

Take note, that WER can only be triggered by an application crash (i.e. the system terminating a process due to an unhandled exception). MiniDumpWriteDump can be called at any time. This may be helpful if you need to dump the current state to diagnose issues other than a crash.

Mandatory reading, if you want to evaluate the applicability of mini dumps:


Even though a correct answer has been provided that describes how to use the GNU libc backtrace() function1 and I provided my own answer that describes how to ensure a backtrace from a signal handler points to the actual location of the fault2, I don't see any mention of demangling C++ symbols output from the backtrace.

When obtaining backtraces from a C++ program, the output can be run through c++filt1 to demangle the symbols or by using abi::__cxa_demangle1 directly.

  • 1 Linux & OS X Note that c++filt and __cxa_demangle are GCC specific
  • 2 Linux

The following C++ Linux example uses the same signal handler as my other answer and demonstrates how c++filt can be used to demangle the symbols.

Code:

class foo
{
public:
    foo() { foo1(); }

private:
    void foo1() { foo2(); }
    void foo2() { foo3(); }
    void foo3() { foo4(); }
    void foo4() { crash(); }
    void crash() { char * p = NULL; *p = 0; }
};

int main(int argc, char ** argv)
{
    // Setup signal handler for SIGSEGV
    ...

    foo * f = new foo();
    return 0;
}

Output (./test):

signal 11 (Segmentation fault), address is (nil) from 0x8048e07
[bt]: (1) ./test(crash__3foo+0x13) [0x8048e07]
[bt]: (2) ./test(foo4__3foo+0x12) [0x8048dee]
[bt]: (3) ./test(foo3__3foo+0x12) [0x8048dd6]
[bt]: (4) ./test(foo2__3foo+0x12) [0x8048dbe]
[bt]: (5) ./test(foo1__3foo+0x12) [0x8048da6]
[bt]: (6) ./test(__3foo+0x12) [0x8048d8e]
[bt]: (7) ./test(main+0xe0) [0x8048d18]
[bt]: (8) ./test(__libc_start_main+0x95) [0x42017589]
[bt]: (9) ./test(__register_frame_info+0x3d) [0x8048981]

Demangled Output (./test 2>&1 | c++filt):

signal 11 (Segmentation fault), address is (nil) from 0x8048e07
[bt]: (1) ./test(foo::crash(void)+0x13) [0x8048e07]
[bt]: (2) ./test(foo::foo4(void)+0x12) [0x8048dee]
[bt]: (3) ./test(foo::foo3(void)+0x12) [0x8048dd6]
[bt]: (4) ./test(foo::foo2(void)+0x12) [0x8048dbe]
[bt]: (5) ./test(foo::foo1(void)+0x12) [0x8048da6]
[bt]: (6) ./test(foo::foo(void)+0x12) [0x8048d8e]
[bt]: (7) ./test(main+0xe0) [0x8048d18]
[bt]: (8) ./test(__libc_start_main+0x95) [0x42017589]
[bt]: (9) ./test(__register_frame_info+0x3d) [0x8048981]

The following builds on the signal handler from my original answer and can replace the signal handler in the above example to demonstrate how abi::__cxa_demangle can be used to demangle the symbols. This signal handler produces the same demangled output as the above example.

Code:

void crit_err_hdlr(int sig_num, siginfo_t * info, void * ucontext)
{
    sig_ucontext_t * uc = (sig_ucontext_t *)ucontext;

    void * caller_address = (void *) uc->uc_mcontext.eip; // x86 specific

    std::cerr << "signal " << sig_num 
              << " (" << strsignal(sig_num) << "), address is " 
              << info->si_addr << " from " << caller_address 
              << std::endl << std::endl;

    void * array[50];
    int size = backtrace(array, 50);

    array[1] = caller_address;

    char ** messages = backtrace_symbols(array, size);    

    // skip first stack frame (points here)
    for (int i = 1; i < size && messages != NULL; ++i)
    {
        char *mangled_name = 0, *offset_begin = 0, *offset_end = 0;

        // find parantheses and +address offset surrounding mangled name
        for (char *p = messages[i]; *p; ++p)
        {
            if (*p == '(') 
            {
                mangled_name = p; 
            }
            else if (*p == '+') 
            {
                offset_begin = p;
            }
            else if (*p == ')')
            {
                offset_end = p;
                break;
            }
        }

        // if the line could be processed, attempt to demangle the symbol
        if (mangled_name && offset_begin && offset_end && 
            mangled_name < offset_begin)
        {
            *mangled_name++ = '\0';
            *offset_begin++ = '\0';
            *offset_end++ = '\0';

            int status;
            char * real_name = abi::__cxa_demangle(mangled_name, 0, 0, &status);

            // if demangling is successful, output the demangled function name
            if (status == 0)
            {    
                std::cerr << "[bt]: (" << i << ") " << messages[i] << " : " 
                          << real_name << "+" << offset_begin << offset_end 
                          << std::endl;

            }
            // otherwise, output the mangled function name
            else
            {
                std::cerr << "[bt]: (" << i << ") " << messages[i] << " : " 
                          << mangled_name << "+" << offset_begin << offset_end 
                          << std::endl;
            }
            free(real_name);
        }
        // otherwise, print the whole line
        else
        {
            std::cerr << "[bt]: (" << i << ") " << messages[i] << std::endl;
        }
    }
    std::cerr << std::endl;

    free(messages);

    exit(EXIT_FAILURE);
}

Forget about changing your sources and do some hacks with backtrace() function or macroses - these are just poor solutions.

As a properly working solution, I would advice:

  1. Compile your program with "-g" flag for embedding debug symbols to binary (don't worry this will not impact your performance).
  2. On linux run next command: "ulimit -c unlimited" - to allow system make big crash dumps.
  3. When your program crashed, in the working directory you will see file "core".
  4. Run next command to print backtrace to stdout: gdb -batch -ex "backtrace" ./your_program_exe ./core

This will print proper readable backtrace of your program in human readable way (with source file names and line numbers). Moreover this approach will give you freedom to automatize your system: have a short script that checks if process created a core dump, and then send backtraces by email to developers, or log this into some logging system.


I can help with the Linux version: the function backtrace, backtrace_symbols and backtrace_symbols_fd can be used. See the corresponding manual pages.


I found that @tgamblin solution is not complete. It cannot handle with . I think because by default signal handler is called with the same stack and SIGSEGV is thrown twice. To protect you need register an independent stack for the signal handler.

You can check this with code below. By default the handler fails. With defined macro STACK_OVERFLOW it's all right.

#include <iostream>
#include <execinfo.h>
#include <signal.h>
#include <stdlib.h>
#include <unistd.h>
#include <string>
#include <cassert>

using namespace std;

//#define STACK_OVERFLOW

#ifdef STACK_OVERFLOW
static char stack_body[64*1024];
static stack_t sigseg_stack;
#endif

static struct sigaction sigseg_handler;

void handler(int sig) {
  cerr << "sig seg fault handler" << endl;
  const int asize = 10;
  void *array[asize];
  size_t size;

  // get void*'s for all entries on the stack
  size = backtrace(array, asize);

  // print out all the frames to stderr
  cerr << "stack trace: " << endl;
  backtrace_symbols_fd(array, size, STDERR_FILENO);
  cerr << "resend SIGSEGV to get core dump" << endl;
  signal(sig, SIG_DFL);
  kill(getpid(), sig);
}

void foo() {
  foo();
}

int main(int argc, char **argv) {
#ifdef STACK_OVERFLOW
  sigseg_stack.ss_sp = stack_body;
  sigseg_stack.ss_flags = SS_ONSTACK;
  sigseg_stack.ss_size = sizeof(stack_body);
  assert(!sigaltstack(&sigseg_stack, nullptr));
  sigseg_handler.sa_flags = SA_ONSTACK;
#else
  sigseg_handler.sa_flags = SA_RESTART;  
#endif
  sigseg_handler.sa_handler = &handler;
  assert(!sigaction(SIGSEGV, &sigseg_handler, nullptr));
  cout << "sig action set" << endl;
  foo();
  return 0;
} 

I have seen a lot of answers here performing a signal handler and then exiting. That's the way to go, but remember a very important fact: If you want to get the core dump for the generated error, you can't call exit(status). Call abort() instead!


If your program crashes, it's the operating system itself that generates crash dump information. If you're using a *nix OS, you simply need to not prevent it from doing so (check out the ulimit command's 'coredump' options).


In addition to above answers, here how you make Debian Linux OS generate core dump

  1. Create a “coredumps” folder in the user's home folder
  2. Go to /etc/security/limits.conf. Below the ' ' line, type “ soft core unlimited”, and “root soft core unlimited” if enabling core dumps for root, to allow unlimited space for core dumps.
  3. NOTE: “* soft core unlimited” does not cover root, which is why root has to be specified in its own line.
  4. To check these values, log out, log back in, and type “ulimit -a”. “Core file size” should be set to unlimited.
  5. Check the .bashrc files (user, and root if applicable) to make sure that ulimit is not set there. Otherwise, the value above will be overwritten on startup.
  6. Open /etc/sysctl.conf. Enter the following at the bottom: “kernel.core_pattern = /home//coredumps/%e_%t.dump”. (%e will be the process name, and %t will be the system time)
  7. Exit and type “sysctl -p” to load the new configuration Check /proc/sys/kernel/core_pattern and verify that this matches what you just typed in.
  8. Core dumping can be tested by running a process on the command line (“ &”), and then killing it with “kill -11 ”. If core dumping is successful, you will see “(core dumped)” after the segmentation fault indication.

It's important to note that once you generate a core file you'll need to use the gdb tool to look at it. For gdb to make sense of your core file, you must tell gcc to instrument the binary with debugging symbols: to do this, you compile with the -g flag:

$ g++ -g prog.cpp -o prog

Then, you can either set "ulimit -c unlimited" to let it dump a core, or just run your program inside gdb. I like the second approach more:

$ gdb ./prog
... gdb startup output ...
(gdb) run
... program runs and crashes ...
(gdb) where
... gdb outputs your stack trace ...

I hope this helps.



Might be worth looking at Google Breakpad, a cross-platform crash dump generator and tools to process the dumps.


On Linux/unix/MacOSX use core files (you can enable them with ulimit or compatible system call). On Windows use Microsoft error reporting (you can become a partner and get access to your application crash data).



Thank you to enthusiasticgeek for drawing my attention to the addr2line utility.

I've written a quick and dirty script to process the output of the answer provided here: (much thanks to jschmier!) using the addr2line utility.

The script accepts a single argument: The name of the file containing the output from jschmier's utility.

The output should print something like the following for each level of the trace:

BACKTRACE:  testExe 0x8A5db6b
FILE:       pathToFile/testExe.C:110
FUNCTION:   testFunction(int) 
   107  
   108           
   109           int* i = 0x0;
  *110           *i = 5;
   111      
   112        }
   113        return i;

Code:

#!/bin/bash

LOGFILE=$1

NUM_SRC_CONTEXT_LINES=3

old_IFS=$IFS  # save the field separator           
IFS=$'\n'     # new field separator, the end of line           

for bt in `cat $LOGFILE | grep '\[bt\]'`; do
   IFS=$old_IFS     # restore default field separator 
   printf '\n'
   EXEC=`echo $bt | cut -d' ' -f3 | cut -d'(' -f1`  
   ADDR=`echo $bt | cut -d'[' -f3 | cut -d']' -f1`
   echo "BACKTRACE:  $EXEC $ADDR"
   A2L=`addr2line -a $ADDR -e $EXEC -pfC`
   #echo "A2L:        $A2L"

   FUNCTION=`echo $A2L | sed 's/\<at\>.*//' | cut -d' ' -f2-99`
   FILE_AND_LINE=`echo $A2L | sed 's/.* at //'`
   echo "FILE:       $FILE_AND_LINE"
   echo "FUNCTION:   $FUNCTION"

   # print offending source code
   SRCFILE=`echo $FILE_AND_LINE | cut -d':' -f1`
   LINENUM=`echo $FILE_AND_LINE | cut -d':' -f2`
   if ([ -f $SRCFILE ]); then
      cat -n $SRCFILE | grep -C $NUM_SRC_CONTEXT_LINES "^ *$LINENUM\>" | sed "s/ $LINENUM/*$LINENUM/"
   else
      echo "File not found: $SRCFILE"
   fi
   IFS=$'\n'     # new field separator, the end of line           
done

IFS=$old_IFS     # restore default field separator 

You can use DeathHandler - small C++ class which does everything for you, reliable.


You did not specify your operating system, so this is difficult to answer. If you are using a system based on gnu libc, you might be able to use the libc function backtrace().

GCC also has two builtins that can assist you, but which may or may not be implemented fully on your architecture, and those are __builtin_frame_address and __builtin_return_address. Both of which want an immediate integer level (by immediate, I mean it can't be a variable). If __builtin_frame_address for a given level is non-zero, it should be safe to grab the return address of the same level.


ulimit -c unlimited

is a system variable, wich will allow to create a core dump after your application crashes. In this case an unlimited amount. Look for a file called core in the very same directory. Make sure you compiled your code with debugging informations enabled!

regards







assert