27: Using a Debugger

This quick little exercise will give you a crash course in running an old school "shell style" debugger. When you use the GNU C++ compiler the debugger is called gdb, which should be the default on Windows and most Linux systems. On macOS you'll have to use the lldb command because Apple uses clang for their compiler.

A debugger allows you to step through your code and inspect it while it's running. I tend to only use them when I really can't get some kind of logging out of my programs. For example, if it's on a server that's crashing or if the crash is so hard I can't even log to find it.

Learning how to use a debugger will help you when you're very stuck finding a bug, but you can also use them to walk through your code and observe how it works.

Windows

On Windows you should have the gdb command if you followed my setup instructions. You can test it in the same way as described below in Ubuntu since it's the same gdb command.

Ubuntu

You already have the command gdb so try it:

gdb 
# lots of text
(gdb) quit

Here I run gdb, it prints a bunch of information, and then prompts with (gdb). You can quit gdb with the command quit.

macOS

The debugger on macOS is called lldb and it mostly works the same as gdb but has slightly different commands. You can test your lldb with this:

lldb
(lldb) quit

You won't see any extra text, then you get a similar (lldb) prompt and can type the same quit command to quit.

Our Buggy Example

For this exercise I'll be using this poorly written piece of code:

View Source file ex27_crasher.cpp Only

#include <fmt/core.h>
#include <vector>

void print_number_at(const std::vector<int>& numbers, size_t at) {
  fmt::println("number: {}", numbers[at]);
}

void print_numbers(const std::vector<int>& numbers) {
  for(size_t i = 0; i != numbers.size(); i+=3) {
    print_number_at(numbers, i);
  }
}

int main() {
  std::vector numbers{1,2,3,4};
  print_numbers(numbers);
}

Type this up and add it to your meson.build so we can debug it.

Running GDB

You can debug this program by running gdb followed by the executable like this:

gdb ./builddir/ex27_crasher

On macOS you want to run lldb:

lldb ./builddir/ex27_crasher

Various Annoyances

There are a few annoyances to deal with on different OS:

Windows

On Windows you might have to add the .exe for gdb to load the file.

Linux

Most Linux distros seem to want to track you by claiming to download debug symbols from some centralized server. The message is something like, "This GDB supports auto-downloading debuginfo from the following URLs." It will do this for every single run. It never remembers the symbols, always downloads it over and over, never caches, and will even do it with executables that are obviously not in their database (like your code you just wrote). You can safely say "n" (no) here.

macOS

On macOS you will have to use lldb which is similar, but the commands are slightly different. I'll tell you the commands for both while we do this, and at the end I'll have a little cheat sheet for both lldb and gdb.

Running Your Executable

The first thing we want to do is run the command and let it crash. In both gdb and lldb you do that like this:

run

When you run this with gdb you should see something like this:

Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=6, no_tid=0)
    at ./nptl/pthread_kill.c:44
warning: 44     ./nptl/pthread_kill.c: No such file or directory

When you run this with lldb you should see something like this:

* thread #1, name = 'ex27_crasher', stop reason = signal SIGABRT
    frame #0: libc.so.6`__pthread_kill_implementation(threadid=<unavailable>, signo=6, no_tid=0) at pthread_kill.c:44:76

The important part in both is the SIGABRT which means your program was killed by an abort signal because you did something wrong. In this case it's because you accessed the numbers vector past its end.

Getting a Stack Trace

In both lldb and gdb you can get a backtrace with the bt command. Right after a crash type:

bt

And it will dump a trace showing how the code reached the point of the crash, and where the crash happened. Here's the relevant snippet from gdb:

#6  0xXXX in std::vector<int, std::allocator<int> >::operator[]
    (this=0x7fffffffd7b0, __n=6)
    at /usr/include/c++/15/bits/stl_vector.h:1282
#7  0xXXX in print_number_at (
    numbers=std::vector of length 4, capacity 4 = {...}, at=6)
    at ../ex27_crasher.cpp:5
#8  0xXXX in print_numbers (
    numbers=std::vector of length 4, capacity 4 = {...})
    at ../ex27_crasher.cpp:10
#9  0xXXX in main () at ../ex27_crasher.cpp:16

The important part is the files listed with their line numbers. I've removed most of this junk leaving only the files so you can see what I mean:

#6  /usr/include/c++/15/bits/stl_vector.h:1282
#7  in print_number_at (
    at ../ex27_crasher.cpp:5
#8  in print_numbers (
    at ../ex27_crasher.cpp:10
#9  in main () at ../ex27_crasher.cpp:16

Above this you are getting a trace in the libraries that come with your compiler and OS, which can still be useful but not in this case. There's also a lot of other information that's useful, but at first just finding the source location is the most useful.

The lldb version is similar:

frame #7: ex27_crasher`print_number_at
    (numbers=size=4, at=6) at ex27_crasher.cpp:5:40
frame #8: ex27_crasher`print_numbers
    (numbers=size=4) at ex27_crasher.cpp:10:20
frame #9: ex27_crasher`main at ex27_crasher.cpp:16:16

This is stripped down in the same way, but the lldb version includes the function parameters (numbers=size=4, at=6) which is very useful.

Reading the Trace

You read the trace "backwards", from the bottom to the top like this:

1. It started at main in file ex27_crasher.cpp at line 16.
2. That then went to print_numbers at line 10.
3. Then to print_number_at on line 5 where it finally crashed.

It helps to have your code open while you look at this so you can figure out where it's going and look for clues about the crash.

Break Points

If you know your program crashes at a certain spot--or you suspect the bug is there--then you can set a break point at that line. Let's say you want to see the line that has the crash:

(gdb) break ex27_crasher.cpp:5
(gdb) run

When your program starts again it will run to that location and you can inspect the variables there.

Printing and Continuing

To inspect the variables use the print command. Continuing from where you did a break on line 5 you can print the at variable:

Breakpoint 1, print_number_at (
    numbers=std::vector of length 4, capacity 4 = {...}, at=0)
    at ../ex27_crasher.cpp:5
5         fmt::println("number: {}", numbers[at]);
(gdb) print at
$1 = 0

The $1 = 0 is gdb's weird way of printing the variable. You can then use cont to do another loop through to this spot, use print again to see at, and then you'll eventually hit the crash. It's because at==6 at that point but there's only 4 elements in the numbers vector.

Listing the Source

You can see where the program is in the source by using the list command:

Breakpoint 1, print_number_at (
    numbers=std::vector of length 4, capacity 4 = {...}, at=0)
    at ../ex27_crasher.cpp:5
5         fmt::println("number: {}", numbers[at]);
(gdb) list
1       #include <fmt/core.h>
2       #include <vector>
3
4       void print_number_at(const std::vector<int>& numbers, size_t at) {
5         fmt::println("number: {}", numbers[at]);
6       }
7
8       void print_numbers(const std::vector<int>& numbers) {
9         for(size_t i = 0; i != numbers.size(); i+=3) {
10          print_number_at(numbers, i);

You can see I just use list and it print out the snippet of code showing me where I am. It doesn't really show you, but you know it's line 5 and there's line numbers on the left.

Watching Variables

You can also watch a variable change, but I find these to be mostly useless. They're very unreliable because you have to set them in specific locations and gdb doesn't show them reliably enough. You'll set it, cont and then see no output, then a few iterations later see the value change. Other times you'll set it, cont then gdb complains with:

Watchpoint 8 deleted because the program has left the block in
which its expression is valid.

Not only is this information pointless, but it's also not helping you watch that variable change. Finally, watchpoints are erased each time you run the program forcing you to set them again on every run. For all these reasons I don't bother with watchpoints and simply print variables.

Better than Print Debugging?

It's things like this that make me simply avoid debuggers most of the time. Not being able to save sessions, having to recreate breakpoints and watchpoints every single time I run, and not getting a log of changes to analyze after a crash are all reasons I find debuggers to be significantly lacking

With print debugging my "watchpoints" stay until I delete the code that prints them. I don't need special software that's annoying to run. I don't have to remember what I was doing each time. I can get a full log of everything I need to find the bug. It works all the time, on any machine, in any language, in any situation, and is easy to use.

The only time I'll use a debugger is when I need a stack trace, or for situations where I can't figure out why something is crashing to even get a log of it. An example is if a server is running in production and crashing randomly (it's always a memory error). In that case I'll attach to the server with a debugger remotely, wait for it to crash, and get the stack trace and other information.

Visual Debuggers

I did a whole review of visual debuggers and couldn't find one that was easy to use and worked well on Linux and Windows. On Windows the RadDebugger is touted as excellent, but it wouldn't work with binaries created by GCC. On Linux I tried many, but the only one that was kind of useable was gf. There was also nnd which was promising for Linux, but the UI was a little clunky.

gdb Cheat Sheet

lldb Cheat Sheet