Debugging Linux processes in an embedded system using GDB and remote GDBServer stub

In today's embedded world, developers often come across a situation where they need to debug an embedded application which will be running on Linux/Android or other such embedded operating systems.

Mostly, to ease the debugging effort, an x86 port of the application will be developed at first. Initial developer testing will be done in the development machine itself so that native debugging tools can be used. Entry level bugs (which may constitute general typos and coding standard issues), of course, can be detected and fixed like this. 

However, there are lots of other issues which can arise only when the application is run in the target. Some of the issues may be due to 

• cross compiling of the application 
• resource constraints (memory, CPU)
• timing constraints
• differences in the platform-dependent behavior of Linux.

Such issues will make the developers say 'It is working perfectly in my machine; but when run in the target, it is behaving weirdly'.

In such cases, it will be best to debug the application right in the environment where it is executing.

GDB (GNU Debugger) helps a lot in this regard. It has got an extensive and robust support for remote debugging as well as support for an exhaustive range of processor architectures (like ARC, ARM, MIPS, PowerPC etc). Remote debugging is the mechanism to debug an application running remotely in an embedded target or another machine using a GDB running in a host machine, say an x86 PC.

Setting up a remote-debug environment

It is really easy to setup one remote debugging environment. Once you read this article completely, you will think "Why the hell I hadn't done this before???".

To set up a remote debugging environment, the following components are required:

• A GDB cross-compiled for the required target architecture which can run in your host machine.
• A GDBServer cross-comiletd for the required target architecture which can run in your target system.
• A communication medium - either a serial cable or an ethernet cable through which the host machine can communicate with the remote target.

NOTE: How to cross compile GDBServer, GDB and your application for a specific platform is out of the scope of current discussion and shall be discussed separately.

Usually, developers will have to handle two sorts of situations

1. Debug a process from its start-up.
2. Debug a process which is already running in target.

For both these cases, things to do in the host GDB are same.

Step 1: Run GDB in the host machine.

Step 2: Run GDBServer in the remote target. This can be done through a remote shell via teraterm, hyper-terminal, minicom or through SSH if SSH is running in the target.

GDBServer can be invoked in 3 different modes:

1. gdbserver :<port number> <application name>
2. gdbserver :<port number> --attach <Process ID>
3. gdbserver :<port number> --multi

First method is used to debug a process from start-up, second method is used to debug an already running application and the third one is used to debug multiple processes.

Step 3: In host GDB's console, give target remote <ip>:<port number>. This will connect host GDB with the remote GDBServer listening at <port number> with ip-address <ip>.

Step 4: That's all folks. Now you can debug the application running in a remote target as anything which was running in your PC. You can step in, step over, put breakpoints, watch variables etc as you normally do in GDB.

Try this and make your developer's life better and easier. Enjoy!!!

More internal details of the above mentioned steps will follow soon... Before diving deeper, you better try this out and get your hands dirty...:).