Version ======= This version of the gdbstub package was developed and tested on kernel version 2.3.48. It will not install on a 2.2 kernel. It may not work on earlier versions of 2.3 kernels. It is possible that it will continue to work on later versions of 2.3 and then versions of 2.4 (I hope). Debugging Setup =============== Designate one machine as the "development" machine. This is the machine on which you run your compiles and which has your source code for the kernel. Designate a second machine as the "target" machine. This is the machine that will run your experimental kernel. The two machines will be connected together via a serial line out one or the other of the COM ports of the PC. You will need a modem eliminator and the appropriate cables. On the DEVELOPMENT machine you need to apply the patch for the gdb hooks. You have probably already done that if you are reading this file. On your DEVELOPMENT machine, go to your kernel source directory and do "make menuconfig". Go down to the kernel hacking menu item and open it up. Enable the kernel gdb stub code by selecting that item. Save and exit the menuconfig program. Then do "make clean" and "make bzImage" (or whatever target you want to make). This gets the kernel compiled with the "-g" option set -- necessary for debugging. You have just built the kernel on your DEVELOPMENT machine that you intend to run on our TARGET machine. To install this new kernel, use the following installation procedure. Remember, you are on the DEVELOPMENT machine patching the kernel source for the kernel that you intend to run on the TARGET machine. Copy this kernel to your target machine using your usual procedures. I usually arrange to copy development:/usr/src/linux/arch/i386/boot/zImage to /vmlinuz on the TARGET machine via a LAN based NFS access. That is, I run the cp command on the target and copy from the development machine via the LAN. Run Lilo on the new kernel on the target machine so that it will boot! Then boot the kernel on the target machine. There is an utility program named "gdbstart" in the development:/usr/src/linux/arch/i386/kernel directory. You should copy this program over to your target machine, probably into /sbin. This utility program is run on the target machine to activate the kernel hooks for the debugger. It is invoked as follows: gdbstart [-s speed] [-t tty-dev] defaults: /dev/ttyS0 with speed unmodified by gdbstart Don't run the program just yet. We'll get to that in a bit. Decide on which tty port you want the machines to communicate, then cable them up back-to-back using the null modem. COM1 is /dev/ttyS0 and COM2 is /dev/ttyS1. On the DEVELOPMENT machine, create a file called .gdbinit in the directory /usr/src/linux. An example .gdbinit file looks like this: define rmt set remotebaud 38400 target remote /dev/ttyS0 end Assuming that you added my gdbinit stuff to your .gdbinit, edit .gdbinit and find the section that looks like this: define rmt set remotebaud 38400 target remote /dev/ttyS0 end Change the "target" definition so that it specifies the tty port that you intend to use. Change the "remotebaud" definition to match the data rate that you are going to use for the com line. On the TARGET machine I find it helpful to create shell script file named "debug" in the root home directory with the following contents: gdbstart -s 38400 -t /dev/ttyS0 < EOF This runs the gdbstart program and gives it the carriage return that it prompts for. This sets the data rate from the target machine's side. You are now ready to try it out. On your TARGET machine, freshly rebooted with your gdbstub-equipped kernel, type "debug" in the root home directory. The system will appear to hang with some messages on the screen from the debug stub. What it is doing is waiting for contact from the development machine. On your DEVELOPMENT machine, cd /usr/src/linux and enter "gdb vmlinux". When gdb gets the symbols loaded and prompts you, enter "rmt" (that's the macro from the .gdbinit file that you just edited). If everything is working correctly you should see gdb print out a few lines indicating that a breakpoint has been taken. It will actually show a line of code in the target kernel inside the gdbstub activation code. The gdb interaction should look something like this: linux-dev:/usr/src/linux# gdb vmlinux GDB is free software and you are welcome to distribute copies of it under certain conditions; type "show copying" to see the conditions. There is absolutely no warranty for GDB; type "show warranty" for details. GDB 4.15.1 (i486-slackware-linux), Copyright 1995 Free Software Foundation, Inc... (gdb) rmt breakpoint () at i386-stub.c:750 750 } (gdb) You can now use whatever gdb commands you like to set breakpoints. Enter "continue" to start your target machine executing again. At this point the target system will run at full speed until it encounters your breakpoint or gets a segment violation in the kernel, or whatever. Triggering gdbstub at Kernel Boot Time ====================================== The gdbstub patch now has the ability for gdb to connect to the kernel during bootup (as opposed to waiting for the system to come all the way up and then running the gdbstart program on the target machine). This new functionality was added by Scott Foehner at SGI. To force a kernel that has been compiled with gdbstub to pause during the boot process and wait for a connection from gdb, the paramter "gdb" should be passed to the kernel. This can be done by typing "gdb" after the name of the kernel on the LILO command line. The patch defaults to use ttyS1 at a baud rate of 38400. These parameters can be changed by using "gdbttyS=" and "gdbbaud=" on the command line. Example: LILO boot: linux gdb gdbttyS=1 gdbbaud=38400 Note that this command is entered on the TARGET machine as it is booting the kernel that was compiled on the DEVELOPMENT machine. An alternate approach is to place a line in the /etc/lilo.conf file on your TARGET machine. Under the heading for the kernel that you intend to boot, place a line that looks like this: append = "gdb gdbttyS=1 gdbbaud=38400" This will cause the kernel to enter the gdbstub automatically at boot time. BE SURE to run "lilo" after changing the /etc/lilo.conf file. The "gdbstart" Program ===================== This utility program is used to set up the com port and data rate for the connection from the target system to the development system. Its usage has been described above. This version of the patch uses the same tty ioctl for kernel versions 2.0.30 onwards. Thus, the gdbstart utility does not need to be re-compiled to install the patch in a later version of the kernel. The ioctl added to the kernel for this purpose is far enough "off the end" of existing ioctls (as of 2.1.120) that it should not interfere with any new kernel tty ioctls for quite some time (famous last words). The source for the gdbstart program resides in the arch/i386/kernel directory. Debugging hints =============== You can break into the target machine at any time from the development machine by typing ^C. If the target machine has interrupts enabled this will stop it in the kernel and enter the debugger. There is unfortunately no way of breaking into the kernel if it is in a loop with interrupts disabled, so if this happens to you then you need to place exploratory breakpoints or printk's into the kernel to find out where it is looping. There is a copy of an e-mail in the kgdb distribution directory which describes how to create an NMI on an ISA bus machine using a paper clip. I have a sophisticated version of this made by wiring a push button switch into a PC104/ISA bus adapter card. The adapter card nicely furnishes wire wrap pins for all the ISA bus signals. When you are done debugging the kernel on the target machine it is a good idea to leave it in a running state. This makes reboots faster, bypassing the fsck. So do a gdb "continue" as the last gdb command if this is possible. To terminate gdb itself on the development machine and leave the target machine running, type ^Z to suspend gdb and then kill it with "kill %1" or something similar. If gdbstub Does Not Work ======================== If it doesn't work, you will have to troubleshoot it. Do the easy things first like double checking your cabling and data rates. You might try some non-kernel based programs to see if the back-to-back connection works properly. Just something simple like cat /etc/hosts >/dev/ttyS0 on one machine and cat /dev/ttyS0 on the other will tell you if you can send data from one machine to the other. There is no point in tearing out your hair in the kernel if the line doesn't work. All of the real action takes place in the file /usr/src/linux/arch/i386/kernel/gdbstub.c. That is the code on the target machine that interacts with gdb on the development machine. In gdb you can turn on a debug switch with the following command: set remotedebug This will print out the protocol messages that gdb is exchanging with the target machine. Another place to look is /usr/src/linux/drivers/char/gdbserial.c That is the code that talks to the serial port on the target side. There might be a problem there. If you are really desperate you can use printk debugging in the gdbstub code in the target kernel until you get it working. In particular, there is a global variable in /usr/src/linux/arch/i386/kernel/gdbstub.c named "remote_debug". Compile your kernel with this set to 1, rather than 0 and the debug stub will print out lots of stuff as it does what it does. Debugging Loadable Modules ========================== This technique comes courtesy of Edouard Parmelan When you run gdb, enter the command source gdbinit-modules This will read in a file of gdb macros that was installed in your kernel source directory with kgdb was installed. This file implements the following commands: mod-list Lists the loaded modules in the form mod-print-symbols Prints all the symbols in the indicated module. mod-add-symbols Loads the symbols from the object file and associates them with the indicated module. After you have loaded the module that you want to debug, use the command mod-list to find the of your module. Then use that address in the mod-add-symbols command to load your module's symbols. From that point onward you can debug your module as if it were a part of the kernel. The file gdbinit-modules also contains a command named mod-add-lis as an example of how to construct a command of your own to load your favorite module. The idea is to "can" the pathname of the module in the command so you don't have to type so much. Threads ======= Each process in a target machine is seen as a gdb thread. gdb thread related commands (info threads, thread n) can be used. ia-32 hardware breakpoints ========================== gdb stub contains support for hardware breakpoints using debugging features of ia-32(x86) processors. These breakpoints do not need code modification. They use debugging registers. 4 hardware breakpoints are available in ia-32 processors. Each hardware breakpoint can be of one of the following three types. 1. Execution breakpoint - An Execution breakpoint is triggered when code at the breakpoint address is executed. As limited number of hardware breakpoints are available, it is advisable to use software breakpoints ( break command ) instead of execution hardware breakpoints, unless modification of code is to be avoided. 2. Write breakpoint - A write breakpoint is triggered when memory location at the breakpoint address is written. A write or can be placed for data of variable length. Length of a write breakpoint indicates length of the datatype to be watched. Length is 1 for 1 byte data , 2 for 2 byte data, 3 for 4 byte data. 3. Access breakpoint - An access breakpoint is triggered when memory location at the breakpoint address is either read or written. Access breakpoints also have lengths similar to write breakpoints. IO breakpoints in ia-32 are not supported. Since gdb stub at present does not use the protocol used by gdb for hardware breakpoints, hardware breakpoints are accessed through gdb macros. gdb macros for hardware breakpoints are described below. hwebrk - Places an execution breakpoint hwebrk breakpointno address hwwbrk - Places a write breakpoint hwwbrk breakpointno length address hwabrk - Places an access breakpoint hwabrk breakpointno length address hwrmbrk - Removes a breakpoint hwrmbrk breakpointno exinfo - Tells whether a software or hardware breakpoint has occured. Prints number of the hardware breakpoint if a hardware breakpoint has occured. Arguments required by these commands are as follows breakpointno - 0 to 3 length - 1 to 3 address - Memory location in hex digits ( without 0x ) e.g c015e9bc MP support ========== When a breakpoint occurs or user issues a break ( Ctrl + C ) to gdb client, all the processors are forced to enter the debugger. Current thread corresponds to the thread running on the processor where breakpoint occured. Threads running on other processor(s) appear similar to other non running threads in the 'info threads' output. ia-32 hardware debugging registers on all processors are set to same values. Hence any hardware breakpoints may occur on any processor. gdb troubleshooting =================== 1. gdb hangs Kill it. restart gdb. Connect to target machine. 2. gdb cannot connect to target machine (after killing a gdb and restarting another) If the target machine was not inside debugger when you killed gdb, gdb cannot connect because the target machine won't respond. In this case echo "Ctrl+C"(ascii 3) in the serial line. e.g. echo -e "\003" > /dev/ttyS1 This forces that target machine into debugger after which you can connect. 3. gdb cannot connect even after echoing Ctrl+C into serial line Try changing serial line settings min to 1 and time to 0 e.g. stty min 1 time 0 < /dev/ttyS1 Try echoing again check serial line speed and set it to correct value if required e.g. stty ispeed 115200 ospeed 115200 < /dev/ttyS1 Final Items =========== I picked up this code from Dave Grothe and enhanced it. If you make some really cool modification to this stuff, or if you fix a bug, please let me know. Amit S. Kale (First kgdb by David Grothe ) (modified by Tigran Aivazian ) Putting gdbstub into the kernel config menu. (modified by Scott Foehner ) Hooks for entering gdbstub at boot time. (modified by Amit S. Kale ) Threads, ia-32 hw debugging, mp support, console support, nmi watchdog handling.