Home › Development › Debugging › Debugging the kernel over JTAG

Debugging kernel modules

The JTAG kernel debugger also debugs all loaded kernel modules. When the Linux Awareness Layer has been loaded, kernel modules are handled using GDB's shared library debug mechanism. Kernel modules are analogous to userspace shared libraries.

To debug kernel modules, GDB must be able to access their DWARF information. This means they must be built with debugging information, and that GDB must be told the location of the .ko files.

Kernel modules are built using the kernel build system, which dictates their compile settings. When the kernel has been built with debug info, rebuild the kernel modules so they inherit the following configuration option:

Kernel hacking  --->
    Compile the kernel with debug info

Note: like the kernel itself, kernel modules cannot be built without optimization. There is some advice on debugging optimized code here.

To confirm that a binary has DWARF information, you can use the following command: 

% sh4-linux-readelf -S <binary> | grep debug
  [21] .debug_aranges    PROGBITS        00000000 30d910 005b90 00      0   0  1
  [22] .debug_pubnames   PROGBITS        00000000 3134a0 01caf9 00      0   0  1
  [23] .debug_info       PROGBITS        00000000 32ff99 12ac333 00      0   0  1
  [24] .debug_abbrev     PROGBITS        00000000 15dc2cc 0a89c4 00      0   0  1
  [25] .debug_line       PROGBITS        00000000 1684c90 14f8da 00      0   0  1
  [26] .debug_frame      PROGBITS        00000000 17d456c 057b9c 00      0   0  4
  [27] .debug_str        PROGBITS        00000000 182c108 0783ed 01  MS  0   0  1
  [28] .debug_loc        PROGBITS        00000000 18a44f5 2eb222 00      0   0  1
  [29] .debug_ranges     PROGBITS        00000000 1b8f717 098c88 00      0   0  1

If the .debug sections are present, the binary contains debugging information.

To debug the kernel modules, the GDB internal variable module-search-path (an alias of solib-search-path) must be set to a colon-separated list of directories containing .ko files.

Note: This is not recursive, so each directory must be added individually. 

(gdb) set module-search-path /tmp/smithc/some_modules:/tmp/smithc/other_modules
(gdb) show module-search-path
The search path for loading non-absolute shared library symbol files is /tmp/smithc/some_modules:/tmp/smithc/other_modules

The copies of the .ko files pointed to by module-search-path need not be the exact copies visible to the target, if, for example, the target filesystem is in Flash. They must be exactly the same as those binaries, however.

When a module is loaded, GDB displays a message like this: 

[New module 'stsys_ioctl' (/tmp/smithc/some_modules/stsys_ioctl.ko)]

This indicates that GDB has successfully found and read the DWARF info for the newly loaded module. If the module-search-path is not set correctly, GDB displays a message like this: 

Error while mapping shared library sections:
You just loaded the 'bad_module' module.
The debugger searched for 'bad_module.ko' in module-search-path,
but didn't find the module file.
Current module-search-path is:
/tmp/smithc/some_modules:/tmp/smithc/other_modules

To list the information about the currently loaded modules, use the command info modules (an alias for info sharedlibrary). 

(gdb) info modules
From        To          Syms Read   Shared Object Library
0xc0189000  0xc018a494  Yes         /export/rootfs/smithc-stfae/root/modules/stsys_ioctl.ko
0xc015d000  0xc015edc0  Yes         /export/rootfs/smithc-stfae/root/modules/stos_core.ko
0xc0165000  0xc01678c8  Yes         /export/rootfs/smithc-stfae/root/modules/stcommon_core.ko
0xc016c000  0xc0171078  Yes         /export/rootfs/smithc-stfae/root/modules/stavmem_core.ko
0xc0176000  0xc0179770  Yes         /export/rootfs/smithc-stfae/root/modules/stevt_core.ko
0xc017d000  0xc017db6c  Yes         /export/rootfs/smithc-stfae/root/modules/sti2c_core.ko
0xc0180000  0xc0183144  Yes         /export/rootfs/smithc-stfae/root/modules/stmerge_core.ko
0xc03cc000  0xc03f76c0  Yes         /export/rootfs/smithc-stfae/root/modules/stpti4_core.ko
0xc018d000  0xc01e7d40  Yes         /export/rootfs/smithc-stfae/root/modules/sttuner_core.ko
0xc0268000  0xc026b8a0  Yes         /export/rootfs/smithc-stfae/root/modules/stdenc_core.ko
0xc020f000  0xc021679c  Yes         /export/rootfs/smithc-stfae/root/modules/stclkrv_core.ko
0xc021b000  0xc021fc14  Yes         /export/rootfs/smithc-stfae/root/modules/stvtg_core.ko
0xc0186000  0xc018666c  Yes         /export/rootfs/smithc-stfae/root/modules/hdmi_i2c_core.ko
0xc0224000  0xc0236fc0  Yes         /export/rootfs/smithc-stfae/root/modules/stvout_core.ko
0xc023d000  0xc02445b4  Yes         /export/rootfs/smithc-stfae/root/modules/sthdmi_core.ko
0xc0248000  0xc025db30  Yes         /export/rootfs/smithc-stfae/root/modules/stlayer_core.ko
0xc02ba000  0xc02be66c  Yes         /export/rootfs/smithc-stfae/root/modules/stvmix_core.ko
0xc026f000  0xc029fa48  Yes         /export/rootfs/smithc-stfae/root/modules/stblit_core.ko
0xc0360000  0xc0367fe4  Yes         /export/rootfs/smithc-stfae/root/modules/stfdma_core.ko
0xc02d2000  0xc02d6560  Yes         /export/rootfs/smithc-stfae/root/modules/embxshell.ko
0xc02b3000  0xc02b43c0  Yes         /export/rootfs/smithc-stfae/root/modules/embxloopback.ko
0xc02ce000  0xc02cf700  Yes         /export/rootfs/smithc-stfae/root/modules/embxmailbox.ko
0xc02c2000  0xc02c6fa0  Yes         /export/rootfs/smithc-stfae/root/modules/embxshm.ko
0xc02ed000  0xc02f2680  Yes         /export/rootfs/smithc-stfae/root/modules/mme_host.ko
0xc02b7000  0xc02b8560  Yes         /export/rootfs/smithc-stfae/root/modules/rpc_userver.ko
0xc02f6000  0xc02fb5e0  Yes         /export/rootfs/smithc-stfae/root/modules/embxshmc.ko
0xc085a000  0xc08c7370  Yes         /export/rootfs/smithc-stfae/root/modules/stvid_core.ko
0xc02ff000  0xc032bc80  Yes         /export/rootfs/smithc-stfae/root/modules/staudlx_core.ko
0xc02db000  0xc02ded94  Yes         /export/rootfs/smithc-stfae/root/modules/stevt_ioctl.ko
0xc02e2000  0xc02e3780  Yes         /export/rootfs/smithc-stfae/root/modules/stavmem_ioctl.ko
0xc02e6000  0xc02e7ce0  Yes         /export/rootfs/smithc-stfae/root/modules/stmerge_ioctl.ko
0xc0372000  0xc0381a5c  Yes         /export/rootfs/smithc-stfae/root/modules/stpti4_ioctl.ko
0xc033c000  0xc033e444  Yes         /export/rootfs/smithc-stfae/root/modules/stclkrv_ioctl.ko
0xc02cb000  0xc02cbb7c  Yes         /export/rootfs/smithc-stfae/root/modules/stfdma_ioctl.ko
0xc0341000  0xc0342ce8  Yes         /export/rootfs/smithc-stfae/root/modules/sti2c_ioctl.ko
0xc0345000  0xc034871c  Yes         /export/rootfs/smithc-stfae/root/modules/stdenc_ioctl.ko
0xc034b000  0xc034ca84  Yes         /export/rootfs/smithc-stfae/root/modules/stvtg_ioctl.ko
0xc034f000  0xc0352fe0  Yes         /export/rootfs/smithc-stfae/root/modules/stvout_ioctl.ko
0xc0356000  0xc035ccc8  Yes         /export/rootfs/smithc-stfae/root/modules/stlayer_ioctl.ko
0xc03a4000  0xc03a7b50  Yes         /export/rootfs/smithc-stfae/root/modules/stvmix_ioctl.ko
0xc03aa000  0xc03b0232  Yes         /export/rootfs/smithc-stfae/root/modules/stblit_ioctl.ko
0xc03b3000  0xc03bb100  Yes         /export/rootfs/smithc-stfae/root/modules/stvid_ioctl.ko
0xc0482000  0xc0492adc  Yes         /export/rootfs/smithc-stfae/root/modules/staudlx_ioctl.ko
0xc03be000  0xc03c12d8  Yes         /export/rootfs/smithc-stfae/root/modules/sthdmi_ioctl.ko
0xc0471000  0xc04772cc  Yes         /export/rootfs/smithc-stfae/root/modules/sttuner_ioctl.ko
0xc02ea000  0xc02ea9c0  Yes         /export/rootfs/smithc-stfae/root/modules/stfastfilter_ioctl.ko

When the kernel has been loaded and its debug info has been read, it can be transparently debugged the same as any other part of the kernel.

Another useful GDB concept when debugging kernel modules is the "pending breakpoint". This is a breakpoint that is set on a function that is not currently present, which is then resolved when a shared library (or in our case, a kernel module) is loaded that contains that function.

An example of using a pending breakpoint to stop on a module initialization function (executed when the module is first loaded): 

(gdb) set solib-search-path /scratch/smithc/new_module
(gdb) break new_init
Function "new_init" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 6 (new_init) pending.
(gdb) info breakpoints
<snip>
6   breakpoint     keep y   <PENDING>  new_init
<snip>
(gdb) continue
Continuing.
[New module 'new_module' (/scratch/smithc/new_module/new_module.ko)]
Breakpoint 7 at 0xc016c020: file /scratch/smithc/new_module/new_module.c, line 6.
Pending breakpoint "new_init" resolved
[Switching to insmod]
 
Breakpoint 7, new_init () at /scratch/smithc/new_module/new_module.c:6
6               printk("new_init %d\n", *ptr);
(gdb)