Category Archives: Software

Intercept function calls of running program with GDB

I was recently debugging an issue at work relating to VRF where I found that getaddrinfo was not using the VRF interface and therefore DNS would not work if it needed to go over that interface. I found out that in the 4.10 kernel and later, the ip command allows you to run a program with `ip vrf exec ` that would presumably tell every socket created as a result of that command to use the specific VRF interface. Unfortunately our platform is currently on 4.9 kernel.

The trick is, when you open a socket(), you need to call something like:

setsockopt(, SOL_SOCKET, SO_BINDTODEVICE, “vrfname”, )

Which will send the packet over the vrf. Knowing roughly this must be what the ip vrf exec must be doing, I wondered if I could hack something up to do this myself? My first though was actually to replace getaddrinfo() with myown implementation and load with LD_PRELOAD. I experimented with this but found that it would be too difficult, since I would need to copy a bunch of glibc (where getaddrinfo is) code, and also it would not solve the problem in the general sense, if other programs don’t know about VRF. While debugging with GDB I had the idea of creating a GDB script that could call the above setsockopt on each successful call to socket(). This is what I came up with:

break socket
class MyFinishBreakpoint (gdb.FinishBreakpoint):
    def stop (self):
        eax = gdb.parse_and_eval("$eax")
        if eax > 0:
            vrfname = "myvrf"
            gdb.execute("p (int)setsockopt($eax, 1, 25, \"%s\", %d)"%(vrfname, len(vrfname)))

        return False # don't want to stop
py MyFinishBreakpoint()

The above GDB script can be saved in a a file, lets call it fix_vrf.gdb. You can then run a command that has an issue like this:

sudo gdb -x fix_vrf.gdb –args ping -c 1 -I


The -I option in ping will send the icmp packet out the vrf correctly, but it will try to resolve out the wrong interface, that is until the script intercepts the socket() call in getaddrinfo and calls setsockopt on it using these steps:

  1. Set break point for all socket() calls
  2. use the python capability in gdb to define a new class inherited from FinishBreakpoint that will be called only AFTER socket() function is finished. This is important since we want the resultant socket file descriptor to call setsockopt on later.
  3. the stop method is defined which will be called when the socket() is done and will extract the socket() return value from the register eax. The result of any syscall will always be in eax as per the linux syscal calling conventions.
  4. If the result is > 0 (successful) we finally execute setsockopt with the correct options. I used the numeric versions of the #defines SOL_SOCKET, SO_BINDTODEVICE(1 & 25) for simplicity.
  5. The commands command is a list of commands to run when a breakpoint is encountered. In here I tell it to instantiate my python class, then continue with the program.
  6. The last two lines start the program running then exit when done.

I ended up having to use the python FinishBreakpoint method instead of the simply using the “finish” command because there is a limitation (or bug) in GDB which will not let you use the continue command after finish in a list of “commands”, while the python version works for some reason.


This seems to work well! I can’t speak for the performance impact, but for the purposes of proving an issue, I think this approach could be used generally to intercept and modify function results for your debugging or hacking needs!

System wide python profiling

I recently had to profile an embedded system that runs a bunch of python code. The goal was to improve the start-up speed and identify bottlenecks. This is a large and complex system that has many process starting and stopping and forking all over the place. It looked to be a difficult task to hook the profiler into each and every process that would get launched. I came up with a solution to temporarily hook the profiler into the interpreter itself and dump the stats at exit using the atexit module.

It turns out, python has the nifty ability to run code at start-up by adding it to the file. This file should be put in your python’s lib directory, in my case: /usr/lib/python2.7/ Anything you add to this file will get run when the python interpreter gets started. For example, if your file has only:

print "Hello World"

You will get this on starting python:

Hello World
Python 2.7.2 (default, Jan 30 2013, 17:47:27) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

So, in order to get the interpreter to fire up the profiler and dump the stats at exit, I just need to add some code to this file to do just that. Here is what I used:

import cProfile
import atexit
import os,sys
def ProfExit(p):
  prof_f_name = "%d"%os.getpid()
profile_hook = cProfile.Profile()
atexit.register(ProfExit, profile_hook)

So what does this do? It imports the profiler and the atexit module. It creates an instance of the profiler, registers with atexit to stop the profiler and dump the stats to a file named with the process ID of that python process, and finally starts the profiler. So every python process run on the system will now be profiled! FYI, the stats won’t get dumped until the process exits, so make sure you stop all of them.

What do I do will all of these stats files now that I have them. Well, you could go inspect them with the pstats module… or you could use this other great tool I found gprof2dot to convert them into call graphs. You must also install Graphviz to convert the dot files into images (apt-get install graphviz). But the results are worth it! The image show an example of the output, each block is a function called, and color codded by how much time was spent in that call. This makes it pretty easy to identify slow paths and easy optimizations. To generate call graphs for each stats file, I used something like this, from the directory where my stats files were:

for i in *; do ../ -f pstats $i | dot -Tsvg -o ../callgraphs/$i.svg; done;


Call Graph

Call Graph