SWIG C++ & Python: C++ & Python Object Lifetime

Spread the love

I’m currently struggling with a bug in my open source project and it has to do with temporary object lifetimes.  I believe the core issue is that I have a Python object that holds a pointer to an underlying C++ object that’s gotten deleted too early.  I’ve created a toy SWIG project to try to isolate the issue.

The code: swig_test.tar.gz

To build the project on your machine, just unzip and run make:

This will build a Python C++ extension called swig_test and install it into your home directory.

The Buffer class:

The Buffer class is just a container, I’m using operator<< to concatenate data to the end of the Buffer.  This is just a toy example illustrating the problem I’m having with SWIG, my actual class does a whole lot more.

The _data member is just a list of numbers that the container holds, _id is a unique identifier so we can tell which Buffer gets deleted.

The swig_test.i file:

This is about as basic as it can be.

The go_test.py Python script:

Now when running make, the following test script is executed that illustrates the issue I’m running into:

The ‘Deleting Buffer(id = X)’ is being generated from inside the C++ code, so we can see here that in the ‘Funny business’ section, the C++ Buffer objects are getting deleted too early! The Python objects ‘b2’ and ‘b3’ should be holding references to the C++ Buffer objects with id=2 and id=4.

This summarizes the issue I’m having.

Pure Python Implementation

I’ve been searching the web for information to learn what’s going one here. I found this stackoverflow question and though it might be related, so I’ve implemented a pure Python version of Buffer in the script go_test_pure.py, here’s it’s output:

However, everything works as expected. Notice that Buffer ids 2, 4 are deleted at the end, so the pure Python version works as I expect.

Experiment 1: No Destructors

So the first experiment I tried was to explicitly turn off destructors in the SWIG file with %nodefaultdtor and ignoring the C++ destructor:

And here’s the output:

Now the Python objects b2, b3 are pointing to ids 2, 4 and they are valid. But this approach leaks memory, so this is not a solution.


Experiment 2: Using A Typemap

My next experiment involves using a typemap like so:

If you were to compare the generated SWIG wrapper code in swig_test.cpp, you will see that this typemap appears to do the right thing, by returning the original PyObject that points to the C++ Buffer object on the left-hand-side of the expression, just like the pure Python method returns self. However, at run-time, a SWIG internal conversion error occurs and the code fails with:

Asking The Community For Help

I’ve run out of ideas and now I need some new eyes to take a look. Hopefully a SWIG expert can quickly point out my error and I can move on with writing code.

So I’ve posted this question to stackoverflow with a link to this blog post. If a solution comes up I’ll be sure to link to it from here.

A Solution!

After some more searching I came across this thread which eventually lead to a working solution. Using a typemap(out) in combination of a Py_INCREF() did the trick.

Now I get the desired behavior with no memory leaks!

This entry was posted in programming and tagged , , . Bookmark the permalink.