I’m currently struggling with a bug in my open source project and it has to do with temporary object lifetimes.\u00a0 I believe the core issue is that I have a Python object that holds a pointer to an underlying C++ object that’s gotten deleted too early.\u00a0 I’ve created a toy SWIG project to try to isolate the issue.<\/p>\n
The code: swig_test.tar.gz<\/a><\/p>\n To build the project on your machine, just unzip and run make:<\/p>\n This will build a Python C++ extension called The The The The This is about as basic as it can be.<\/p>\n The go_test.py Python script:<\/p>\n Now when running make, the following test script is executed that illustrates the issue I’m running into:<\/p>\n The ‘Deleting Buffer(id = X)’ is being generated from inside the C++ code, so we can see here that in the ‘Funny business’ section, the C++ Buffer objects are getting deleted too early! The Python objects ‘b2’ and ‘b3’ should be holding references to the C++ Buffer objects with id=2 and id=4.<\/p>\n This summarizes the issue I’m having.<\/p>\n Pure Python Implementation<\/strong><\/p>\n I’ve been searching the web for information to learn what’s going one here. I found this stackoverflow question<\/a> and though it might be related, so I’ve implemented a pure Python version of Buffer in the script However, everything works as expected. Notice that Buffer ids 2, 4 are deleted at the end, so the pure Python version works as I expect.<\/p>\n Experiment 1: No Destructors<\/strong><\/p>\n So the first experiment I tried was to explicitly turn off destructors in the SWIG file with %nodefaultdtor<\/strong> and ignoring the C++ destructor:<\/p>\n And here’s the output:<\/p>\n Now the Python objects b2, b3 are pointing to ids 2, 4 and they are valid. But this approach leaks memory, so this is not a solution.<\/p>\n <\/p>\n Experiment 2: Using A Typemap<\/strong><\/p>\n My next experiment involves using a typemap like so:<\/p>\n If you were to compare the generated SWIG wrapper code in swig_test.cpp, you will see that this typemap appears to do the right thing, by returning the original PyObject that points to the C++ Buffer object on the left-hand-side of the expression, just like the pure Python method returns self. However, at run-time, a SWIG internal conversion error occurs and the code fails with:<\/p>\n Asking The Community For Help<\/strong><\/p>\n I’ve run out of ideas and now I need some new eyes to take a look. Hopefully a SWIG expert can quickly point out my error and I can move on with writing code.<\/p>\n So I’ve posted this<\/a> question to stackoverflow with a link to this blog post. If a solution comes up I’ll be sure to link to it from here.<\/p>\n A Solution!<\/strong><\/p>\n After some more searching I came across this thread<\/a> which eventually lead to a working solution. Using a Now I get the desired behavior with no memory leaks!<\/p>\n","protected":false},"excerpt":{"rendered":" I’m currently struggling with a bug in my open source project and it has to do with temporary object lifetimes.\u00a0 I believe the core issue is that I have a Python object that holds a pointer to an underlying C++ … Continue reading tar xfz swig_test.tar.gz\r\ncd swig_test\r\nmake<\/pre>\n
swig_test<\/code> and install it into your home directory.<\/p>\nBuffer<\/code> class:<\/p>\nstruct Buffer\r\n{\r\n Buffer();\r\n Buffer(const Buffer & copy);\r\n ~Buffer();\r\n\r\n Buffer & operator=(const Buffer & rhs);\r\n\r\n Buffer & operator<<(const Buffer & rhs);\r\n Buffer & operator<<(double rhs);\r\n\r\n std::string __str__() const;\r\n std::string __repr__() const;\r\n\r\n private:\r\n\r\n std::vector<double> _data;\r\n int _id;\r\n};<\/pre>\nBuffer<\/code> class is just a container, I’m using operator<<<\/code> to concatenate data to the end of the Buffer<\/code>.\u00a0 This is just a toy example illustrating the problem I’m having with SWIG, my actual class does a whole lot more.<\/p>\n_data<\/code> member is just a list of numbers that the container holds, _id<\/code> is a unique identifier so we can tell which Buffer<\/code> gets deleted.<\/p>\nswig_test.i<\/code> file:<\/p>\n%module swig_test\r\n\r\n%include \"std_string.i\"\r\n\r\n%{\r\n#include \"Buffer.hpp\"\r\n#include <iostream>\r\n%}\r\n\r\n%ignore Buffer::operator=;\r\n\r\n%include \"Buffer.hpp\"\r\n<\/pre>\nfrom swig_test import Buffer\r\n\r\ndef zeros(n):\r\n '''\r\n Returns a Buffer filled with 'n' zeros.\r\n '''\r\n\r\n b = Buffer()\r\n\r\n for i in xrange(n):\r\n b << 0.0\r\n\r\n return b\r\n\r\ndef ones(n):\r\n '''\r\n Returns a Buffer filled with 'n' ones.\r\n '''\r\n\r\n b = Buffer()\r\n\r\n for i in xrange(n):\r\n b << 1.0\r\n\r\n return b\r\n\r\ndef main():\r\n\r\n #--------------------------------------------------------------------------\r\n # This sections works as expected\r\n\r\n print \"-\" * 80\r\n print \"Works as expected:\"\r\n\r\n b0 = zeros(3)\r\n\r\n print \" b0 = \", b0\r\n\r\n b1 = ones(3)\r\n\r\n print \" b1 = \", b1\r\n\r\n y = b0 << b1\r\n\r\n print \" b0 = \", b0\r\n print \" y = \", y\r\n print \" b1 = \", b1\r\n\r\n print \" repr(b0) = \", repr(b0)\r\n print \" repr(y) = \", repr(y)\r\n\r\n #--------------------------------------------------------------------------\r\n # Funny things are happening here!\r\n\r\n print \"Funny business:\"\r\n\r\n b2 = zeros(3) << ones(3)\r\n\r\n print \" repr(b2) = \", repr(b2)\r\n\r\n b3 = zeros(3) << 4.0\r\n\r\n print \" repr(b3) = \", repr(b3)\r\n\r\nif __name__ == \"__main__\":\r\n main()\r\n<\/pre>\n
python go_test.py\r\n--------------------------------------------------------------------------------\r\nWorks as expected:\r\n b0 = Buffer(0, 0, 0, )\r\n b1 = Buffer(1, 1, 1, )\r\n b0 = Buffer(0, 0, 0, 1, 1, 1, )\r\n y = Buffer(0, 0, 0, 1, 1, 1, )\r\n b1 = Buffer(1, 1, 1, )\r\n repr(b0) = Buffer(id = 0, vector at 0x020bf450, data at 0x020aeb30, size = 6)\r\n repr(y) = Buffer(id = 0, vector at 0x020bf450, data at 0x020aeb30, size = 6)\r\nFunny business:\r\n Deleting Buffer(id = 2)\r\n Deleting Buffer(id = 3)\r\n repr(b2) = Buffer(id = 2, vector at 0x020bf790, data at 0x00, size = 4257068)\r\n Deleting Buffer(id = 4)\r\n repr(b3) = Buffer(id = 4, vector at 0x02037040, data at 0x0204a4e0, size = 6)\r\n Deleting Buffer(id = 0)\r\n Deleting Buffer(id = 1)\r\n<\/pre>\n
\ngo_test_pure.py<\/code>, here’s it’s output:<\/p>\npython go_test_pure.py\r\n--------------------------------------------------------------------------------\r\nWorks as expected:\r\n b0 = Buffer(0.0, 0.0, 0.0, )\r\n b1 = Buffer(1.0, 1.0, 1.0, )\r\n b0 = Buffer(0.0, 0.0, 0.0, 1.0, 1.0, 1.0, )\r\n y = Buffer(0.0, 0.0, 0.0, 1.0, 1.0, 1.0, )\r\n b1 = Buffer(1.0, 1.0, 1.0, )\r\n repr(b0) = Buffer(id = 0, data at 0x7FFEBD772CB0, size = 6)\r\n repr(y) = Buffer(id = 0, data at 0x7FFEBD772CB0, size = 6)\r\nNo funny business:\r\n Deleting Buffer(id = 3)\r\n repr(b2) = Buffer(id = 2, data at 0x7FFEBD7AB050, size = 6)\r\n repr(b3) = Buffer(id = 4, data at 0x7FFEBD772CF8, size = 4)\r\n Deleting Buffer(id = 1)\r\n Deleting Buffer(id = 0)\r\n Deleting Buffer(id = 2)\r\n Deleting Buffer(id = 4)\r\n<\/pre>\n
\n%module swig_test\r\n\r\n%include \"std_string.i\"\r\n\r\n%{\r\n#include \"Buffer.hpp\"\r\n#include <iostream>\r\n%}\r\n\r\n%ignore Buffer::operator=;\r\n%ignore Buffer::~Buffer; \/\/ ignoring C++ destructor\r\n%nodefaultdtor Buffer; \/\/ don't generate a destructor\r\n\r\n%include \"Buffer.hpp\"\r\n<\/pre>\npython go_test.py\r\n--------------------------------------------------------------------------------\r\nWorks as expected:\r\n b0 = Buffer(0, 0, 0, )\r\n b1 = Buffer(1, 1, 1, )\r\n b0 = Buffer(0, 0, 0, 1, 1, 1, )\r\n y = Buffer(0, 0, 0, 1, 1, 1, )\r\n b1 = Buffer(1, 1, 1, )\r\n repr(b0) = Buffer(id = 0, vector at 0x02865fa0, data at 0x02855b30, size = 6)\r\n repr(y) = Buffer(id = 0, vector at 0x02865fa0, data at 0x02855b30, size = 6)\r\nFunny business:\r\nswig\/python detected a memory leak of type 'Buffer *', no destructor found.\r\nswig\/python detected a memory leak of type 'Buffer *', no destructor found.\r\n repr(b2) = Buffer(id = 2, vector at 0x028662e0, data at 0x028119f0, size = 6)\r\nswig\/python detected a memory leak of type 'Buffer *', no destructor found.\r\n repr(b3) = Buffer(id = 4, vector at 0x0281d930, data at 0x027f1520, size = 4)\r\nswig\/python detected a memory leak of type 'Buffer *', no destructor found.\r\nswig\/python detected a memory leak of type 'Buffer *', no destructor found.\r\n<\/pre>\n
\n%module swig_test\r\n\r\n%include \"std_string.i\"\r\n\r\n%{\r\n#include \"Buffer.hpp\"\r\n#include <iostream>\r\n%}\r\n\r\n%ignore Buffer::operator=;\r\n\r\n%typemap(out) Buffer &\r\n{\r\n \/\/ TYPEMAP\r\n $result = $self;\r\n \/\/ TYPEMAP\r\n}\r\n\r\n%include \"Buffer.hpp\"\r\n<\/pre>\npython go_test.py\r\n--------------------------------------------------------------------------------\r\nWorks as expected:\r\n Deleting Buffer(id = 0)\r\nTraceback (most recent call last):\r\n File \"go_test.py\", line 71, in \r\n main()\r\n File \"go_test.py\", line 38, in main\r\n b0 = zeros(3)\r\n File \"go_test.py\", line 12, in zeros\r\n b << 0.0\r\nTypeError: unsupported operand type(s) for <<: 'Buffer' and 'float'\r\nmake: *** [all] Error 1\r\n<\/pre>\n
\n
\ntypemap(out)<\/code> in combination of a Py_INCREF()<\/code> did the trick.<\/p>\n%module swig_test\r\n\r\n%include \"std_string.i\"\r\n\r\n%{\r\n #include \"Buffer.hpp\"\r\n #include <iostream>\r\n%}\r\n\r\n%ignore Buffer::operator=;\r\n\r\n%typemap(out) Buffer & operator<<\r\n{\r\n if(result) { \/* suppress unused warning *\/ }\r\n Py_INCREF($self);\r\n $result = $self;\r\n}\r\n\r\n%include \"Buffer.hpp\"\r\n<\/pre>\n