I’ve made an introduction video for Nsound!
Nsound Introduction Video
SWIG C++ & Python: C++ & Python Object Lifetime
I’m currently struggling with a bug in my open source project and it has to do with temporary object lifetimes. I believe the core issue is that I have a Python object that holds a pointer to an underlying C++ object that’s gotten deleted too early. I’ve created a toy SWIG project to try to isolate the issue.
The code: swig_test.tar.gz
To build the project on your machine, just unzip and run make:
1 2 3 |
tar xfz swig_test.tar.gz cd swig_test make |
This will build a Python C++ extension called swig_test
and install it into your home directory.
The Buffer
class:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
struct Buffer { Buffer(); Buffer(const Buffer & copy); ~Buffer(); Buffer & operator=(const Buffer & rhs); Buffer & operator<<(const Buffer & rhs); Buffer & operator<<(double rhs); std::string __str__() const; std::string __repr__() const; private: std::vector<double> _data; int _id; }; |
The Buffer
class is just a container, I’m using operator<<
to concatenate data to the end of the Buffer
. This is just a toy example illustrating the problem I’m having with SWIG, my actual class does a whole lot more.
The _data
member is just a list of numbers that the container holds, _id
is a unique identifier so we can tell which Buffer
gets deleted.
The swig_test.i
file:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
%module swig_test %include "std_string.i" %{ #include "Buffer.hpp" #include <iostream> %} %ignore Buffer::operator=; %include "Buffer.hpp" |
This is about as basic as it can be.
The go_test.py Python script:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 |
from swig_test import Buffer def zeros(n): ''' Returns a Buffer filled with 'n' zeros. ''' b = Buffer() for i in xrange(n): b << 0.0 return b def ones(n): ''' Returns a Buffer filled with 'n' ones. ''' b = Buffer() for i in xrange(n): b << 1.0 return b def main(): #-------------------------------------------------------------------------- # This sections works as expected print "-" * 80 print "Works as expected:" b0 = zeros(3) print " b0 = ", b0 b1 = ones(3) print " b1 = ", b1 y = b0 << b1 print " b0 = ", b0 print " y = ", y print " b1 = ", b1 print " repr(b0) = ", repr(b0) print " repr(y) = ", repr(y) #-------------------------------------------------------------------------- # Funny things are happening here! print "Funny business:" b2 = zeros(3) << ones(3) print " repr(b2) = ", repr(b2) b3 = zeros(3) << 4.0 print " repr(b3) = ", repr(b3) if __name__ == "__main__": main() |
Now when running make, the following test script is executed that illustrates the issue I’m running into:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
python go_test.py -------------------------------------------------------------------------------- Works as expected: b0 = Buffer(0, 0, 0, ) b1 = Buffer(1, 1, 1, ) b0 = Buffer(0, 0, 0, 1, 1, 1, ) y = Buffer(0, 0, 0, 1, 1, 1, ) b1 = Buffer(1, 1, 1, ) repr(b0) = Buffer(id = 0, vector at 0x020bf450, data at 0x020aeb30, size = 6) repr(y) = Buffer(id = 0, vector at 0x020bf450, data at 0x020aeb30, size = 6) Funny business: Deleting Buffer(id = 2) Deleting Buffer(id = 3) repr(b2) = Buffer(id = 2, vector at 0x020bf790, data at 0x00, size = 4257068) Deleting Buffer(id = 4) repr(b3) = Buffer(id = 4, vector at 0x02037040, data at 0x0204a4e0, size = 6) Deleting Buffer(id = 0) Deleting Buffer(id = 1) |
The ‘Deleting Buffer(id = X)’ is being generated from inside the C++ code, so we can see here that in the ‘Funny business’ section, the C++ Buffer objects are getting deleted too early! The Python objects ‘b2’ and ‘b3’ should be holding references to the C++ Buffer objects with id=2 and id=4.
This summarizes the issue I’m having.
Pure Python Implementation
I’ve been searching the web for information to learn what’s going one here. I found this stackoverflow question and though it might be related, so I’ve implemented a pure Python version of Buffer in the script go_test_pure.py
, here’s it’s output:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
python go_test_pure.py -------------------------------------------------------------------------------- Works as expected: b0 = Buffer(0.0, 0.0, 0.0, ) b1 = Buffer(1.0, 1.0, 1.0, ) b0 = Buffer(0.0, 0.0, 0.0, 1.0, 1.0, 1.0, ) y = Buffer(0.0, 0.0, 0.0, 1.0, 1.0, 1.0, ) b1 = Buffer(1.0, 1.0, 1.0, ) repr(b0) = Buffer(id = 0, data at 0x7FFEBD772CB0, size = 6) repr(y) = Buffer(id = 0, data at 0x7FFEBD772CB0, size = 6) No funny business: Deleting Buffer(id = 3) repr(b2) = Buffer(id = 2, data at 0x7FFEBD7AB050, size = 6) repr(b3) = Buffer(id = 4, data at 0x7FFEBD772CF8, size = 4) Deleting Buffer(id = 1) Deleting Buffer(id = 0) Deleting Buffer(id = 2) Deleting Buffer(id = 4) |
However, everything works as expected. Notice that Buffer ids 2, 4 are deleted at the end, so the pure Python version works as I expect.
Experiment 1: No Destructors
So the first experiment I tried was to explicitly turn off destructors in the SWIG file with %nodefaultdtor and ignoring the C++ destructor:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
%module swig_test %include "std_string.i" %{ #include "Buffer.hpp" #include <iostream> %} %ignore Buffer::operator=; %ignore Buffer::~Buffer; // ignoring C++ destructor %nodefaultdtor Buffer; // don't generate a destructor %include "Buffer.hpp" |
And here’s the output:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
python go_test.py -------------------------------------------------------------------------------- Works as expected: b0 = Buffer(0, 0, 0, ) b1 = Buffer(1, 1, 1, ) b0 = Buffer(0, 0, 0, 1, 1, 1, ) y = Buffer(0, 0, 0, 1, 1, 1, ) b1 = Buffer(1, 1, 1, ) repr(b0) = Buffer(id = 0, vector at 0x02865fa0, data at 0x02855b30, size = 6) repr(y) = Buffer(id = 0, vector at 0x02865fa0, data at 0x02855b30, size = 6) Funny business: swig/python detected a memory leak of type 'Buffer *', no destructor found. swig/python detected a memory leak of type 'Buffer *', no destructor found. repr(b2) = Buffer(id = 2, vector at 0x028662e0, data at 0x028119f0, size = 6) swig/python detected a memory leak of type 'Buffer *', no destructor found. repr(b3) = Buffer(id = 4, vector at 0x0281d930, data at 0x027f1520, size = 4) swig/python detected a memory leak of type 'Buffer *', no destructor found. swig/python detected a memory leak of type 'Buffer *', no destructor found. |
Now the Python objects b2, b3 are pointing to ids 2, 4 and they are valid. But this approach leaks memory, so this is not a solution.
Experiment 2: Using A Typemap
My next experiment involves using a typemap like so:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
%module swig_test %include "std_string.i" %{ #include "Buffer.hpp" #include <iostream> %} %ignore Buffer::operator=; %typemap(out) Buffer & { // TYPEMAP $result = $self; // TYPEMAP } %include "Buffer.hpp" |
If you were to compare the generated SWIG wrapper code in swig_test.cpp, you will see that this typemap appears to do the right thing, by returning the original PyObject that points to the C++ Buffer object on the left-hand-side of the expression, just like the pure Python method returns self. However, at run-time, a SWIG internal conversion error occurs and the code fails with:
1 2 3 4 5 6 7 8 9 10 11 12 13 |
python go_test.py -------------------------------------------------------------------------------- Works as expected: Deleting Buffer(id = 0) Traceback (most recent call last): File "go_test.py", line 71, in main() File "go_test.py", line 38, in main b0 = zeros(3) File "go_test.py", line 12, in zeros b << 0.0 TypeError: unsupported operand type(s) for <<: 'Buffer' and 'float' make: *** [all] Error 1 |
Asking The Community For Help
I’ve run out of ideas and now I need some new eyes to take a look. Hopefully a SWIG expert can quickly point out my error and I can move on with writing code.
So I’ve posted this question to stackoverflow with a link to this blog post. If a solution comes up I’ll be sure to link to it from here.
A Solution!
After some more searching I came across this thread which eventually lead to a working solution. Using a typemap(out)
in combination of a Py_INCREF()
did the trick.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
%module swig_test %include "std_string.i" %{ #include "Buffer.hpp" #include <iostream> %} %ignore Buffer::operator=; %typemap(out) Buffer & operator<< { if(result) { /* suppress unused warning */ } Py_INCREF($self); $result = $self; } %include "Buffer.hpp" |
Now I get the desired behavior with no memory leaks!
How to clone Linux disk of different sizes; large HDD to smaller SSD
WARNING: MISUSE OF THE COMMANDS IN THIS TUTORIAL CAN CAUSE DATA LOSS!
BACKUP YOUR DATA!
YOU’VE BEEN WARNED!
Step 1: Clone original HDD to larger, spare HDD
1.1. Connect SRC and DST disk to computer
1.2. Disconnect external drives that you may be using for backups, you won’t want to accidentally overwrite them!
1.3. Boot off of Ubuntu 14.04 live USB stick or your Ubuntu tower with SRC and DST connected
1.4. Use gparted to identify your boot OS, original SRC, and DST disk device ids, for example:
1 2 3 |
OS = /dev/sda SRC = /dev/sdc DST = /dev/sdb |
Write these down! If you mix these up you will LOSE ALL YOU DATA! Using all different sized disks makes this easier. If you have multiple disks of the same size, connect one disk at a time and label them. You don’t want to make a mistake here.
1.5. Clone SRC disk to DST disk using the pv & dd utilities (pv is available in the Ubuntu “universe” apt-get repo):
1 2 |
sudo su pv -tpreb {SRC} | dd of={DST} bs=4K conv=notrunc,noerror,sync |
where:
1 2 |
{SRC} = source device id {DST} = destination device id |
for example on my system:
1 |
pv -tpreb /dev/sdc | dd of=/dev/sdb bs=4K conv=notrunc,noerror,sync |
This may take several hours depending on your disk sizes, using pipe view (pv) will give you an estimate for how long it will take.
1.6. Power off your computer after the disk is cloned, the Linux kernel may not read the newly cloned disk correctly.
Step 2: Resize Cloned HDD
2.1. After powering off, disconnect the original drive to keep it safe from modification
2.2. Boot computer again, use gparted to resize and move all partitions so that they will fit on the new SSD
You can launch gparted with the DST device id to operate on just that disk, for example:
1 |
sudo gparted /dev/sdb |
NOTE: gparted should display all partitions on the new disk without any warnings like /!\ next to a partition. All partitions should be readable by gparted with partition size and used listed next to each partition. If you are booting off a live USB stick, you will probably need to turn off the swap partition by right-clicking and selecting swapoff in the menu.
2.3. After resizing everything power off the computer
Step 3: Clone HDD To New SSD
3.1. With the power off, connect the new SSD to the computer
3.2. Boot computer with the resized HDD and the new SSD
3.3. Use gparted to identify your boot OS, original SRC, and DST disk device ids, for example:
1 2 3 |
OS = /dev/sda SRC = /dev/sdb DST = /dev/sdc |
Write these down! If you mix these up you will LOSE ALL YOU DATA!
3.4. Clone SRC disk to DST disk using the pv & dd utilities:
1 2 |
sudo su pv -tpreb {SRC} | dd of={DST} bs=4K conv=notrunc,noerror,sync |
where:
1 2 |
{SRC} = source device id {DST} = destination device id |
for example on my system:
1 |
pv -tpreb /dev/sdb | dd of=/dev/sdc bs=4K conv=notrunc,noerror,sync |
This may take several hours depending on your disk sizes, using pipe view (pv) will give you an estimate for how long it will take.
Since the destination SSD is likely smaller than the source HDD, dd will error out at the end stating the destination disk is out of space:
dd: error writing ‘/dev/sdc’: No space left on device
This is OKAY.
3.5. Power off the computer and disconnect the spare HDD, so moving forward we are only modifying the SDD
Step 4: Repair The Partition Table
The original partition table that got cloned to the SSD is now incorrect, the physical size of the disk has changed, we need to repair it. With Ubuntu 14.04, the partition table being used is something called a GPT. We will repair the partition using the tool gdisk.
4.1. Boot the computer without the spare HDD
4.2. Use gparted to identify your boot OS, and the new SSD device ids, for example:
1 2 |
OS = /dev/sda SSD = /dev/sdb |
Write these down! If you mix these up you will LOSE ALL YOU DATA!
4.3. Use gdisk to repair the GUID Partition Table (GPT):
1 2 3 4 5 6 7 |
sudo su gdisk /dev/sdb Command (? for help): x # enter Expert mode Expert command (? for help): e # relocate backup to end of disk Expert command (? for help): w # write table to disk and exit OK; writing new GUID partition table (GPT) to /dev/sdb. The operation has completed successfully. |
4.4. Reboot the computer so the Linux kernel sees the new partition table
Step 5: Repair Boot Image
After all the work of resizing the partitions and repairing the partition table, the GRUB boot image is likely non functional. From my experience the BIOS can try to boot from the disk but we are left with a single ‘-‘ blinking character and nothing happens. The second stage boot loader can’t be found. Here we’ll repair it.
5.1. Boot computer
5.2. Use cgdisk to create a tiny partition to install grub2 on:
1 2 |
sudo su cgdisk /dev/sdb |
5.3. Select the free space at the beginning of the disk to create the new partition, it MUST be the first partition on the disk:
1 2 3 4 5 6 7 8 9 |
[ New ] First sector (34-2047, default = 34): <return> Size in sectors or {KMGTP} (default = 2014): <return> Current type is 8300 (Linux filesystem) Hex code or GUID (L to show codes, Enter = 8300): ef02 # bios boot Current partition name is '' Enter new partition name, or to use the current name: grub_boot<return> [ Write ] |
Step 6. Install Grub2
Now that we have a dedicated grub boot partition, we can proceed to install grub2 on the disk.
6.1. Mount the data partition disk for chroot operations:
1 2 3 4 5 6 |
sudo su mount /dev/sdb1 /mnt mount --bind /dev /mnt/dev mount --bind /dev/pts /mnt/dev/pts mount --bind /proc /mnt/proc mount --bind /sys /mnt/sys |
6.2. Change root into mounted disk:
1 |
chroot /mnt |
6.3. Reinstall grub2, recheck, recreate menu:
1 2 3 |
grub-install /dev/sdb grub-install --recheck /dev/sdb update-grub |
6.4. Exit the chroot shell and unmount everything:
1 2 3 4 5 6 |
exit umount /mnt/sys umount /mnt/proc umount /mnt/dev/pts umount /mnt/dev umount /mnt |
6.5. Shutdown the computer and install the SSD into the new computer, it should now boot!
Nsound Intro & Realtime Demo
I’ve decided to start bloging about Nsound on weegreenblobbie.com since I haven’t been really using this space for anything else.
Nsound is a C++ library with a Python module that makes it easy to manipulate sound. Here’s the project’s homepage, misspellings and all.
I’ve created two mailing lists so if you’re interested you can join here:
nsound-users@lists.sourceforge.net
nsound-developers@lists.sourceforge.net
I’ve posted an introductory video about Nsound on youtube:
How To Use Macros And Blender
I wanted a way to perform a “Repeat Last Action” command in Blender. I found an easy way to do it on Ubuntu Linux with a little tool called xmacro. Xmacro is a simple keyboard and mouse recording utility. It will record keystrokes that can be played back at a later time. It is quite old but it works with a little help.
Step 1. Install Xmacro
$ sudo apt-get install xmacro
Step 2. Install Nick’s Helper Scripts
Save these two scripts somewhere on your Linux box.
record_macro.bash:
#!/usr/bin/env bash
ESC_KEY_CODE=9
# A delay is needed to ensure all keys are released that
# activate the macro prior to recording.sleep 1.0
# Press escape to stop recording the macro.
xmacrorec2 -k $ESC_KEY_CODE | tee /tmp/temp_xmacro.txt
play_macro.bash:
#!/usr/bin/env bash
# WARNING, a delay is needed to ensure all keys are released
# that activate the macro playback, if a key is still being
# held on the keyboard, who knows what key combinations will
# play on your desktop!sleep 0.5
# A loop is used to control the playback speed.
while read line; do
echo $line | xmacroplay $DISPLAY
sleep 0.005 # the delay between events
done < /tmp/temp_xmacro.txt
Step 3. Assign Macro Scripts To The Keyboard
In Gnome, open up the keyboard shortcuts by selecting: System –> Preferences –> Keyboard Shortcuts
Click on the Add button at the bottom.
Name: Record Macro
Command: /install_path/record_macro.bash
where install_path is the location you saved the helper scripts.
Next, assign a keyboard shortcut to “Record Macro”, I use CTRL+ALT+M
Click on the Add button again.
Name: Play Macro
Command: /install_path/play_macro.bash
where install_path is the location you saved the helper scripts.
Next, assign a keyboard shortcut to “Play Macro”, I use CTRL+ALT+P
Step 4. Try It Out In Blender
Open Blender, with the default cube open, enter edit mode.
We’ll record extruding two verticies.
Select two verticies.
Now press CTL+ALT+M to start macro recording.
Press e
Press RETURN
Press x
Press 1.0
Press RETURN
Press ESCAPE
Pressing ESCAPE will end macro recording.
Now repeat the last action by pressing CTR+ALT+P. Make sure you release the keys on the keyboard while the macro plays!
Enjoy!
-Nick