Python binding in Visual Studio(without bjam)

I am hoping that you have already gone through my previous post Python binding with IDE.
In that post, I would have demonstrated it for Visual Studio IDE, but using Boost's own build tool bjam.

In this post, I am gonna explain how to configure Visual Studio to build out Python binding C++ code without bjam and the additional files it depends on(boost_build.jam and Jamroot. Its a relief that you dont have to change the Jamroot file accordingly, if your projects are in different folders. Helped me to resolve a lot of dependency issues without using bjam). But I would recommend bjam, if all your project files are in the same source directory. Its a great build tool by Boost!

Okay, lets get started.

1. Start with an Empty C++ Project as mentioned earlier
   (File->New->Project Select Under 'Visual C++', select 'Empty Project')
2. Create a new C++ file, say hello.cpp Copy the contents of the already existing hello.cpp from .../examples/tutorial.
3. Open Project properties page. Right-click Project, select Properties. Make the following changes in that window 

• Under 'General' :

           Target Extension : .pyd

           Configuration Type : Change to Static Library(.lib) ( Really have no idea why we have to do                  this, but the other Configuration types may cause unnecessary link errors , Dll worked for me                        though :) )

• Under 'VC++ Directories' :
  Include Directories, add C:\Boost\1.55.0;C:\Python27\include;
  Library Directories, add C:\Boost\1.55.0\stage\lib;C:\Python27\libs;
  ( Note : Add appropriate file location in your system )
  
  
  
• Under 'Linker' , you will have to add boost-python library and python library as shown below. (Copy the lib name, available under ..\Python27\libs\ and ..\Boost\1.55.0\stage\lib\ and press OK.

• Go ahead and build the file. You will find a *.pyd file either in your Project source directory or under Debug folder.
• Dont forget the most important step : Make sure to copy the boost_python-vc110-mt-gd-1_55.dll file available under ..\Boost\1.55.0\stage\lib\. Copy it to wherever your *.pyd file is.
  
  Then run your code using Python interpreter as mentioned in the previous post(Boost.Python setup )
  Hope you got the desired output.

          Feel free to provide your comments below!!!

 Reference :   

         http://www.sigverse.org/wiki/en/index.phpCompiling%20Boost.Python%20for%20Python%20in%20Visual%20Studio

Wrapping Functions using MACRO INVOCATION

When you want to wrap functions (or member functions) that either :

• have default arguments (or)
• are overloaded with a common sequence of initial arguments

we can either do a Manual wrapping or we can go for MACRO INVOCATION introduced by Boost.Python which makes wrapping simpler and easier.

Manual Wrapping 

Overloaded Functions 

We have here our C++ example class :

Class hello has 3 overloaded functions. To expose these in Python, we have to wrap them manually as shown below, else Python wouldn't understand which function to call when.

This is achieved by introducing some member function pointer variables :

Default Arguments

To have an idea about how Boost.Python manual wrapping works with Default arguments, have a look at it here. Boost website provides a clear explanation for this. So, am not gonna explain it here.

Lets move onto see how MACROS makes things easier.

MACROS

The macros used for this are as below.

For global functions and static methods :

For Class Methods :

Using MACROS our above program, can be written as,

When built and run using Python interpreter, we get the following desired output.

Reference :

[1] https://wiki.python.org/moin/boost.python/FunctionOverloading

[2] http://www.boost.org/doc/libs/1_55_0/libs/python/doc/tutorial/doc/html/python/functions.html#python.default_arguments

Exposing C++ std::vectors to Python

Boost.Python by default includes functionality for wrapping Vectors. This makes it quite simple for exposing C++ std::vectors to Python. The following sample program will provide you a detailed understanding on how to achieve it!

Program to add the sum of integers in a vector.

C++ program (SumOfVector.h)

When you build and run the above in the command Prompt using Python, you get the following output :

Python binding with IDE

How brilliant would it be, if Python binding can be achieved using IDE that aims to simplify software development process.
Am sure you would love it when you are able to configure your IDE to use bjam for its custom building process.
The IDE am gonna use here is Microsoft Visual Studio.Net 2012 ( you can use any IDE and configure it appropriately with the below steps).

So lets start!

Configure Visual C++ Project to use Boost.Build

1. As Boost uses it own Build system - bjam, you will have to start with an Empty C++ Project
   (File->New->Project Select Under 'Visual C++', select 'Empty Project')
2. And new text document and rename it to Jamroot (Note :  not Jamroot.txt)  Copy the entire Jamroot file contents from the .../example/tutorial folder here. Make sure the appropriate changes are made according to your project as mentioned in my previous post)
3. Add another text document and rename it into boost-build.jam It just needs to contain this line,
   boost-build "C:/Boost/1.55.0/tools/build/v2";
4. Create a new C++ file, say hello.cpp Copy the contents of the already existing hello.cpp from .../examples/tutorial.
5. Open Project properties page. Right-click Project, select Properties. Make the following changes in that window 

• Under 'General' :
  Configuration Type : Change to Static Library(.lib) ( Really have no idea why we have to do this, but the other Configuration types cause unnecessary link errors , Dll worled for me though :) )
• Under 'VC++ Directories' :
  Include Directories, add C:\Boost\1.55.0;C:\Python27\include;
  Library Directories, add C:\Boost\1.55.0\stage\lib;C:\Python27\libs;
  ( Note : Add appropriate file location in your system )
  
  
  
• Under 'Custom Build Step' :
  Command Line : bjam
  Outputs : .pyd

Finally, build your project. If you have done everything correctly, you should be able to Build it successfully without errors and find a *.pyd file in your project root folder.

Congratulations! You have built your project successfully with Visual Studio. The steps are quite similar for any IDE.

Reference :

[1]   http://pythonthusiast.pythonblogs.com/230_pythonthusiast/archive/1335_pycocos2d-x__how_to_properly_setup_boostpython_in_your_visual_c_2012_project.html

Boost.Python setup

Now, for the tough part! Are you ready to test your patience?
What are we waiting for, lets start!

Please note, the setup process mentioned here is for Windows environment!

Software Requirements :

1. Microsoft Visual Studio (Version I am using 2012) including Microsoft Visual C++
2. Python ( I have used version 2.7 - Boost.Python requires version 2.2 or newer)
3. Boost ( I have used version 1.55.0)

Build Boost.Python

1. Download Boost and extract it to a path in your system, lets say C:\Boost\1.55.0
2. Put this into your PATH variable, so that bjam can be run anywhere within the system. (bjam is one part of Boost's own build system, called Boost.Build).
3. Download Python and extract it to a path in your system say C:\Python27 Add this to the PATH variable.(To check if python is installed correctly, type python in the Command Prompt and you should be able to see the python interpreter with >>> )
4. Open 'Developer Command Prompt for VS2012' (Start->Microsoft Visual Studio 2012-> Visual Studio Tools)
5. Navigate to the Boost_root location(C:\Boost\1.55.0)
6. Enter bootstrap to prepare Boost in building it for our system.
7. Then, run b2  --toolset=msvc-11.0  --build-type=complete  stage (Time for a cup of tea! :) )
   (If you ever come across link.jam : No such file or directory during the build process. Dont panic and waste your time trying to resolve it like I did, its quite harmless!)
   As a result you should be able to see two new directories in your Boost_root location 
   bin.v2 (Build directory that contains generated library files, object files and other auxillary files) and  stage (contains library files alone, ready to be deployed)
   Note :  --toolset=msvc-11.0 tells bjam to use Microsoft Visual Studio 2012 toolkit(varies depending on your visual studio version number, for 2012, its 11.0)
              --build-type=complete tells Bjam to build all possible variations of the libraries.
8. One unfortuante thing is, for our Python project to be successfully integrated, you have to check if C:\Boost\1.55.0\stage\lib\libboost_python-vc110-mt-gd-1_55.lib is renamed to boost_python-vc110-mt-gd-1_55.lib, else you will have to copy this file into root folder of your project.
   (This annoying problem left me hours wondering what the error was!!!)

Checking correctness of our Boost.Build system installation!

Inorder to test the correctness of our installation, lets use the already existing Boost.Python example. Still in Command Prompt environment, go to C:\Boost\1.55.0\libs\python\example\tutorial You will see three files

1. Jamroot : Boost.build system will use this file to build your application. It is similar to a Makefile
2. hello.cpp : Main C++ class whose members are ready to be exposed to Python
3. hello.py : A python code that uses the exposed information from the above class (Optional)

Note : Dont forget to copy the aforementioned  C:\Boost\1.55.0\stage\lib\boost_python-vc110-mt-gd-1_55.lib file into C:\Boost\1.55.0\libs\python\example\tutorial. Otherwise your build process will not be successful.

Run bjam (It will not work if you haven't set your PATH variable as mentioned in step 2)

If everything works fine, you will see a newly created hello_ext.pyd in the tutorial folder as a result of the build process. This is very much similar to a regular Windows DLL file but renamed into *.pyd file.(Python will not import a DLL file instead it will import only PYD file)

Test this by running python and import this new PYD file as below :

(or) since we already have an existing hello.py with the python code. You can simply run python hello.py to see the same output.

Let's take a look at what is inside hello.cpp file. I have added comments for better understanding.

Hope its quite starightforward!

So what are the corresponding changes that need to be made in the Jamroot file. Let's take a look at what's inside Jamroot file.

In the above file, I have changed the default BOOST_PYTHON_MODULE(hello_ext) to BOOST_PYTHON_MODULE(sample) for better understanding on what it does!

Take a look at the lines the corresponding changes ought to be made! If you are planning to change the name in your project, make sure to make the necessary changes in your Jamroot file as well.
Also, if your project is in some other directory and not within Boost directory, make sure you give the Boost path under use-project boost as shown above(see the first blue dot!)

Note : If the file path contains spaces, make sure you enclose the path within double quotes. This is applicable to any path you use in this project.
Eg : C:/Boost 1.55.0/tools/build/v2 then change it to "C:/Boost 1.55.0/tools/build/v2"
Also note for Windows, single backward slash or two forward slashes only are acceptable between filenames in a path.

This is it! We are almost done!
As a final reminder, make sure your project folder always has these three important files along with your C++ files :

1. Jamroot (with the corresponding changes made as mentioned above)
2. boost.build (Mainly used to locate the installation directory of Boost distribution. Its contents would be : boost-build C:\\Boost\\1.55.0\\tools\\build\\v2 )
3. Boost library file boost_python-vc110-mt-gd-1_55.lib

Okay! Thats brilliant! We have successfully achieved python binding with this simple module.
Now, you are good to start with your own Python Binding project. 

Happy coding!

Important Note : 

If suppose you encounter an error while building Boost, it may be due to the following reason.
A file called user-config.jam may be found either in your home directory or most probably in C:\Boost\1.55.0\tools\build\v2. This file specifies where the tools and libraries are available to the Build system. If build fails, one of the many reasons might be that Boost.Build is unable to find Python and link to its corresponding libraries. You may need to edit this file to help Boost.Build find the necessary.
Add these two lines in user-config.jam and try building Boost again.

using msvc : 11.0 : "C:\\Program Files\\Microsoft Visual Studio 11.0\\VC\\bin\\cl.exe" ;
#msvc:11.0 is Microsoft Visual C++ version number, for 12 its 11. You need to change it accordingly).
using python 
    : 2.7                                    # Version
    : C:\\Python27\\python.exe  # Python Path
    : C:\\Python27\\include        # include path
    : C:\\Python27\\libs             # lib path(s)
    : <define>BOOST_ALL_NO_LIB=1
    ;

Reference :

[1]    'Knowledge Base Wiki'
http://www.thalesians.com/finance/index.php/Knowledge_Base/CPP/Boost_Python_Step_By_Step

[2]    'Boost.Python Build HowTo'
http://www.boost.org/doc/libs/1_50_0/libs/python/doc/building.html 

Python Binding - Introduction

In this post am gonna give a quick overview on what Python binding actually is.

Python binding is nothing but, exposing C/C++ classes, data members/functions to Python. What does 'exposing' mean? "Member functions and data members written in C/C++ can be accessed by(or will be available to) Python by this method".

You may ask, why C++ - Python interface? Parts of program that require more flexibility than speed/efficiency can be written in Python and critical portions can be coded in C++. 

According to PythonWiki, Python has more than one way to make C/C++ functionality accessible to Python.

I am choosing Boost.Python, mainly because of its stable support.

According to QuickStart, the Boost Python Library is a framework for interfacing Python and C++. It allows quick and seamless exposure of C++ classes,functions and objects to Python, and vice-versa using no special tools - just C++ compiler. It is designed to wrap C++ interfaces non-intrusively, so that the C++ code need not be modified/altered at all inorder to wrap it, making Boost.Python ideal for exposing 3rd part libraries to Python.

The library's use of advanced metaprogramming techniques simplifies its syntax for users, so that wrapping code takes on the look of a kind of declarative Interface Definition Language(IDL).

Advantages of Boost.Python

• Comprehensive lifetime management facilities for low-level C++ pointers and references (Call Policies).
• Support for C++ virtual functions that can be overridden in Python.
• Wrapping of overloaded operators, STL container classes (like vector and map).
• Support for organizing extensions as Python packages, with a central registry for inter-language type conversions.
• As mentioned earlier, it allows exposing C++ classes and functions to Python, without an additional wrapping language, but simply using C++ compiler.

Compiling the library initially and getting it to work in your environment or IDE can be quite frustrating and time consuming.

In the next post I will guide you through the setup of Boost.Python that will ease your burden!

Happy learning!