Creating a pSysmon development environment with virtualenv

To develop pSysmon it is useful to work in a clean and well defined python environment. During the time a large variety of python modules are collected on the everyday computer system. To guarantee consistent development regarding the python packages required by pSysmon working in a virtual python environment is more or less mandatory. The virtual environment can be easily setup using the virtualenv package. To facilitate the handling of the virtual environments I recommend using the virtualenvwrapper.

Virtualenv is a tool to create isolated python environments which are completely disconnected from the global python installation. With virtualenv one can setup environments to test pSysmon against various versions of packages and to test the pSysmon setup on a clean python environment. For more information on virtualenv see Working with virtualenv or IPython and virtualenv to see how IPython and virtualenv go together.

For creating the virtual environments I'm using the virtualenvwrapper tools. Make sure, that you have installed these tools on your system. To install virtualenvwrapper simply follow the instructions given on the virtualenvwrapper documentation homepage.

I'm using the directory ~/python-environments as the virtualenvwrapper WORKON_HOME variable.

Create a directory which will hold the virtual environment, change into this directory and call the virtualenv command.

user@computer ~ $ mkdir ~/python-environments
user@computer ~ $ mkvirtualenv psysmonDev

This will create the virtual environment psysmonDev in the ~/python-environments folder.

Usually, the virtual environment is activated automatically after it has been created. If it's not activated, activate the virtual environment psysmonDev and to add the packages required by pSysmon.

To activate the psysmonDev virtual environment run

user@computer ~ $ workon psysmonDev
(psysmonDev)user@computer ~ $ 

After executing the workon command you can see the (psysmonDev) prefix appearing in your shell command line. This lets you know with which virtual environment you are currently working.

To check the python packages installed in the current virtual environment, the command lssitepackages which is part of virtualenvwrapper can be used.

(psysmonDev)user@computer ~ $ lssitepackages 
easy_install.py   _markerlib  pip-6.0.6.dist-info  setuptools
easy_install.pyc  pip         pkg_resources        setuptools-11.0.dist-info
(psysmonDev)user@computer ~ $ 

As you can see, there are only the python core packages installed and we are working on a clean and isolated python environment.

To build all the following Python packages, some Linux packages have to be installed which are usually not part of the default installations of a Linux operating system. These packages are usually development packages needed for compilation of the Python packages. The given Linux package names vary from distribution to distribution. So if you can't find the package with the exact name, search for similiar packages on your distribution.

Install the packages using the apt-get install PACKAGE_NAME command or the package manager of your Linux distribution.

Linux package name needed by Python package
python-pip general
python-dev general
libblas-dev scipy
gfortran scipy
liblapack-dev scipy
g++ scipy
libfreetype6-devmatplotlib, pillow
libpng12-devmatplotlib
python-gtk2-devmatplotlib
libqt4-devmatplotlib
libgeos-devbasemap
libjpeg-devpillow
zlib1g-devpillow
libtiff5-devpillow
libopenjpeg-devpillow
tcl8.6-devpillow
tk8.6-devpillow
python-tkpillow
libmysqlclient-devmysql-python
libxslt1-devlxml
mesa-common-devwxPython
libglu1-mesa-devwxPython
libgstreamer1.0-devwxPython
libgstreamer-plugins-base1.0-devwxPython

Here's a copy and paste version to install the packages:

psysmon_packages.bash
sudo apt update
sudo apt install build-essential
sudo apt install python-pip
sudo apt install python-dev
sudo apt install libblas-dev
sudo apt install gfortran
sudo apt install liblapack-dev
sudo apt install g++
sudo apt install libfreetype6-dev
sudo apt install libpng12-dev
#sudo apt install python-gtk2-dev
sudo apt install libqt4-dev
sudo apt install libgeos-dev
sudo apt install libjpeg-dev
sudo apt install zlib1g-dev
sudo apt install libtiff5-dev
sudo apt install libopenjpeg-dev
sudo apt install tcl8.6-dev
sudo apt install tk8.6-dev
sudo apt install python-tk
sudo apt install libmysqlclient-dev
sudo apt install libxslt1-dev
sudo apt install mesa-common-dev
sudo apt install libglu1-mesa-dev
sudo apt install libgstreamer1.0-dev
sudo apt install libgstreamer-plugins-base1.0-dev
sudo apt install freeglut3
sudo apt install freeglut3-dev
sudo apt install libwebkitgtk-3.0-dev
sudo apt install libxtst-dev
sudo apt install libsdl1.2-dev
sudo apt install libgtk-3-dev

Most of the packages required by pSysmon (see the Requirements section for minimum version requirements and download links) can be installed using pip. Some packages (pycairo and wxPython) can't be installed using pip and need seperate compilation.

Following you can find some hints and problems that I stumbled over when installing the packages.

numpy

Use pip to install numpy:

(psysmonDev)user@computer ~ $ pip install numpy

scipy

Use pip to install scipy:

Eventually you might have to set the BLAS environment variable.

(psysmonDev)user@computer ~ $ pip install scipy

matplotlib

Use pip to install matplotlib:

(psysmonDev)user@computer ~ $ pip install matplotlib

basemap

Basemap is not found in the pypi repository. You have to download the source tarball from sourceforge and use the tarball to install basemap. Alternatively you can specify the sourceforge URL directly in the pip command.

For best performance, please make sure to point the installation URL to the most recent version of basemap. You can also download the tar.gz file and point the installation URL to the file on your local filesystem. This might be useful when you are going to rebuild your python environment frequently.

On Mint Linux 17.1 there are problems with the libgeos library when installing basemap. The libgeos library is not found by the compiler although it is installed. The problem is a missing link in the folder /usr/lib. Add the following link:

(psysmonDev)user@computer /usr/lib $ sudo ln -s libgeos-3.4.2.so libgeos.so

For example to install the basemap-1.0.7 package use the following URL in the pip command:

(psysmonDev)user@computer ~ $ pip install http://sourceforge.net/projects/matplotlib/files/matplotlib-toolkits/basemap-1.0.7/basemap-1.0.7.tar.gz/download

Pyro4

Use pip to install pyro4:

(psysmonDev)user@computer ~ $ pip install pyro4

sqlAlchemy

Install sqlAlchemy using pip from the source tarball or directly from the pypi repository.

(psysmonDev)user@computer ~ $ pip install sqlAlchemy

Pillow - Python Imaging Library

PIL hasn't seen any development since 2009, therefore I recommend using the PIL fork Pillow.

User pip to install pillow:

(psysmonDev)user@computer ~ $ pip install pillow

If everything works out fine, the following output should be found at the end of the pillow installation output:

    --------------------------------------------------------------------
    PIL SETUP SUMMARY
    --------------------------------------------------------------------
    version      Pillow 2.7.0
    platform     linux2 2.7.6 (default, Mar 22 2014, 22:59:56)
                 [GCC 4.8.2]
    --------------------------------------------------------------------
    --- TKINTER support available
    --- JPEG support available
    *** OPENJPEG (JPEG2000) support not available
    --- ZLIB (PNG/ZIP) support available
    --- LIBTIFF support available
    --- FREETYPE2 support available
    *** LITTLECMS2 support not available
    *** WEBP support not available
    *** WEBPMUX support not available
    --------------------------------------------------------------------

mysql-python

Use pip to install mysql-python:

(psysmonDev)user@computer ~ $ pip install mysql-python

lxml

User pip to install lxml:

(psysmonDev)user@computer ~ $ pip install lxml

obspy

Install obspy using pip from the source tarball or directly from the pypi repository.

(psysmonDev)user@computer ~ $ pip install obspy

pycairo

Installing pycairo is a little bit tricky. The Waf installer was working with Linux Mint 13, but on Linux Mint 17.1 it stopped with an error:

Checking for program python2.7-config    : /usr/bin/python2.7-config 
command ['/home/stefan/python-environments/psysmon_dev/bin/python', '/usr/bin/python2.7-config', '--includes'] returned 1

This is caused by the change of the python2.7-config from a Python script to a shell script. The Waf command check_python_headers(conf) is used by the pycairo configuration to get the headers and libraries, but it seems, that Waf calls the python2.7-config script using a Python command.

Therefore I recommend using the py2cairo version downloaded from the git repository.

Download the py2cairo from the Git repository:

(psysmonDev)user@computer ~ $ git clone git://git.cairographics.org/git/py2cairo

To install pycairo use the following commands:

(psysmonDev)user@computer ~/py2cairo $ ./autogen.sh --prefix=$VIRTUAL_ENV
(psysmonDev)user@computer ~/py2cairo $ ./configure --prefix=$VIRTUAL_ENV
(psysmonDev)user@computer ~/py2cairo $ make
(psysmonDev)user@computer ~/py2cairo $ make install

wxPython 4

The installation of wxPython 4 can be done using pip. No wheel was found for my distribution, so pip started the compilation of wxPython. The compilation succeeded without any problems. It's important to install all required development packages before creating the virtual environment, so that all links are set correctly.

(psysmonDev)user@computer ~ $ pip install lxml

On Linux Mint 13 I had to adjust the LD_LIBRARY_PATH to point to $VIRTUAL_ENV/lib to ensure that wxPython finds all the needed libraries. The postactivate script can be used to automatically set the LD_LIBRARY_PATH when activating a virtual environment. It might also be good to reset the LD_LIBRARY_PATH to the original value when deactivating the virtual environment. The two scripts I'm using are:

postactivate
#!/bin/bash
# This hook is sourced after this virtualenv is activated.
OLD_LD_LIBRARY_PATH=$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$VIRTUAL_ENV/lib
postdeactivate
#!/bin/bash
# This hook is sourced after this virtualenv is deactivated.
export LD_LIBRARY_PATH=$OLD_LD_LIBRARY_PATH

That's it. You have finished creating the pSysmon development environment and you can now activate this virtual environment and start coding in a clean python environment:

user@computer ~ $ workon psysmonDev
(psysmonDev)user@computer ~/01_gtd/04_aktuelleProjekte/pSysmon/01_src/psysmon/scripts $

pSysmon uses some code which needs to be compiled, therefore the setup.py script should be used to build all needed libraries. Just setting the PYTHON_PATH to the psysmon directory is not sufficient. To install pSysmon in the vitural environment in a way that it still can be updated using git and edited one should use the –editable (-e) option of the pip install command. The editable option requires a path to the folder which contains the psysmon setup.py file: pip install -e PATH_TO_FOLDER_CONTAINING_SETUP.PY.

(psysmonDev)user@computer ~$ pip install -e psysmon

This will create a psysmon.egg-link file in the folder $VIRTUAL_ENV/lib/python2.7/site-packages and an entry to the psysmon path in the file $VIRTUAL_ENV/lib/python2.7/site-packages/easy-install.pth.