This page describes how we upgraded python and django and how we installed python virtualenv.
We are using virtualenv to simplify our upgrade from python 2.5/django 1.1 to python 2.7/django 1.3.
More to the point, we may need to release an emergency patch to PROD while we are in the middle of upgrading python and django in QA.
This will also simplify future upgrades.
install python 2.7.3
$ cd /usr/local/src $ mkdir TARFILES $ chown -R dev . $ cd TARFILES $ wget http://python.org/ftp/python/2.7.3/Python-2.7.3.tgz ... $ cd .. $ tar xvfz TARFILES/Python-2.7.3.tgz $ cd Python-2.7.3 $ ./configure --help $ ./configure --rpe $ make ... Python build finished, but the necessary bits to build these modules were not found: _curses _curses_panel _sqlite3 _ssl bsddb185 bz2 dl gdbm imageop readline sunaudiodev To find the necessary bits, look in setup.py in detect_modules() for the module's name. ... $ sudo yum install openssl-devel readline-devel ImageMagick sqllite-devel \ bzip2-devel gdbm-devel ncurses-devel curses-devel ... # make yum stop complaining about incomplete transaction... $ sudo yum install yum-utils $ rpm -ql yum-i $ sudo /usr/sbin/yum-complete-transaction ... $ ./configure --prefix=/usr/local/python27 $ make clean $ make $ sudo make install
install virtualenv 2.7
$ wget http://python-distribute.org/distribute_setup.py $ sudo /usr/local/python27/bin/python distribute_setup.py $ wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py $ sudo /usr/local/python27/bin/python get-pip.py $ sudo /usr/local/python27/bin/pip install virtualenv $ sudo ln -s /usr/local/python27/bin/virtualenv /usr/local/bin/virtualenv-2.7
setting up a virtualenv
$ mkdir virtualenvs $ cd virtualenvs $ virtualenv-2.7 --no-site-packages --python=/usr/local/python27/bin/python python27_paul $ . ~/virtualenvs/python27_paul/bin/activate (python27)$ deactivate $
install gdal 1.8.1 into a virtualenv
$ . ~/virtualenvs/python27/bin/activate (python27_paul)$ cd /usr/local/src/gdal-1.8.1/ (python27_paul)$ ./configure --prefix=/home/dev/virtualenvs/python27_paul (python27_paul)$ make && make install (python27_paul)$ pip install --no-install GDAL==1.8.1 (python27_paul)$ vim ~/virtualenvs/python27_paul/build/GDAL/setup.py # # replace # extra_link_args = [] # with # extra_link_args = ['-Wl,-R/home/dev/virtualenvs/python27_paul/lib'] # (python27_paul)$ cd virtualenvs/python27_paul/build/GDAL (python27_paul)$ python setup.py build_ext --gdal-config=gdal-config --library-dirs=/home/dev/virtualenvs/python27_paul/lib --include-dirs=/home/dev/virtualenvs/python27_paul/include install # # confirm linked library is correct # $ ldd /home/dev/virtualenvs/python27/lib/python2.7/site-packages/GDAL-1.8.1-py2.7-linux-x86_64.egg/osgeo/_gdal.so # # should be pointing to the correct version of the libs # # if you start python in the GDAL dir, and try to import gdal you get an error # (python27_paul)$ pwd /home/dev/virtualenvs/python27_paul/build/GDAL (python27_paul)$ python Python 2.7.3 (default, May 9 2012, 14:28:18) [GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] ... >>> import gdal ImportError: No module named _gdal >>> exit() # # therefore do not start python in build/GDAL! # (python27_paul)$ cd ~ (python27_paul)$ pwd /home/dev (python27_paul)$ python ... >>> import gdal >>> exit() (python27_paul)$ deactivate $
email from Jeff Frost
We win!
Here's how I got it to work.
Recreated our virtualenv with --no-site-packages just to be clean (I don't think that was necessary, but just in case):
rm -rf ~/virtualenvs/python27
virtualenv-2.6 --no-site-packages --python=/usr/local/python27/bin/python python27 . ~/virtualenvs/python27/bin/activate
As you recall, we installed gdal-1.8.1 in the virtualenv like so:
./configure --prefix=/home/dev/virtualenvs/python27 --with-python make && make install
Well, this time I installed it without the --with-python option:
./configure --prefix=/home/dev/virtualenvs/python27
make && make install
EDIT: I decided to go back and test after creating a pristine virtualenv, and you don't have to do anything but:
pip-install GDAL==1.8.1
(python27)[dev@easdevweb
python27]$ python Python 2.7.3 (default, May 9 2012, 14:28:18) [GCC
4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2 Type "help", "copyright",
"credits" or "license" for more information.
>>> import gdal
>>>
So,
I think what happened is that allowing access to the site-packages also
allowed access to the other rpaths and thus linked against the wrong
gdal at runtime.
Read on for more interesting info though:
So, how to change the rpath for the python module?
pip install --no-install GDAL==1.8.1
cd build/GDAL/
vim setup.py
Just gotta add some extra_link_args. Here's a diff, so you can see what I
changed:
(python27)[dev@easdevweb GDAL]$ diff setup.py setup.py.orig
165c165
< extra_link_args = ['-Wl,-R/home/dev/virtualenvs/python27/lib']
---
> extra_link_args = []
python setup.py build_ext --gdal-config=gdal-config --library-dirs=/home/dev/virtualenvs/python27/lib
--include-dirs=/home/dev/virtualenvs/python27/include
and voila:
(python27)[dev@easdevweb GDAL]$ ldd
build/lib.linux-x86_64-2.7/osgeo/_gdal_array.so
libgdal.so.1 => /home/dev/virtualenvs/python27/lib/libgdal.so.1
(0x00002ae0b273c000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002ae0b329e000)
libm.so.6 => /lib64/libm.so.6 (0x00002ae0b359e000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002ae0b3822000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002ae0b3a30000)
libc.so.6 => /lib64/libc.so.6 (0x00002ae0b3c4b000)
libgeos_c.so.1 => /usr/local/lib/libgeos_c.so.1 (0x00002ae0b3fa3000)
libsqlite3.so.0 => /usr/lib64/libsqlite3.so.0 (0x00002ae0b41ba000)
libexpat.so.0 => /lib64/libexpat.so.0 (0x00002ae0b4415000)
libpq.so.4 => /usr/lib64/libpq.so.4 (0x00002ae0b4638000)
libz.so.1 => /usr/lib64/libz.so.1 (0x00002ae0b485a000)
librt.so.1 => /lib64/librt.so.1 (0x00002ae0b4a6e000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002ae0b4c78000)
/lib64/ld-linux-x86-64.so.2 (0x00000039f2e00000)
libgeos-3.1.1.so => /usr/local/lib/libgeos-3.1.1.so (0x00002ae0b4e7c000)
libssl.so.6 => /lib64/libssl.so.6 (0x00002ae0b51dd000)
libcrypto.so.6 => /lib64/libcrypto.so.6 (0x00002ae0b5429000)
libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00002ae0b577a000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00002ae0b5a10000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00002ae0b5c48000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002ae0b5e5d000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00002ae0b6076000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00002ae0b62a4000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00002ae0b64a6000)
libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00002ae0b66cc000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00002ae0b68d4000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00002ae0b6ad7000)
libsepol.so.1 => /lib64/libsepol.so.1 (0x00002ae0b6cef000)
python setup.py build_ext --gdal-config=gdal-config --library-dirs=/home/dev/virtualenvs/python27/lib --include-dirs=/home/dev/virtualenvs/python27/include install
(python27)[dev@easdevweb
site-packages]$ python Python 2.7.3 (default, May 9 2012, 14:28:18)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2 Type "help",
"copyright", "credits" or "license" for more information.
>>> import gdal
>>>
--
Jeff Frost <jeff@pgexperts.com>
CTO, PostgreSQL Experts, Inc.
Phone: 1-888-PG-EXPRT x506
FAX: 415-762-5122
http://www.pgexperts.com/
Add Comment