How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?)
Here's how I do it:
- < pip install packages here >
- < symlink the pip installed packages in ../lib/python2.5/site-packages
project directory is the top level directory where the virtualenv sits. I get the virtualenv using the following commands:
cd project virtualenv -p /usr/bin/python2.5 --no-site-packages --distribute .
src directory is where all your code goes. When you deploy your code to GAE, *only* deploy those in the src directory and nothing else. The
appcfg.py will resolve the symlinks and copy the library files to GAE for you.
I don't install my libraries as zip files mainly for convenience in case I need to read the source code, which I happen to do a lot just out of curiosity. However, if you really want to zip the libraries, put the following code snippet into your main.py
import sys for p in ['librarie.zip', 'package.egg'...]: sys.path.insert(0, p)
After this you can import your zipped up packages as usual.
One thing to watch out for is setuptools'
pkg_resources.py. I copied that directly into my
src directory so my other symlinked packages can use it. Watch out for anything that uses
entry_points. In my case I'm using Toscawidgets2 and I had to dig into the source code to manually wire up the pieces. It can become annoying if you had a lot of libraries that rely on
What's the best strategy for managing third-party Python libraries with Google App Engine?
Say I want to use Flask, a webapp framework. A blog entry says to do this, which doesn't seem right:
$ cd /tmp/ $ wget http://pypi.python.org/packages/source/F/Flask/Flask-0.6.1.tar.gz $ tar zxf Flask-0.6.1.tar.gz $ cp -r Flask-0.6.1/flask ~/path/to/project/ (... repeat for other packages ...)
There must be a better way to manage third-party code, especially if I want to track versions, test upgrades or if two libraries share a subdirectory. I know that Python can import modules from zipfiles and that pip can work with a wonderful REQUIREMENTS file, and I've seen that pip has a
zip command for use with GAE.
Note: this answer is specific for Flask on Google App Engine.
See the flask-appengine-template project for an example of how to get Flask extensions to work on App Engine. https://github.com/kamalgill/flask-appengine-template
Drop the extension into the namespace package folder at src/packages/flaskext and you're all set. https://github.com/kamalgill/flask-appengine-template/tree/master/src/lib/flaskext
Non-Flask packages can be dropped into the src/packages folder as zip files, eggs, or unzipped packages, as the project template includes the sys.path.insert() snippet posted above.
I recently created a tool for this called gaenv. It follows a requirements.txt format, but doesn't install it, you can install with pip install -r requirements.txt then run the command line tool gaenv.
$ pip install -r requirements.txt $ gaenv
This creates symlinks automatically, you could install gaenv in your virtualenv too and run the binary from there. Here is a blog post about it:
also on github