How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?)


Answers

Here's how I do it:

  • project
    • .Python
    • bin
    • lib
      • python2.5
        • site-packages
          • < pip install packages here >
    • include
    • src
      • app.yaml
      • index.yaml
      • main.yaml
      • < symlink the pip installed packages in ../lib/python2.5/site-packages

The project directory is the top level directory where the virtualenv sits. I get the virtualenv using the following commands:

cd project
virtualenv -p /usr/bin/python2.5 --no-site-packages --distribute .

The src directory is where all your code goes. When you deploy your code to GAE, *only* deploy those in the src directory and nothing else. The appcfg.py will resolve the symlinks and copy the library files to GAE for you.

I don't install my libraries as zip files mainly for convenience in case I need to read the source code, which I happen to do a lot just out of curiosity. However, if you really want to zip the libraries, put the following code snippet into your main.py

import sys
for p in ['librarie.zip', 'package.egg'...]:
    sys.path.insert(0, p)

After this you can import your zipped up packages as usual.

One thing to watch out for is setuptools' pkg_resources.py. I copied that directly into my src directory so my other symlinked packages can use it. Watch out for anything that uses entry_points. In my case I'm using Toscawidgets2 and I had to dig into the source code to manually wire up the pieces. It can become annoying if you had a lot of libraries that rely on entry_point.

Question

What's the best strategy for managing third-party Python libraries with Google App Engine?

Say I want to use Flask, a webapp framework. A blog entry says to do this, which doesn't seem right:

$ cd /tmp/
$ wget http://pypi.python.org/packages/source/F/Flask/Flask-0.6.1.tar.gz
$ tar zxf Flask-0.6.1.tar.gz
$ cp -r Flask-0.6.1/flask ~/path/to/project/
(... repeat for other packages ...)

There must be a better way to manage third-party code, especially if I want to track versions, test upgrades or if two libraries share a subdirectory. I know that Python can import modules from zipfiles and that pip can work with a wonderful REQUIREMENTS file, and I've seen that pip has a zip command for use with GAE.

(Note: There's a handful of similar questions — 1, 2, 3, 4, 5 — but they're case-specific and don't really answer my question.)




You simply copy the folder containing the library you want to use into your app engine project.

Then when you deploy it's uploaded with your application and is available for use.

EDIT: Jesse's answer is how I now do this. So do that!




How to include third party Python libraries in Google App Engine?

Actually I think this answer fits better here.

If you want to use 3rd party libraries that are not included in this list, then you'll have to add them manually.

In order to include manually any other library you have to have them inside the directory where the app.yaml lives. So for example if you have the following structure:

hello
├── libs
   └── bs4 
├── hello.py 
└── app.yaml

then in your hello.py you have to put these two lines in the beginning of the file:

import sys
sys.path.insert(0, 'libs')

After doing that you'll be able to use any 3rd party library that you're going to put in that libs directory.

For example:

from bs4 import BeautifulSoup






Your question is extremely broad, but there appears to be no big problems using Flask on Google App Engine.

This mailing list thread links to several templates:

http://flask.pocoo.org/mailinglist/archive/2011/3/27/google-app-engine/#4f95bab1627a24922c60ad1d0a0a8e44

And here is a tutorial specific to the Flask / App Engine combination:

http://www.franciscosouza.com/2010/08/flying-with-flask-on-google-app-engine/

Also, see App Engine - Difficulty Accessing Twitter Data - Flask, Flask message flashing fails across redirects, and How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?) for issues people have had with Flask and Google App Engine.