How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?)
Here's how I do it:
- < pip install packages here >
- < symlink the pip installed packages in ../lib/python2.5/site-packages
project directory is the top level directory where the virtualenv sits. I get the virtualenv using the following commands:
cd project virtualenv -p /usr/bin/python2.5 --no-site-packages --distribute .
src directory is where all your code goes. When you deploy your code to GAE, *only* deploy those in the src directory and nothing else. The
appcfg.py will resolve the symlinks and copy the library files to GAE for you.
I don't install my libraries as zip files mainly for convenience in case I need to read the source code, which I happen to do a lot just out of curiosity. However, if you really want to zip the libraries, put the following code snippet into your main.py
import sys for p in ['librarie.zip', 'package.egg'...]: sys.path.insert(0, p)
After this you can import your zipped up packages as usual.
One thing to watch out for is setuptools'
pkg_resources.py. I copied that directly into my
src directory so my other symlinked packages can use it. Watch out for anything that uses
entry_points. In my case I'm using Toscawidgets2 and I had to dig into the source code to manually wire up the pieces. It can become annoying if you had a lot of libraries that rely on
What's the best strategy for managing third-party Python libraries with Google App Engine?
Say I want to use Flask, a webapp framework. A blog entry says to do this, which doesn't seem right:
$ cd /tmp/ $ wget http://pypi.python.org/packages/source/F/Flask/Flask-0.6.1.tar.gz $ tar zxf Flask-0.6.1.tar.gz $ cp -r Flask-0.6.1/flask ~/path/to/project/ (... repeat for other packages ...)
There must be a better way to manage third-party code, especially if I want to track versions, test upgrades or if two libraries share a subdirectory. I know that Python can import modules from zipfiles and that pip can work with a wonderful REQUIREMENTS file, and I've seen that pip has a
zip command for use with GAE.
You simply copy the folder containing the library you want to use into your app engine project.
Then when you deploy it's uploaded with your application and is available for use.
EDIT: Jesse's answer is how I now do this. So do that!
How to include third party Python libraries in Google App Engine?
Actually I think this answer fits better here.
If you want to use 3rd party libraries that are not included in this list, then you'll have to add them manually.
In order to include manually any other library you have to have them inside the directory where the
app.yaml lives. So for example if you have the following structure:
hello ├── libs │ └── bs4 ├── hello.py └── app.yaml
then in your
hello.py you have to put these two lines in the beginning of the file:
import sys sys.path.insert(0, 'libs')
After doing that you'll be able to use any 3rd party library that you're going to put in that
from bs4 import BeautifulSoup
No modules named flask (GAE)
You have to tell your app where third-party libraries are by adding
appengine_config.py to your project root.
Your question is extremely broad, but there appears to be no big problems using Flask on Google App Engine.
This mailing list thread links to several templates:
And here is a tutorial specific to the Flask / App Engine combination:
Also, see App Engine - Difficulty Accessing Twitter Data - Flask, Flask message flashing fails across redirects, and How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?) for issues people have had with Flask and Google App Engine.