python pip-tools - Upgrading all packages with pip




upgrade windows (25)

How about:

pip install -r <(pip freeze) --upgrade

Is it possible to upgrade all Python packages at one time with pip?

Note that there is a feature request for this on the official issue tracker.


You can just print the packages that are outdated

pip freeze | cut -d = -f 1 | xargs -n 1 pip search | grep -B2 'LATEST:'

More Robust Solution

For pip3 use this:

pip3 freeze --local |sed -rn 's/^([^=# \t\\][^ \t=]*)=.*/echo; echo Processing \1 ...; pip3 install -U \1/p' |sh

For pip, just remove the 3s as such:

pip freeze --local |sed -rn 's/^([^=# \t\\][^ \t=]*)=.*/echo; echo Processing \1 ...; pip install -U \1/p' |sh

OSX Oddity

OSX, as of July 2017, ships with a very old version of sed (a dozen years old). To get extended regular expressions, use -E instead of -r in the solution above.

Solving Issues with Popular Solutions

This solution is well designed and tested1, whereas there are problems with even the most popular solutions.

  • Portability issues due to changing pip command line features
  • Crashing of xargs because common pip or pip3 child process failures
  • Crowded logging from the raw xargs output
  • Relying on a Python-to-OS bridge while potentially upgrading it3

The above command uses the simplest and most portable pip syntax in combination with sed and sh to overcome these issues completely. Details of sed operation can be scrutinized with the commented version2.


Details

[1] Tested and regularly used in a Linux 4.8.16-200.fc24.x86_64 cluster and tested on five other Linux/Unix flavors. It also runs on Cygwin64 installed on Windows 10. Testing on iOS is needed.

[2] To see the anatomy of the command more clearly, this is the exact equivalent of the above pip3 command with comments:

# match lines from pip's local package list output
# that meet the following three criteria and pass the
# package name to the replacement string in group 1.
# (a) Do not start with invalid characters
# (b) Follow the rule of no white space in the package names
# (c) Immediately follow the package name with an equal sign
sed="s/^([^=# \t\\][^ \t=]*)=.*"

# separate the output of package upgrades with a blank line
sed="$sed/echo"

# indicate what package is being processed
sed="$sed; echo Processing \1 ..."

# perform the upgrade using just the valid package name
sed="$sed; pip3 install -U \1"

# output the commands
sed="$sed/p"

# stream edit the list as above
# and pass the commands to a shell
pip3 freeze --local |sed -rn "$sed" |sh

[3] Upgrading a Python or PIP component that is also used in the upgrading of a Python or PIP component can be a potential cause of a deadlock or package database corruption.


I have tried the code of Ramana and I found out on Ubuntu you have to write sudo for each command. Here is my script which works fine on ubuntu 13.10:

#!/usr/bin/env python
import pip
from subprocess import call

for dist in pip.get_installed_distributions():
    call("sudo pip install --upgrade " + dist.project_name, shell=True)


Sent through a pull-request to the pip folk; in the meantime use this pip library solution I wrote:

from pip import get_installed_distributions
from pip.commands import install

install_cmd = install.InstallCommand()

options, args = install_cmd.parse_args([package.project_name
                                        for package in
                                        get_installed_distributions()])

options.upgrade = True
install_cmd.run(options, args)  # Chuck this in a try/except and print as wanted

To upgrade all local packages; you could use pip-review:

$ pip install pip-review
$ pip-review --local --interactive

pip-review is a fork of pip-tools. See pip-tools issue mentioned by @knedlsepp. pip-review package works but pip-tools package no longer works.

pip-review works on Windows since version 0.5.


This option seems to me more straightforward and readable:

pip install -U `pip list --outdated | awk '{ print $1}'`

(awk '{ print $1}' selects the first word of the line (separated by a space))

And this version allows for the suppression of warning message from pip list --outdated:

pip install -U `pip list --outdated | awk '!/Could not|ignored/ { print $1}'`

(awk '!/<pattern>/' removes line containing a given pattern. In my case the warning messages include "Could not" and "ignored" respectively)

This could also be used to tackle the coming default columns format:

pip install -U `pip list --format=columns --outdated | awk '!/Package|---/{ print $1}'`

You can try this :

for i in ` pip list|awk -F ' ' '{print $1}'`;do pip install --upgrade $i;done

Windows Powershell solution

pip freeze | %{$_.split('==')[0]} | %{pip install --upgrade $_}

The shortest and easiest on Windows.

pip freeze > requirements.txt && pip install --upgrade -r requirements.txt && rm requirements.txt


This seems more concise.

pip list --outdated | cut -d ' ' -f1 | xargs -n1 pip install -U

Explanation:

pip list --outdated gets lines like these

urllib3 (1.7.1) - Latest: 1.15.1 [wheel]
wheel (0.24.0) - Latest: 0.29.0 [wheel]

In cut -d ' ' -f1, -d ' ' sets "space" as the delimiter, -f1 means to get the first column.

So the above lines becomes:

urllib3
wheel

then pass them to xargs to run the command, pip install -U, with each line as appending arguments

-n1 limits the number of arguments passed to each command pip install -U to be 1


Windows version after consulting excellent documentation for FOR by Rob van der Woude

for /F "delims===" %i in ('pip freeze -l') do pip install -U %i

@Ramana's answer worked the best for me, of those here, but I had to add a few catches:

import pip
for dist in pip.get_installed_distributions():
    if 'site-packages' in dist.location:
        try:
            pip.call_subprocess(['pip', 'install', '-U', dist.key])
        except Exception, exc:
            print exc

The site-packages check excludes my development packages, because they are not located in the system site-packages directory. The try-except simply skips packages that have been removed from PyPI.

@endolith: I was hoping for an easy pip.install(dist.key, upgrade=True), too, but it doesn't look like pip was meant to be used by anything but the command line (the docs don't mention the internal API, and the pip developers didn't use docstrings).


One-liner version of @Ramana's answer.

python -c 'import pip, subprocess; [subprocess.call("pip install -U " + d.project_name, shell=1) for d in pip.get_installed_distributions()]'

`


when using a virtualenv and if you just want to upgrade packages added to your virtualenv, you may want to do:

pip install `pip freeze -l | cut --fields=1 -d = -` --upgrade

My script:

pip list --outdated --format=legacy | cut -d ' ' -f1 | xargs -n1 pip install --upgrade

The rather amazing yolk makes this easy.

pip install yolk3k # don't install `yolk`, see https://github.com/cakebread/yolk/issues/35
yolk --upgrade

For more info on yolk: https://pypi.python.org/pypi/yolk/0.4.3

It can do lots of things you'll probably find useful.


Isn't this more effective?

pip3 list -o | grep -v -i warning | cut -f1 -d' ' | tr " " "\n" | awk '{if(NR>=3)print}' | cut -d' ' -f1 | xargs -n1 pip3 install -U 
  1. pip list -o lists outdated packages;
  2. grep -v -i warning inverted match on warning to avoid errors when updating
  3. cut -f1 -d1' ' returns the first word - the name of the outdated package;
  4. tr "\n|\r" " " converts the multiline result from cut into a single-line, space-separated list;
  5. awk '{if(NR>=3)print}' skips header lines
  6. cut -d' ' -f1 fetches the first column
  7. xargs -n1 pip install -U takes 1 argument from the pipe left of it, and passes it to the command to upgrade the list of packages.

You can use the following Python code. Unlike pip freeze, this will not print warnings and FIXME errors. For pip < 10.0.1

import pip
from subprocess import call

packages = [dist.project_name for dist in pip.get_installed_distributions()]
call("pip install --upgrade " + ' '.join(packages), shell=True)

For pip >= 10.0.1

import pkg_resources
from subprocess import call

packages = [dist.project_name for dist in pkg_resources.working_set]
call("pip install --upgrade " + ' '.join(packages), shell=True)

Here is a scripts that only updates the outdated packages.

import os, sys
from subprocess import check_output, call

file = check_output(["pip.exe",  "list", "--outdated", "--format=legacy"])
line = str(file).split()

for distro in line[::6]:
    call("pip install --upgrade " + distro, shell=True)

here is another way of doing with a script in python

import pip, tempfile, contextlib

with tempfile.TemporaryFile('w+') as temp:
    with contextlib.redirect_stdout(temp):
        pip.main(['list','-o'])
    temp.seek(0)
    for line in temp:
        pk = line.split()[0]
        print('--> updating',pk,'<--')
        pip.main(['install','-U',pk])

Here is my variation on rbp's answer, which bypasses "editable" and development distributions. It shares two flaws of the original: it re-downloads and reinstalls unnecessarily; and an error on one package will prevent the upgrade of every package after that.

pip freeze |sed -ne 's/==.*//p' |xargs pip install -U --

Related bug reports, a bit disjointed after the migration from bitbucket:


>>> def a():
>>>    print "a executed"
>>>    return []
>>> x =a()
a executed
>>> def b(m=[]):
>>>    m.append(5)
>>>    print m
>>> b(x)
[5]
>>> b(x)
[5, 5]




python pip