Managing dependencies with pip-tools

January 13, 2021 | Daryl Yu

pip-tools

Let’s use pip-tools to manage dependencies for a hypothetical Django project.

It provides two commands:

pip-compile which more or less replaces pip freeze for generating a list of pinned dependencies
pip-sync which more or less replaces manually running pip install and pip uninstall

pip-compile

With pip-compile we only need to specify the high level dependencies like Django. The convention is to use a file with a .in prefix.

# base.in
Django==3.0

To generate the .txt requirements file that we are familiar with, we use pip-compile.

pip-compile --generate-hashes base.in

Quick side note on --generate-hashes: one problem with Python packaging is that a package maintainer can remove a version from PyPI and then re-upload it with different code altogether. Using the --generate-hashes flag provides a bit of protection from this as it also considers the hash of the package that we want to install at the time of compilation and the hash of the package that we are downloading at the time of installation.

Once our pip-compile command for base.in succeeds, it generates a file called base.txt. It will pin the version number of Django as well as its dependencies like asgiref, pytz, and sqlparse.

#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --generate-hashes base.in
#
asgiref==3.2.3 \
--hash=sha256:7e06d934a7718bf3975acbf87780ba678957b87c7adc056f13b6215d610695a0 \
--hash=sha256:ea448f92fc35a0ef4b1508f53a04c4670255a3f33d22a81c8fc9c872036adbe5 \
# via django
django==3.0 \
--hash=sha256:6f857bd4e574442ba35a7172f1397b303167dae964cf18e53db5e85fe248d000 \
--hash=sha256:d98c9b6e5eed147bc51f47c014ff6826bd1ab50b166956776ee13db5a58804ae
pytz==2019.3 \
--hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \
--hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be \
# via django
sqlparse==0.3.0 \
--hash=sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177 \
--hash=sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873 \
# via django

One nice thing about pip-compile is that it lets you know which packages introduced which packages as sub-dependencies of the project.

Since this is for a Django project, we want to separate dependencies for the different environments.

For example, we want a development environment and a production environment.

For the development environment, we’ll install django-debug-toolbar. As the name implies, it is useful for debugging and provides helpful info like the exact SQL used which only makes sense for a developer’s machine but not for a server on the public internet.

Let’s make a new file called dev.in.

-c base.in
django-debug-toolbar==2.1

Since we still want Django in our development environment, we can use -c base.in to let pip-compile know that we won’t be listing those files again.

The -c base.in option doesn’t mean “include the dependencies in the base.in file.” The -c is short for constraint; similar to -c for pip install. It means “whatever dependencies that are resolved here in dev.in that also happen to be resolved in base.in should use the same version as the ones in base.in.

When we run pip-compile --generate-hashes dev.in, this generates a file called dev.txt.

#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --generate-hashes dev.in
#
django-debug-toolbar==2.1 \
--hash=sha256:24c157bc6c0e1648e0a6587511ecb1b007a00a354ce716950bff2de12693e7a8 \
--hash=sha256:77cfba1d6e91b9bc3d36dc7dc74a9bb80be351948db5f880f2562a0cbf20b6c5
django==3.0 \
--hash=sha256:6f857bd4e574442ba35a7172f1397b303167dae964cf18e53db5e85fe248d000 \
--hash=sha256:d98c9b6e5eed147bc51f47c014ff6826bd1ab50b166956776ee13db5a58804ae \
# via django-debug-toolbar
sqlparse==0.3.0 \
--hash=sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177 \
--hash=sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873 \
# via django, django-debug-toolbar

You may have noticed that django==3.0 is listed in dev.txt. This is because django-debug-toolbar explicitly lists django as one of its dependencies.

For the production environment the steps are basically the same, but this time we’ll install the WSGI application server gunicorn which only makes sense for a server on the public internet but not for a developer’s machine.

Let’s make a new file named prod.in.

-c base.in
gunicorn==20.0.4

When we run pip-compile --generate-hashes prod.in, this generates a file named prod.txt.

#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --generate-hashes prod.in
#
gunicorn==20.0.4 \
--hash=sha256:1904bb2b8a43658807108d59c3f3d56c2b6121a701161de0ddf9ad140073c626 \
--hash=sha256:cd4a810dd51bf497552cf3f863b575dabd73d6ad6a91075b65936b151cbf4f9c

# WARNING: The following packages were not pinned, but pip requires them to be
# pinned when the requirements file includes hashes. Consider using the --allow-unsafe flag.
# setuptools

You definitely noticed the WARNING at the end. This is because gunicorn explicitly lists setuptools as one of its dependencies but it did not pin the version.

Now we have three .txt files that list dependencies for our Django app. The next section covers how they are used to install the dependencies.

pip-sync

One problem with using plain pip is that when we do a pip install -r requirements.txt and there were packages installed in the virtualenv that were not specified in requirements.txt, those packages would still remain in the virtualenv.

With generated .txt requirements files from pip-compile, we are able to use pip-sync to use the exact packages and versions in those .txt files - no more no less.

While in theory we can use a hand-made .txt file, pip-compile was guaranteed to work with those generated by pip-sync.

To only install the packages that are common for development and production environments:

pip-sync requirements/base.txt

To install all the packages needed for a development environment:

pip-sync requirements/base.txt requirements/dev.txt

It is important to list both files in a single run. If only one is available, the virtualenv will only contain the packages listed in that file.

To install all the packages needed for a production environment:

pip-sync requirements/base.txt requirements/prod.txt

It plays nicely with pip

Another nice thing about pip-compile is that the generated files still work with pip.

pip install -r requirements/base.txt -r requirements/dev.txt

If you found this useful and would like to get updates on new posts, please subscribe to our newsletter and follow us on Twitter @DjangoCookbook.