As a developer, it is important to keep track of what 3rd party packages are installed on your virtual environment. In order for your project to work on other machines, you need to make sure each environment uses the same packages and the same versions.

For Python projects, pip is the package manager.

If you have used JavaScript before, you might be familiar with npm, a package manager. npm makes it easy to keep track of requirements. When packages are installed using npm install <package-name>, a file called package.json is automatically updated with the package name and the installed version.

Unfortunately, pip doesn’t do that. You can still install packages using pip install <package-name>, but it won't keep requirements.txt up to date for you.

Some developers keep track of requirements using a command called pip freeze. This will list all the packages installed on the virtual environment and their versions:

~/ctrl-z/django-htmx-todo ❯ pip freeze                                                                      
asgiref==3.5.2
backports.zoneinfo==0.2.1
build==0.8.0
click==8.1.3
Django==4.0.6
packaging==21.3
pep517==0.12.0
pip-tools==6.8.0
pyparsing==3.0.9
sqlparse==0.4.2
tomli==2.0.1

You can write this to a file using this command:

pip freeze > requirements.txt

To make sure your application works on other machines, you will need to run that line before committing your code.

This is valid, but there is another way to keep track of your project requirements.

The problem is that many packages rely on other packages to work. For example, when you install Django, you also install asgiref, backports-zoneinfo and sqlparse. Your requirements.txt file will list all of those. This makes it hard to keep track of what packages you installed with pip install and what packages are dependencies of other packages.

I am going to show you another way to manage your project requirements.

Why use pip-tools?

pip-tools is a command line tool (docs) to manage your Python project dependencies.

It allows you to define a second file requirements.in to manage the high-level dependencies. pip-tools provides a command called pip-compile which will compile the contents of requirements.in into requirements.txt.

When you want to add a package, you can add it to requirements.in. This file will have fewer lines than requirements.txt, making it easier to keep track of your dependencies. requirements.txt will state why each package is included.

For a basic Django blog, this is my requirements.in file. Notice how it only has 4 lines.

# requirements.in

django
django-autoslug
pillow
django-summernote

This is requirements.txt which was compiled from requirements.in.

# requirements.txt
#
# This file is autogenerated by pip-compile with python 3.8
# To update, run:
#
#    pip-compile --output-file=requirements.txt requirements.in
#

asgiref==3.5.2
    # via django
backports-zoneinfo==0.2.1
    # via django
bleach==5.0.1
    # via django-summernote
django==4.0.5
    # via
    #   -r requirements.in
    #   django-summernote
django-autoslug==1.9.8
    # via -r requirements.in
django-summernote==0.8.20.0
    # via -r requirements.in
pillow==9.2.0
    # via -r requirements.in
six==1.16.0
    # via bleach
sqlparse==0.4.2
    # via django
webencodings==0.5.1
    # via bleach

The requirements.txt file has a lot more lines because it includes the dependencies of all the packages listed in requirements.in. Notice that dependencies can have their own dependencies.

For example, I installed django-summernote, a 3rd party package that adds a text editor to forms (tutorial). django-summernote requires bleach, a Python package that will sanitise text submitted in forms. bleach then requires six and webencodings.

If we had done pip freeze > requirements.txt, then it would not be clear why bleach, six or webencodings are needed for the project.

How to use pip-tools

Step 1: Create a virtual environment

If you’re working on an existing project, you can skip to Step 2. Remember to activate your virtual environment.

Otherwise, start in your project folder and run this line in the terminal.

python3 -m venv venv

This will create a virtual environment called “venv” inside your folder. All of your installed packages will be placed in this folder. If you are familiar with JavaScript, a virtual environment is similar to a node_modules folder.

Now that you have a virtual environment, you need to activate it.

Activate your virtual environment by running this line (Mac OS and Linux) (Python docs):

source venv/bin/activate

If you are a Windows user, then you will need this command:

venv\Scripts\activate.bat

Step 2: Install pip-tools

Start in a directory with an active virtual environment.

When you create a virtual environment, there aren’t any 3rd party packages. You can verify this by typing pip freeze, which will return a list of installed packages. On a fresh virtual environment, that command will not return anything.

Step 2 is to install pip-tools. You can do it by running this line. This will be the only time you need to run pip install.

pip install pip-tools

Once pip-tools has been installed, you can run pip freeze again to see what packages were installed as a result of that command:

~/s/pip-test ❯ pip freeze                   
build==0.8.0
click==8.1.3
packaging==21.3
pep517==0.12.0
pip-tools==6.8.0
pyparsing==3.0.9
tomli==2.0.1

Step 3: Create requirements.in

Now that pip-tools has been installed, we can create a file called requirements.in:

touch requirements.in

Now, whenever we want to install a package using pip install <package-name>, we can put the package name into requirements.in instead.

In this example, we are just going to install Django.

On Line 1 of requirements.in, add “django”.

# requirements.in

django

Step 4: Compile the requirements (create requirements.txt)

You can run the line below to compile requirements.in into requirements.txt.

pip-compile --output-file=requirements.txt requirements.in

This will create a file called requirements.txt in the same directory.

Let’s have a look at requirements.txt:

#
# This file is autogenerated by pip-compile with python 3.8
# To update, run:
#
#    pip-compile --output-file=requirements.txt requirements.in
#
asgiref==3.5.2
    # via django
backports-zoneinfo==0.2.1
    # via django
django==4.0.6
    # via -r requirements.in
sqlparse==0.4.2
    # via django

Note that this doesn’t include pip-tools or its dependencies.

It is important to know that packages installed using pip install <package-name> won’t be included in requirements.txt. pip-compile only uses packages listed in requirements.in.

It’s acceptable to not include the packages installed via pip-tools.

This is because when you set up your project on another machine, you will need to install pip-tools using pip install so you can have access to pip-compile to compile the requirements.

Step 5: Install the requirements

The final step is to install the requirements:

pip install -r requirements.txt

This is the step that will install Django.

With Django installed, this line will now run without errors:

django-admin startproject myproject

Setting up a project on another machine

From here, you shouldn’t install packages using pip install <package-name>

Every time you want to add a package, you need to go through the following steps:

1. Add the package name to requirements.in

2. Compile the requirements using pip-compile --output-file=requirements.txt requirements.in

3. Install the requirements using pip install -r requirements.txt

4. Commit both requirements.txt and requirements.in to your remote repository.

By committing requirements.txt as well as requirements.in, you won’t have to recompile the requirements when setting up your project on another machine.

Specifying versions

You can specify a version for each package in requirements.in.

Take Django for example.

django on its own will use the latest version (4.0.6 at time of writing).

django==4.0.6 will install Django version 4.0.6

django>=3.2 specifies the minimum version, which will install 4.0.6.

Should you specify a version?

Generally, it is a good idea to specify a version. When a new version of Django is released, certain features of old versions may be deprecated and no longer work.

Conclusion

In Django projects, it’s important to keep a record of what packages are installed and their versions.

pip, the Python package manager has tools to like pip freeze to keep track of what’s installed in your virtual environment, but when many packages have their own dependencies, it’s hard to keep track of why each package is there.

There is a Python package called pip-tools, that makes it easier to keep track of the installed packages. pip-tools lets us define the high level packages (anything you would install using pip install) in a file called requirements.in.

This file can be compiled to produce requirements.txt, a file listing all packages installed in the virtual environment.

When requirements.txt is compiled, pip-tools states whether the package has been added via requirements.in or as a dependency of another pac