The language of Django is Python, but the language of relational databases is SQL. So how do you manage your application’s database without having to learn SQL?

What makes Django a powerful web framework is it lets you code all aspects of web server in Python. However, it does need a way to turn Python code into SQL to manage the database.

For querying the database, Django has something called Django ORM.

For administering the database, that is adding or altering tables, it has migrations.

What are migrations?

Migrations are used by Django to change the structure of the database without the developer having to write any SQL themselves.

When you update your model (a class which defines the structure of a particular table), Django automatically detects the changes and calculates (in Python) what it needs to do to the database. This is stored in a file called a migration.

Running a migration applies the changes to the database. Django compiles the contents of the migration file into SQL (code that the database will understand) and runs it on the database.

Migrations in Django are grouped by app and stored in folders called “migrations”. The migration files keep a record of all the changes that have been applied to the database, from creating the table, through to column alterations. Even small details, like changing a default value, are recorded in migrations.

By having a record of all changes to the database structure, you can recreate your database (apart from its contents) if you ever lose your database and have to start again. Migrations also ensure that the database structure is in-sync with your models.

Migrations can often be the source of frustration for developers. Deleting migration files or tampering with migration records in the database can lead to errors that are very difficult to fix. By understanding how migrations work and how to reverse migrations you no longer want, will save you a lot of pain.

How migrations are created

Whenever you make a change to a field in one of your models, your model becomes out of sync with the database. You will need to create and run a migration to bring your database back in sync.

Thankfully, Django is very clever at detecting these changes. The command: python manage.py makemigrations, will make Django check your models for any changes and automatically create a migration file.

The migration file contains a summary of the changes Django will make to the database, written in python.

Creating a migration file does not make any changes to the database. If you haven’t run the migration, then migration files are safe to delete.

Numbering of migrations

Migrations always begin with a number. The number is there to ensure migrations are applied in order. You wouldn’t want to run a migration that alters a table if the migration that creates it hasn’t been run yet.

As long as you haven’t run the migration and the name of the migration file begins with the number followed by an underscore, you can name the migration what you like.

What goes into a migration file?

Migration files have two parts:

1. Dependencies

2. Operations

Dependencies

The order in which migrations are applied is critical. The dependencies specify which migrations must be run first. In most cases, it is the preceding migration in the app, but there can be multiple.

Operations

Migrations have a list of operations. Operations include creating a table, adding a field or altering a field. Migrations can have multiple operations and the list sets out the order in which the operations are applied.

How to run migrations

The command python manage.py migrate will apply all new migrations to the database. This means Django will compile the migration written in Python into SQL, the language of the database.

Once a migration has been run, migration files should not be deleted. Deleting a migration file will not reverse the changes to the database. If you want to get rid of a migration file, you will need to reverse the migration first.

How to reverse a migration

You will need the name of your app (e.g. movies) and the number of the migration you want to rollback to. So if I’ve just applied a migration called 0004_movie_slug.py , but I want to reverse it. I can roll it back to 0003_alter_movie_popularity.py with the following command:

python manage.py migrate movies 0003

Notice that we only need the number of the migration, not the full file name.

Once the migration has been reversed, it is now safe to delete the migration file for 0002.

Records of migrations are stored in the database

In the database, there is a table called django_migrations. Every time Django runs a migration, it saves a record of it to the database. It stores which app the migration was for, the name of the migration file and the date/time the migration was made.

By default, Django initiates projects with a SQLite database. I recommend installing a VScode extension like SQLite Viewer, so you can look inside the database and see the migrations table for yourself.

It takes 17 migrations to start a Django project

If you look in the django_migrations table, you will find there are 17 entries in there already.

When you start a Django project, you don’t start on a blank database. For Django to work properly, it needs to create tables for sessions, migrations, users, user groups, and permissions.

Alongside apps you’ve added yourself, there are apps like django.contrib.admin and django.contrib.auth. Each of these apps will have their own folder of migrations. It’s necessary to run these migrations to initialise the project database.

Avoid deleting migrations from the database.

Running a migration involves applying operations to the database in SQL. Deleting a migration will only delete the record of it. It will not reverse the operations of the migration. If you want to get rid of a migration, reverse the migration. Reversing a migration will then remove the migration record from the database for you.

How to add non-nullable fields

Databases are strict about what can and can’t be done to it. Therefore Django has to be strict when evaluating the changes made to your models.

A common error is when you try to add a field to an existing table without allowing null values or providing a default. The database must have a value to put into existing rows.

Deleting the existing rows in your database will not avoid this error.

Instead, you must define a default either upon making the migrations or in your model.

If you need to add a non-nullable field, but the value for existing rows will be different for each row, you can customise your migration.

In this tutorial, I’ll show you how to make your migration more flexible by adding custom python.

Migrations gone wrong - two stories

Lesson 1: don't delete migrations and commit them

This happened in the very early stages of a project with just me and one other developer.

I pulled a colleague's branch and immediately got lots of errors. It took a while to figure out that the database didn't match what was in the models. It turned out that my colleague had dropped his database, deleted the migration files and started again, leaving his migrations out of sync with my database.

Lesson 2: be careful when working on multiple branches and never delete migration records from the database

I was working on a branch (Feature A) where I had made a couple of migrations. I had to switch branches to work on a different feature but my database was still in the same state as Feature A. The correct thing to do was to reverse my migrations in the Feature A branch before switching branches. Instead (not understanding how migrations work), I deleted the migration records from the database instead.

I ended up not being able to run migrations. I had to manually re-add the migration records to the database.

One of my colleagues recommends creating a copy of the database before starting work on a new feature. Then, when you switch branches, you can point the branch at the right database rather than roll back migrations.

Conclusion

Migrations change the structure of your database, whether that’s adding tables or altering columns. Changes to any field of your models will require a migration to keep your database in sync with your model.

Django can automate the creation and execution of migrations for you.

python manage.py makemigrations will create a file describing the migration in Python but will not execute changes to the database.

python manage.py migrate will execute the migration. Django compiles the migration code into SQL commands to run on the database.

Once migrations have been run, they should never be renamed or deleted. However, migrations are reversible. Once reversed, the migration files are safe to rename or delete.