In this post, we will go through how to add a non-nullable field and handle the existing rows. Instead of providing a default value or forcing the column to accept null values, we will automatically add values to existing rows, where each value is specific to that row.

If you're still building confidence handling migrations, I recommend reading my beginner's guide to migrations, where you will learn how to avoid database errors.

The problem

I wanted to add a slug field to an existing model.

It’s important that I don’t allow blank values, as the slug will be used for URLs.

But when I tried running python manage.py makemigrations , I got this error:

It is impossible to add a non-nullable field 'slug' to movie without specifying a default. This is because the database needs something to populate existing rows.
Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
 2) Quit and manually define a default value in models.py.

These are my options:

1. Assign an arbitrary default

I’d rather not as I’d be deliberately assigning invalid values to my existing data. I could run a script after to assign the correct slug to each existing movie, but I’d rather not have the extra step. It creates confusion if you’re trying to set the project up on another machine.

2. Allow blank values

I’ve done this a few times in personal projects as a quick fix but it’s not ideal. If I see blank=True in a model, then that tells me that blank values are acceptable. Slugs are used to construct URLs, so blank values are definitely not acceptable, in this case.

3. Drop the database

This is something to avoid as much as possible. Deleting the db.sqlite3 file, rerunning your migrations, and recreate all your data can seem tempting when you’re in a mess, but it takes so much effort to get your database contents to the same state it was before. Once your product has been deployed, dropping the database will never be an option, so I say learn to avoid this sooner rather than later.

4. Customise your migration

This is where you add code to your migration to assign the correct value for each existing row. If you’re working with another developer or on multiple machines, this approach will get the results you need in one step, without running additional scripts.

How do we do this?

We are going to temporarily assign an arbitrary default value.

Then we’re going to add an operation to our migration that will go through each of our existing rows and update the new slug field with the correct value.

Let's get started.

How to customise a migration

1. Create your migration

Start by running python manage.py makemigration

When you get the following prompt, press 1:

Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
 2) Quit and manually define a default value in models.py.

Provide any value. As the column I’m adding is a string, I’ve chosen “abc”. This will give Django what it needs to create the migration. We're going to customise the migration before we run it. Our Python code will overwrite any column with the "abc" default.

2. Customise the Migration

For now, I’ve set a default of “abc” that I don’t want applied to the database. I’ve done this to appease Django to create the migration, but we’ll make sure “abc” doesn’t get applied.

Django migrations contain a list called operations. The migration is created with a single operation: migrations.AlterField. What we’re going to do is add a second operation that will go through each of our movies, calculate the correct slug from the movie’s title and update the movie. Operations are executed in order, so we'll update each movie's slug after the field has been created.

Write the custom code

We need to write two functions: one that will update the movie with our desired slug and one that will restore the slug to the default value.

We do this because migrations should always be reversible. When we run the migration, forwards will be called. Should we ever need to reverse the migration, backwards is called.

Add the operation

Add the following to operations :

migrations.RunPython(forwards, backwards)

See the gist below for the full code of the migration:

gist.github.com/aliceridgway/4af16fe3b45c49..

3. Run the Migration

The migration is now ready to run. Run python manage.py migrate in your terminal to run the migration.

Result

Go into admin to check that the correct value for the new field has been applied to existing rows.

Conclusion

We have shown that you don’t have to drop your database or compromise on defaults when you want to add a non-nullable field to one of your models.

By understanding how Django migrations execute operations in order, we can use migrations.RunPython(forwards, backwards) to add custom code. This is useful when we want to add a field, and give each existing row a custom value.