Using Your Django Custom User Model as Foreign Key

The Django docs recommend that when you create a new project you use a custom user model. Doing so lets you do things like add fields to store information about your users quite easily.

There are at least 4 ways to extend the user model on a new project. Each has its own advantages. On an existing project you might try this approach to extend the user model.

If you are using django-cookiecutter like me, you'll get a custom user model automatically. In that case you'll get a custom user model based on the AbstractUser class. You'll find it in {your_app}.users.models and it will look something like this:

class User(AbstractUser):
    class Meta:
        unique_together = (("name", "agency"),)
    # First Name and Last Name do not cover name patterns
    # around the globe.
    name = models.CharField(_('Name of User'), blank=True, max_length=255)
    is_active = models.BooleanField(_('active'), default=True,
                                    help_text=_('Designates whether this user should be treated as '
                                                'active. Unselect this instead of deleting accounts.'))

Adding a new field is as easy as defining it in the class and running the migrations. Simple. But what if you want create a foreign key relationship between the User model and another model you define?

When I tried to do something like this:

class SomeRelatedModel(models.Model):
    user = models.ForeignKey('User' on_delete=models.CASCADE, )

I received this error when I ran my migrations: Field defines a relation with model 'User', which is either not installed, or is abstract.

When I try to import the model first and then use it I get this error: ImportError: cannot import name 'User'. No matter how I did my imports I couldn't get it working correctly.

So what is the solution? It is in the docs of course where they tell us "When you define a foreign key or many-to-many relations to the user model, you should specify the custom model using the AUTH_USER_MODEL setting." In my example that would look like this:

from django.conf import settings

class SomeRelatedModel(models.Model):
    user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE,)

Changing a Django Model Breaks All Your Tests

I like to think that I learn from my mistakes. But there are a few things that creep up and bite me over and over. This is one of those errors. It gets me when I'm tired or distracted.

If you make a change to a Django model, and you do not create the migrations, when you run your tests all of your database tests will fail. If you are like me, you will experience a moment of deep despair as everything that used to work is now broken and the universe makes no sense. You'll think to yourself "What could I possibly have done to break everything?"

For me, running my tests with PyTest and working with PostgreSQL, after a long list of otuput I see an error like this one:

psycopg2.OperationalError: cursor "_django_curs_7196_10" does not exist

The cursor number is always different, but the cause is always the same. I made a change to some model and didn't make the migrations. It could be adding a new model, adding a field, or making some other model change. It will break all your database-backed tests even if they don't touch the changed model.

Luckily, the fix is easy. Just run makemigrations.

manage.py makemigrations

Rerun your tests, they pass, and suddenly the universe makes sense again.

Using ngrok to Simplify Developing for Alexa

I've recently been toying with building an Alexa skill. Not sure what exactly, something to goof around with the kids with. I was listening to a recent episode of Talk Python To Me and heard of a tool called ngrok which will make my development much easier.

Essentially ngrok creates a tunnel to your computer that is accessible on the internet. You launch ngrok and you get a URL you can access from anywhere. No need to setup port forwarding or anything in your router. This makes working with Alexa much easier, especially in the early stages when I'm trying things to see what works and what doesn't. It means I don't have to push to a public web server to see my changes.

All you need is a free ngrok account. I'll assume you can figure out how to setup an Alexa skill by following Amazon's instructions.

Ngrok Setup

Start by downloading and unzipping ngrok. Then connect your account by running the authtoken command. You can find the exact command you need by signing into your ngrok account and looking at your dashboard.

For Windows ngrok will just be an executable. So you need to either call it from the folder it is placed in or add it to your System Path. Also, for windows you don't need to preface each command with './'. Instead just call ngrok directly.

ngrok authtoken <your_auth_token_goes_here>

Next you just need to launch ngrok to create a new tunnel. I'm working with Flask-Ask and my dev server is running on my localhost on port 5000. So I use this command:

ngrok http 5000

A second or two later my tunnel is up and running and I can see the URL listed in the Forwarding area.

/images/ngrok_online.png

With my free account I get a new random URL each time I launch ngrok. This isn't a problem for what I'm doing here, but I can definitely see how useful it would be to have a paid account if you were doing client demos.

Alexa Setup

In your Alexa dashboard you need to make a couple of settings adjustments.

First, under the 'Configuration' tab set the the Endpoint to HTTPS and set the Default to the HTTPS url that NGROK just gave you. Since this URL changes each time you launch ngrok you'll need to update this field in your skill. Fortunately, this doesn't require any rebuild of the skill and the change is effective immediately.

/images/ngrok_alexa_setup_url.png

Next, under the 'SSL Certificate' tab be sure to pick the option for 'My development endpoint is a sub-domain of a domain that has a wildcard certificate from a certificate authority'. This lets you use the SSL certificate that ngrok provides for the random subdomain they gave you.

/images/ngrok_alexa_ssl_cert.png

If you don't set this option Amazon will refuse to even send traffic to your server. You'll get an error when you test it from the Skill's testing page.

The remote endpoint could not be called, or the response it returned was invalid.

Remember to update your URL in the 'Configuration' tab each time you relaunch ngrok and be sure to checkout the ngrok documentation. Their 'inspector' tool is really helpful to see the values of each request and response, especially when things go wrong.

Formatting Date Fields in Django

By default Django date fields populate with values in the yyyy-mm-dd format. Such as 2017-11-26. This isn't a format that many people actually use. The real problem comes if you want to use a date picker with the date field in your form. If the format used by the date picker, doesn't match the format used by the Django field, you'll get all kinds of issues.

Fortunately there is a really easy fix: set the date format you want in the form itself. As an example let's say we have a model for Appointments that has an appointment_date field.

forms.py

from django import forms
from models import Appointments

class AppointmentForm(forms.ModelForm):

    class Meta:
        model = Appointments
        fields = ['appointment_date', 'name', 'type_of_appointment']

    appointment_date = forms.DateField(
        widget=forms.DateInput(format='%m/%d/%Y'),
        input_formats=('%m/%d/%Y', )
        )

The widget attribute sets the display format for the field. The input_formats attribute lets you define the acceptable formats for date input. You can use all the standard Python string formatting options. You should match this to the format used by your date picker of choice.

While you are at it this can be a convenient place to add a class or two to the HTML so that your date picker can attach itself to your date fields. You'll place the code in the widget, since it defines the display on the page.

Something like this:

appointment_date = forms.DateField(
        widget=forms.DateInput(format='%m/%d/%Y', attrs={'class': 'datepicker'}),
        input_formats=('%m/%d/%Y', )
        )

Read more about Django widgets here.

Using Cookiecutter to Jumpstart a Django Project on Windows with PyCharm

I'm a big fan of automation. So I've been interested in the Cookiecutter project since I saw it at PyCon in 2016. When I decided to get more familiar with Django, I thought it would be a good chance to give it a spin.

This post will go over installing and using Cookiecutter, installing and setting up Postgresql as our database, configuring PyCharm to work with our project, and finally getting the whole thing running on the built-in development server. From there you can build your project from a solid foundation.

This article ended up longer than I expected. So long in fact it needs a table of contents.

Important

If you don't already have Python 3 installed, go to python.org and get it. I'll wait. Ready? Great.

Cookiecutter

/images/cookiecutter_logo.thumbnail.png

Cookiecutter is a tool that lets you define templates that can be used to create projects. It lets you capture input during the project creation and use that to customize the project files. It sounds simple, but it is really helpful, especially when combined with templates put together by experts in various tooling and frameworks. We'll get to that shortly. First, let's install Cookiecutter.

Cookiecutter is a Python project, so I'm going to assume you have Python 3 installed already. If not, do that. As a python package we can install it easily using pip. Normally I'd suggest installing packages into a virtualenv, but this is one of those tools you'll want available system-wide. From your command prompt run this to install the most recent version of Cookiecutter:

> pip install cookiecutter

Now that we have Cookiecutter installed we can move on to using our first cookiecutter.

Cookiecutter-Django

Cookiecutter-Django is a Cookiecutter, well, cookiecutter (template) for a well-configured Django installation. Both the Cookiecutter-Django and the Cookiecutter projects are from the team behind the Two Scoops of Django books. If you've read the book (and you should) you'll notice that the layout is similar, but not identical to what they describe there.

To get our Django project started open up your command prompt and move to the directory where you'd like the project installed. The template will create a new directory to hold all the project files. Once you are there run cookiecutter passing the location of the template you want to use as the first argument. There are shortcuts (like gh for github), but I find copying and pasting the URL easier and less error prone than remembering the shortcuts.

> cookiecutter https://github.com/pydanny/cookiecutter-django

Once you hit enter the fun begins. Cookiecutter will start asking you questions about your project. It uses the information you enter to decide what files to create and what values it should fill in. Most of them are self-explanatory, but I'll describe a few here:

  • project_name - The project name. It is used in some default text that gets displayed and in a few places.
  • project_slug - This gets used for folder names, the app name, database name, etc. This you want to keep one word, all letters, and all lowercase.
  • use_whitenoise - Whitenoise is a project that helps you serve static files (images, css, js, etc) from your app. You probably want to say Yes to this one.
  • use_sentry_for_error_reporting - Up to you. I'm saying no, I'll add it in later if I decide to use it.
  • use_pycharm - For this post, this needs to be a Yes.
  • windows - Are you developing on Windows (and it should be for this post)? Then this is a Yes as well.
  • postgresql_version - If you haven't installed Postgres yet (we'll get to it in a bit) pick 1 (the newest version). Otherwise select the one that matches what is already on your system.
  • js_task_runner - A couple of tools to choose from for helping you do some build and deployment automation (like minifying your javascript). If you don't have a preference go with the default.

Unless you got an error, you are done. Have a look around and if you want peek at the docs to see what got created.

PyCharm

I like PyCharm. I'm a big user of vim too, but I really like the help PyCharm gives me as an IDE. When we ran Cookicutter to create our project we said yes to use_pycharm. That told Cookiecutter to create some files for us that PyCharm expects to see. It is a good start, but we have some more work to do to get everything ready to go.

Open the Project

The first step is to open up our project in PyCharm. That is just as simple as selecting File - Open from the menu and picking our top-level project directory. PyCharm will start indexing the files and examining the structure.

Note

For now you can dismiss the version control warning. We'll come back to that later.

/images/pycharm_vcs_root_error.thumbnail.png

Setup the VirtualEnv

Right now we have a Python project with no Python interpreter setup to run it. We'll have PyCharm create a virtualenv to keep this project's requirements separate from any other project you may start.

  1. From the menu select File - Settings.

  2. Next, expand the Project: tab and click Project Interpreter.

  3. Click the gear icon gear_icon in the top-right corner and select Create VirtualEnv.

  4. In the popup give your virtualenv a name, pick a location (I like a single VirtualEnvs directory in my C:drive), select the python interpreter you want to use, leave the boxes unchecked, and click OK.

    create_virtualenv

Now we have a mostly empty Python environment. The built-in packages are available, but we don't have the libraries we are going to need for our project (like Django!).

Install Packages

There are two ways to install the packages we need. We can use the UI in PyCharm to select them one-by-one or we can use pip from the command line to install everything at once. We can do that because cookiecutter-django helpfully created a set of requirements files for us.

But before we get that far, lets take a look at the requirements. In your project you'll see a requirements directory and in that a list of files. The packages in base.txt get installed in all instances. These are included automatically in the local.txt, production.txt, and test.txt files. These each represent the packages needed in the given environment. You may need different packages in production than you do in development (local.txt) for example.

My version of cookiecutter-django wants to install Django 1.10. But I want to use Django 1.11 because it is available and is considered a long term support release. To get that version we'll open base.txt and edit the line like this and then save the file:

django==1.11.1

And we'll need the Python Postgres database adapter, too. Open up local.txt, add this line somewhere, and save it:

# Python Postgres database adapter
psycopg2==2.7.1

Open up the command terminal from inside PyCharm by clicking the Terminal terminal button at the bottom of the screen. If you don't see it, click the little icon in the lower-left ll_icon (I don't know what to call it) and select Terminal from the menu.

The important thing to notice is that not only is this the Windows command prompt, but that PyCharm has automatically launched our virtualenv for us. You can tell because the prompt is preceded by parens with the name of the virtualenv inside. That means that when we run pip commands they will take place inside the virtualenv and our packages will end up in the right place.

Let's install those packages:

(MyProject) C:\..\MyProject> pip install -r requirements\local.txt

All that output means that pip is downloading and installing the libraries we need. This is the beauty of open source and Python. Think of all the capabilities we just installed without having to write it and test it ourselves. And you can also see why we don't want to install them all one-by-one through the UI.

PostgreSQL

/images/postgres_elephant.thumbnail.png

PostgreSQL is an open source SQL database. It's Django's preferred database, and it's a good one, so lets use it. Yes, you could develop against something like SQLite, but it will save you headaches later if your development environment is as much like your deployment environment as possible. To that end we'll set up Postgres locally.

Download and Install

To begin, download the latest version of PostgreSQL for Windows. Once it has finished run through the installer. The defaults are fine. You will be asked to create a password for the postgres user. Make sure you remember this password. This is the master password for your database.

Now that we have our database server installed and running we need to create the database and users for our Django project. We can do this from the command line or through a tool like pgAdmin4.exe. Let's walk through both options.

Setup Database via Command Line

For this we'll use the terminal built into PyCharm again (but you could also use the normal command prompt). The first thing to note is that the PostgreSQL tools aren't in the system path so we can't call them just by name.

To create the user we'll move into the Postgres bin directory and then use createuser.exe.

>cd "c:\Program Files\PostgreSQL\9.6\bin"
>createuser.exe -P -U postgres YourProjectUserName
>Enter password for new role:
>Enter it again:
>Password:

This will create a new user called YourProjectUserName. The first time they ask for the password they want the password for the new user/role you are creating. The second time they want you to enter it again to make sure you really know it. The password they want at the 'Password:' prompt is looking for the password for the 'postgres' superuser. The one you created when you installed PostgreSQL.

Next we need to create a new database and make our new user the owner.

>cd "c:\Pogram Files\PostgreSQL\9.6\bin"
>createdb.exe createdb.exe -U postgres -O YourProjectUserName YourProjectDatabaseName

Again, the 'Password:' prompt is for the 'postgres' superuser. With that you will create a new database, here called 'YourProjectDatabaseName' with 'YourProjectUserName' as the owner.

Setup Database with pgAdmin

If the command line isn't your thing or you just can't seem to get it right, you can do all of this with the pgAdmin tool. Launch pgAdmin4.exe from c:Program FilespgAdmin 4bin. On my computer at least the tool feels a little slow, but it is useful to know how to work it.

In the top-left expand 'Servers'. Then right-click on your PostgreSQL server listed and select 'Connect Server'.

/images/postgres_connectserver.png

At the prompt enter the password for the 'postgres' superuser and click OK.

/images/postgres_loginpwd.png

Expand your server and right-click on Login/Group Roles and click Create - Login/Group Role.

/images/postgres_menu_createlogin.png

On the 'General' tab enter the user name. On the 'Definition' tab enter the password for the new user. On the 'Privileges' tab toggle 'Can login?' to Yes. The rest of the options on the remainder of the tabs are fine. Click Save to create your new user.

/images/postgres_newuser_general.thumbnail.png /images/postgres_newuser_definition.thumbnail.png /images/postgres_newuser_privileges.thumbnail.png

On to the database! Right-click on Databases, select Create - Database...

/images/postgres_menu_createdb.png

Enter your database's name and then select your user from the drop-down. Click Save and you're done.

/images/postgres_newdb.thumbnail.png

Final PyCharm Settings

Now that we have our database setup there are a few things left over we need to take care of in PyCharm. Cookiecutter-django creates almost everything we need for PyCharm, but not quite everything.

In the PyCharm open up File - Settings. Expand Languages and Frameworks and then Django.

/images/pycharm_django_settings.thumbnail.png

Check the box for 'Enable Django Support'. That tells PyCharm to do things like add a 'Run manage.py task' to the tools menu among other useful changes.

  • In the 'Django project root' find the path to the project folder created by cookiecutter-django.
  • In settings navigate to /config/settings/local.py inside your project folder. The local.py file is for our development environment, so that is the one we want.
  • It should find the path to your manage.py automatically.

Cookiecutter-django follows best practices and prefers to store settings in environment variables rather than configuration files whenever possible. It makes deploying and testing easier and it helps prevent accidentally exposing secrets by committing a config file to an open source repository. PyCharm will set environmental variables for us when we run the Django server. To give it values click the [...] button to the far right of 'Environment variables'.

The only variable we need to set right now is DATABASE_URL. The value will be a url string that includes all the details we need to connect to our database: * username * password * host * port * database name

That's a lot. It will look like this:

# format
postgres://username:password@host:port/database

# filled with example data
postgres://MyUser:MyPassword@127.0.0.1:5432/MyDatabase

Click the green plus and then fill in the name and value and click OK.

/images/pycharm_django_env.thumbnail.png

Run the Server

Finally! We are ready to test everything. From the menu select Tools - Run manage.py Task. It will open up a new console at the bottom of the PyCharm window. In that console run the 'migrate' command. That will configure our database schema for us. If you got errors, check that your DATABASE_URL string is absolutely correct. With luck the error message will point you in the right direction.

Once that is complete run the 'runserver' command. That will start up the Django development server and we can finally see Django running in our browser. Just click the link in the output.

/images/pycharm_runserver_output.thumbnail.png

Optional

Now that everything works there are a few optional things to take care of.

Removing Docker Run Configurations

Even though I answered No to Docker, a handful of Docker run configurations got created. I don't need them, so best to get rid of them and keep things tidy. To do that go to Run - Edit Configuration. Under Django Server highlight the configurations and click the red minus button to delete them.

/images/pycharm_delete_docker_configs.png

Setting up Version Control

We're finally ready to fix that version control error we dismissed earlier. If you want version control (and you probably do) the easiest thing to do is select VCS - Import into Version Control and pick either Create Git Repository or Create Mercurial Repository depending on your VCS of choice. Follow the prompts. They are slightly different, but pretty self explanatory.