Using ngrok to Simplify Developing for Alexa

I've recently been toying with building an Alexa skill. Not sure what exactly, something to goof around with the kids with. I was listening to a recent episode of Talk Python To Me and heard of a tool called ngrok which will make my development much easier.

Essentially ngrok creates a tunnel to your computer that is accessible on the internet. You launch ngrok and you get a URL you can access from anywhere. No need to setup port forwarding or anything in your router. This makes working with Alexa much easier, especially in the early stages when I'm trying things to see what works and what doesn't. It means I don't have to push to a public web server to see my changes.

All you need is a free ngrok account. I'll assume you can figure out how to setup an Alexa skill by following Amazon's instructions.

Ngrok Setup

Start by downloading and unzipping ngrok. Then connect your account by running the authtoken command. You can find the exact command you need by signing into your ngrok account and looking at your dashboard.

For Windows ngrok will just be an executable. So you need to either call it from the folder it is placed in or add it to your System Path. Also, for windows you don't need to preface each command with './'. Instead just call ngrok directly.

ngrok authtoken <your_auth_token_goes_here>

Next you just need to launch ngrok to create a new tunnel. I'm working with Flask-Ask and my dev server is running on my localhost on port 5000. So I use this command:

ngrok http 5000

A second or two later my tunnel is up and running and I can see the URL listed in the Forwarding area.

/images/ngrok_online.png

With my free account I get a new random URL each time I launch ngrok. This isn't a problem for what I'm doing here, but I can definitely see how useful it would be to have a paid account if you were doing client demos.

Alexa Setup

In your Alexa dashboard you need to make a couple of settings adjustments.

First, under the 'Configuration' tab set the the Endpoint to HTTPS and set the Default to the HTTPS url that NGROK just gave you. Since this URL changes each time you launch ngrok you'll need to update this field in your skill. Fortunately, this doesn't require any rebuild of the skill and the change is effective immediately.

/images/ngrok_alexa_setup_url.png

Next, under the 'SSL Certificate' tab be sure to pick the option for 'My development endpoint is a sub-domain of a domain that has a wildcard certificate from a certificate authority'. This lets you use the SSL certificate that ngrok provides for the random subdomain they gave you.

/images/ngrok_alexa_ssl_cert.png

If you don't set this option Amazon will refuse to even send traffic to your server. You'll get an error when you test it from the Skill's testing page.

The remote endpoint could not be called, or the response it returned was invalid.

Remember to update your URL in the 'Configuration' tab each time you relaunch ngrok and be sure to checkout the ngrok documentation. Their 'inspector' tool is really helpful to see the values of each request and response, especially when things go wrong.

Formatting Date Fields in Django

By default Django date fields populate with values in the yyyy-mm-dd format. Such as 2017-11-26. This isn't a format that many people actually use. The real problem comes if you want to use a date picker with the date field in your form. If the format used by the date picker, doesn't match the format used by the Django field, you'll get all kinds of issues.

Fortunately there is a really easy fix: set the date format you want in the form itself. As an example let's say we have a model for Appointments that has an appointment_date field.

forms.py

from django import forms
from models import Appointments

class AppointmentForm(forms.ModelForm):

    class Meta:
        model = Appointments
        fields = ['appointment_date', 'name', 'type_of_appointment']

    appointment_date = forms.DateField(
        widget=forms.DateInput(format='%m/%d/%Y'),
        input_formats=('%m/%d/%Y', )
        )

The widget attribute sets the display format for the field. The input_formats attribute lets you define the acceptable formats for date input. You can use all the standard Python string formatting options. You should match this to the format used by your date picker of choice.

While you are at it this can be a convenient place to add a class or two to the HTML so that your date picker can attach itself to your date fields. You'll place the code in the widget, since it defines the display on the page.

Something like this:

appointment_date = forms.DateField(
        widget=forms.DateInput(format='%m/%d/%Y', attrs={'class': 'datepicker'}),
        input_formats=('%m/%d/%Y', )
        )

Read more about Django widgets here.

Using Cookiecutter to Jumpstart a Django Project on Windows with PyCharm

I'm a big fan of automation. So I've been interested in the Cookiecutter project since I saw it at PyCon in 2016. When I decided to get more familiar with Django, I thought it would be a good chance to give it a spin.

This post will go over installing and using Cookiecutter, installing and setting up Postgresql as our database, configuring PyCharm to work with our project, and finally getting the whole thing running on the built-in development server. From there you can build your project from a solid foundation.

This article ended up longer than I expected. So long in fact it needs a table of contents.

Important

If you don't already have Python 3 installed, go to python.org and get it. I'll wait. Ready? Great.

Cookiecutter

/images/cookiecutter_logo.thumbnail.png

Cookiecutter is a tool that lets you define templates that can be used to create projects. It lets you capture input during the project creation and use that to customize the project files. It sounds simple, but it is really helpful, especially when combined with templates put together by experts in various tooling and frameworks. We'll get to that shortly. First, let's install Cookiecutter.

Cookiecutter is a Python project, so I'm going to assume you have Python 3 installed already. If not, do that. As a python package we can install it easily using pip. Normally I'd suggest installing packages into a virtualenv, but this is one of those tools you'll want available system-wide. From your command prompt run this to install the most recent version of Cookiecutter:

> pip install cookiecutter

Now that we have Cookiecutter installed we can move on to using our first cookiecutter.

Cookiecutter-Django

Cookiecutter-Django is a Cookiecutter, well, cookiecutter (template) for a well-configured Django installation. Both the Cookiecutter-Django and the Cookiecutter projects are from the team behind the Two Scoops of Django books. If you've read the book (and you should) you'll notice that the layout is similar, but not identical to what they describe there.

To get our Django project started open up your command prompt and move to the directory where you'd like the project installed. The template will create a new directory to hold all the project files. Once you are there run cookiecutter passing the location of the template you want to use as the first argument. There are shortcuts (like gh for github), but I find copying and pasting the URL easier and less error prone than remembering the shortcuts.

> cookiecutter https://github.com/pydanny/cookiecutter-django

Once you hit enter the fun begins. Cookiecutter will start asking you questions about your project. It uses the information you enter to decide what files to create and what values it should fill in. Most of them are self-explanatory, but I'll describe a few here:

  • project_name - The project name. It is used in some default text that gets displayed and in a few places.
  • project_slug - This gets used for folder names, the app name, database name, etc. This you want to keep one word, all letters, and all lowercase.
  • use_whitenoise - Whitenoise is a project that helps you serve static files (images, css, js, etc) from your app. You probably want to say Yes to this one.
  • use_sentry_for_error_reporting - Up to you. I'm saying no, I'll add it in later if I decide to use it.
  • use_pycharm - For this post, this needs to be a Yes.
  • windows - Are you developing on Windows (and it should be for this post)? Then this is a Yes as well.
  • postgresql_version - If you haven't installed Postgres yet (we'll get to it in a bit) pick 1 (the newest version). Otherwise select the one that matches what is already on your system.
  • js_task_runner - A couple of tools to choose from for helping you do some build and deployment automation (like minifying your javascript). If you don't have a preference go with the default.

Unless you got an error, you are done. Have a look around and if you want peek at the docs to see what got created.

PyCharm

I like PyCharm. I'm a big user of vim too, but I really like the help PyCharm gives me as an IDE. When we ran Cookicutter to create our project we said yes to use_pycharm. That told Cookiecutter to create some files for us that PyCharm expects to see. It is a good start, but we have some more work to do to get everything ready to go.

Open the Project

The first step is to open up our project in PyCharm. That is just as simple as selecting File - Open from the menu and picking our top-level project directory. PyCharm will start indexing the files and examining the structure.

Note

For now you can dismiss the version control warning. We'll come back to that later.

/images/pycharm_vcs_root_error.thumbnail.png

Setup the VirtualEnv

Right now we have a Python project with no Python interpreter setup to run it. We'll have PyCharm create a virtualenv to keep this project's requirements separate from any other project you may start.

  1. From the menu select File - Settings.

  2. Next, expand the Project: tab and click Project Interpreter.

  3. Click the gear icon gear_icon in the top-right corner and select Create VirtualEnv.

  4. In the popup give your virtualenv a name, pick a location (I like a single VirtualEnvs directory in my C:drive), select the python interpreter you want to use, leave the boxes unchecked, and click OK.

    create_virtualenv

Now we have a mostly empty Python environment. The built-in packages are available, but we don't have the libraries we are going to need for our project (like Django!).

Install Packages

There are two ways to install the packages we need. We can use the UI in PyCharm to select them one-by-one or we can use pip from the command line to install everything at once. We can do that because cookiecutter-django helpfully created a set of requirements files for us.

But before we get that far, lets take a look at the requirements. In your project you'll see a requirements directory and in that a list of files. The packages in base.txt get installed in all instances. These are included automatically in the local.txt, production.txt, and test.txt files. These each represent the packages needed in the given environment. You may need different packages in production than you do in development (local.txt) for example.

My version of cookiecutter-django wants to install Django 1.10. But I want to use Django 1.11 because it is available and is considered a long term support release. To get that version we'll open base.txt and edit the line like this and then save the file:

django==1.11.1

And we'll need the Python Postgres database adapter, too. Open up local.txt, add this line somewhere, and save it:

# Python Postgres database adapter
psycopg2==2.7.1

Open up the command terminal from inside PyCharm by clicking the Terminal terminal button at the bottom of the screen. If you don't see it, click the little icon in the lower-left ll_icon (I don't know what to call it) and select Terminal from the menu.

The important thing to notice is that not only is this the Windows command prompt, but that PyCharm has automatically launched our virtualenv for us. You can tell because the prompt is preceded by parens with the name of the virtualenv inside. That means that when we run pip commands they will take place inside the virtualenv and our packages will end up in the right place.

Let's install those packages:

(MyProject) C:\..\MyProject> pip install -r requirements\local.txt

All that output means that pip is downloading and installing the libraries we need. This is the beauty of open source and Python. Think of all the capabilities we just installed without having to write it and test it ourselves. And you can also see why we don't want to install them all one-by-one through the UI.

PostgreSQL

/images/postgres_elephant.thumbnail.png

PostgreSQL is an open source SQL database. It's Django's preferred database, and it's a good one, so lets use it. Yes, you could develop against something like SQLite, but it will save you headaches later if your development environment is as much like your deployment environment as possible. To that end we'll set up Postgres locally.

Download and Install

To begin, download the latest version of PostgreSQL for Windows. Once it has finished run through the installer. The defaults are fine. You will be asked to create a password for the postgres user. Make sure you remember this password. This is the master password for your database.

Now that we have our database server installed and running we need to create the database and users for our Django project. We can do this from the command line or through a tool like pgAdmin4.exe. Let's walk through both options.

Setup Database via Command Line

For this we'll use the terminal built into PyCharm again (but you could also use the normal command prompt). The first thing to note is that the PostgreSQL tools aren't in the system path so we can't call them just by name.

To create the user we'll move into the Postgres bin directory and then use createuser.exe.

>cd "c:\Program Files\PostgreSQL\9.6\bin"
>createuser.exe -P -U postgres YourProjectUserName
>Enter password for new role:
>Enter it again:
>Password:

This will create a new user called YourProjectUserName. The first time they ask for the password they want the password for the new user/role you are creating. The second time they want you to enter it again to make sure you really know it. The password they want at the 'Password:' prompt is looking for the password for the 'postgres' superuser. The one you created when you installed PostgreSQL.

Next we need to create a new database and make our new user the owner.

>cd "c:\Pogram Files\PostgreSQL\9.6\bin"
>createdb.exe createdb.exe -U postgres -O YourProjectUserName YourProjectDatabaseName

Again, the 'Password:' prompt is for the 'postgres' superuser. With that you will create a new database, here called 'YourProjectDatabaseName' with 'YourProjectUserName' as the owner.

Setup Database with pgAdmin

If the command line isn't your thing or you just can't seem to get it right, you can do all of this with the pgAdmin tool. Launch pgAdmin4.exe from c:Program FilespgAdmin 4bin. On my computer at least the tool feels a little slow, but it is useful to know how to work it.

In the top-left expand 'Servers'. Then right-click on your PostgreSQL server listed and select 'Connect Server'.

/images/postgres_connectserver.png

At the prompt enter the password for the 'postgres' superuser and click OK.

/images/postgres_loginpwd.png

Expand your server and right-click on Login/Group Roles and click Create - Login/Group Role.

/images/postgres_menu_createlogin.png

On the 'General' tab enter the user name. On the 'Definition' tab enter the password for the new user. On the 'Privileges' tab toggle 'Can login?' to Yes. The rest of the options on the remainder of the tabs are fine. Click Save to create your new user.

/images/postgres_newuser_general.thumbnail.png /images/postgres_newuser_definition.thumbnail.png /images/postgres_newuser_privileges.thumbnail.png

On to the database! Right-click on Databases, select Create - Database...

/images/postgres_menu_createdb.png

Enter your database's name and then select your user from the drop-down. Click Save and you're done.

/images/postgres_newdb.thumbnail.png

Final PyCharm Settings

Now that we have our database setup there are a few things left over we need to take care of in PyCharm. Cookiecutter-django creates almost everything we need for PyCharm, but not quite everything.

In the PyCharm open up File - Settings. Expand Languages and Frameworks and then Django.

/images/pycharm_django_settings.thumbnail.png

Check the box for 'Enable Django Support'. That tells PyCharm to do things like add a 'Run manage.py task' to the tools menu among other useful changes.

  • In the 'Django project root' find the path to the project folder created by cookiecutter-django.
  • In settings navigate to /config/settings/local.py inside your project folder. The local.py file is for our development environment, so that is the one we want.
  • It should find the path to your manage.py automatically.

Cookiecutter-django follows best practices and prefers to store settings in environment variables rather than configuration files whenever possible. It makes deploying and testing easier and it helps prevent accidentally exposing secrets by committing a config file to an open source repository. PyCharm will set environmental variables for us when we run the Django server. To give it values click the [...] button to the far right of 'Environment variables'.

The only variable we need to set right now is DATABASE_URL. The value will be a url string that includes all the details we need to connect to our database: * username * password * host * port * database name

That's a lot. It will look like this:

# format
postgres://username:password@host:port/database

# filled with example data
postgres://MyUser:MyPassword@127.0.0.1:5432/MyDatabase

Click the green plus and then fill in the name and value and click OK.

/images/pycharm_django_env.thumbnail.png

Run the Server

Finally! We are ready to test everything. From the menu select Tools - Run manage.py Task. It will open up a new console at the bottom of the PyCharm window. In that console run the 'migrate' command. That will configure our database schema for us. If you got errors, check that your DATABASE_URL string is absolutely correct. With luck the error message will point you in the right direction.

Once that is complete run the 'runserver' command. That will start up the Django development server and we can finally see Django running in our browser. Just click the link in the output.

/images/pycharm_runserver_output.thumbnail.png

Optional

Now that everything works there are a few optional things to take care of.

Removing Docker Run Configurations

Even though I answered No to Docker, a handful of Docker run configurations got created. I don't need them, so best to get rid of them and keep things tidy. To do that go to Run - Edit Configuration. Under Django Server highlight the configurations and click the red minus button to delete them.

/images/pycharm_delete_docker_configs.png

Setting up Version Control

We're finally ready to fix that version control error we dismissed earlier. If you want version control (and you probably do) the easiest thing to do is select VCS - Import into Version Control and pick either Create Git Repository or Create Mercurial Repository depending on your VCS of choice. Follow the prompts. They are slightly different, but pretty self explanatory.

Your First Pull Request

Finding a bug in an open source software library or application is frustrating. Finding the error and coming up with a solution is exciting. Submitting a pull request (PR) to the maintainer is daunting.

Setup GitHub

Chances are good that if you are working with an open source project they've hosted it on GitHub. The first thing to do is get yourself a free GitHub account.

Fork the Repository

Go to the GitHub page for the project you want to fix. Once there click the <Fork> button.

/images/fork_button.thumbnail.png

This will give you a copy of the entire source code where you can fix the bugs and run the tests. You will be automatically redirected to your new copy of the repo. You'll notice that it shows you the project your forked from and that you have an exact copy of everything, including the entire history of changes made to the code.

Install Git

You'll need Git installed on your computer to track and commit the changes you make to any of the code. Depending on your OS you may already have it installed. If not, download and install it.

Clone the Repo

Now that you have a copy of the code in your own repo, you need a copy of the repo on your computer where you can edit it, run it, and test it. To do that we need to clone the repository.

First create an empty directory where we'll clone your repository.

In your command prompt or terminal navigate into the empty folder you created and clone your repository:

git clone https://github.com/your_github_username/your_repository_name.git

You'll see something like this:

Cloning into 'your_repo_name'...
remote: Counting objects: 1926, done.
remote: Compressing objects: 100% (68/68), done.
remote: Total 1926 (delta 37), reused 0 (delta 0), pack-reused 1858
Receiving objects: 100% (1926/1926), 1.59 MiB | 667.00 KiB/s, done.
Resolving deltas: 100% (1164/1164), done.

If you haven't yet you will need to set some basic info into your git config. You can't commit changes until you do.

git config --global user.email "your_email"
git config --global user.name "your name"

Make The Fix

Now that you have a local copy of the code you can begin editing it. Go ahead and make the changes you need to make and test those changes. Testing may be as simple as manually checking the thing you noticed was broken, but it may also include running the project's test suite. You may also need to write a few tests to make sure that the thing you just fixed is not broken in future releases. Doing that depends a great deal on the language and testing framework being used. If you aren't sure, reach out to the project maintainer for guidance.

With the code (and possibly test) changes made you need to commit your changes and then push them to the repository you created on GitHub.

Start by adding the files you edited or created. For each file you added or changed run this command.

git add your_file_name

Next commit the changes you've staged. Add a meaningful message that describes the change.

git commit -m "Your commit message goes here"

Now we are ready to push the changes from your local repository to your remote repository on GitHub. You may be asked for your username and password. Enter those at the prompts.

git push https://github.com/your_github_username/your_repository_name.git

Issue a Pull Request

A pull request is a way of asking the project maintainer to add your code changes to the master repository that others will work from and use. It lets you pick just the change set you want.

From GitHub open the project you want where you want your code included. Click the Pull Requests tab and then click the New pull request button.

/images/pull_request.thumbnail.png

Because you want to have your code changes included in another project click the link for 'compare across forks'. This will give you the chance to pick both the original repository your forked from and your repository with changes.

/images/pull_request_across_forks.thumbnail.png

In the base fork pick the project in which you want to include your code change. Chances are you want it in the master fork. For head fork pick your repository and compare against the master. With any luck you'll see that you are 'Able to merge'. This means that you don't have any code changes that conflict with code changes made by someone else in the time between you forking the repository and filing your pull request. It doesn't mean there are guaranteed not to be any problems, but it does mean that you didn't change the same code as someone else.

/images/create_pull_request_after_selecting.thumbnail.png

Now you are ready to finalize your request. Start by clicking Create pull request. Add some information to give the person reviewing your request an explanation as to what you were fixing and how you fixed it. Once you've given the maintainer as much info as you can click Create pull request.

What Happens Now

At this point the project maintainer will get notified that a pull request has been created. They will see the changes you made and can review them to make sure they want to include them in the source code.

In addition to manual review, your version of the code may be run through something like Travis CI. Continuous integration tools like this will run automated tests to determine if your change breaks anything. It may test the code on multiple platforms or multiple versions of programming language looking for problems.

Four things can happen next:

Acceptance

The maintainer may accept and merge your pull request. This might even happen automatically if your change passes the continuous integration testing.

Feedback

Or they may come back to you with questions or comments. They might request that you change the code you submitted. They might want you to follow a specific style or make the change elsewhere in the code base. Perhaps they require documentation to go along with the change you made.

Rejection

Or they may reject your pull request. It might be that you tried to add a feature they don't want included. Or they may want more thorough unit tests. It could be that while you really did find a bug, that bug has now become a 'feature' and fixing it would cause problems to users they aren't ready to deal with yet. If your pull request is rejected it is okay to start a friendly conversation to find out why. You can always continue using your code in your projects.

Nothing

Finally, they may do nothing. Remember that as a rule the maintainer of the project you are trying to fix isn't getting paid to do this work. They have busy lives and other obligations. Be patient.

Using Git Hooks To Help Enforce Our SOP

This week at work I moved all of our git repositories from a GitHub account owned by one of our employees to one owned and managed by the company. That was easy enough, but reminded me that we've been using git commit hooks to make sure we include our ticket/issue number in the commit message. This lets us link Jira with our commits. Trouble is, I couldn't remember how exactly I did it. I should have written it up last time, so I'm writing it up this time.

Git Hooks

Git can run commands at a number of different events using hooks. The 'hook' files are located in your repository's .git/hooks directory. By default there are sample files for each event. These files make good starting place. To use one, just rename the file to remove the '.sample' from the end of the filename.

The bundled example files are Bash shell scripts. They work on Windows as well as Linux machines because the standard Windows installer for git includes a Bash emulator by default. You can write the scripts in any language you like (such as Python) so long as your system can run them as executable files.

Requiring an Issue Number

Our SOP requires us to include the ticket number we are working on as part of each commit message. You could get creative with this using regular expressions. But I wanted to keep it as simple as possible. I'm just doing simple text matching.

My 'commit-msg' looks like this:

 #!/bin/sh
 # This hook gets run after the commit message is entered, but before the commit
 # is made. If this fails, the commit will not take place. We are using it to:
 #    * Check that the first line of the commit message includes the issue number prefix

 # The path to the file with the commit message is passed in as the first argument
 file_path=$1

 # Get the first line of the file
 first_line=$(head -n 1 $file_path)

 # Check that the first line includes the text we want
 if [[ "$first_line" == *FW-* || "$first_line" == *TD-* ]]
 then
     exit 0
 else
     # The 'echos' will be displayed in the error message. The 'exit 1' tells git this is
     # a failure.
     echo "You must include the Jira issue number in the first line of the commit message."
     echo "The issue number must start with FW- or TD- and must be in all caps."
     exit 1
 fi

If you are familiar with Bash scripts this is pretty straight forward. If not, I'll walk through it below. The key is that this will get called before a commit is accepted. If the file returns a zero the event, in this case a commit, is accepted. If it returns a one it is rejected. You can do whatever you like in these scripts.

The breakdown

file_path=$1

The path to the commit message file is being passed in as the first argument to the script. We can use that to get the contents of the file. In this case I want the ticket number in the first line of the commit message. We get that here:

first_line=$(head -n 1 $file_path)

Next we check to see if the first line contains either "FW-" or "TD-". These are the two valid prefixes for tickets in the Jira instance we use at work. You could look for any string or strings. Or get fancier using some regex.

if [[ "$first_line" == *FW-* || "$first_line" == *TD-* ]]

If we find what we are looking for we return a '0'. This tells git all is well and to keep going with the action.

then
     exit 0

If not, we pop up a message to the user so they know their commit is about to be rejected. In this case the user is me, so I make it very specific and clear. After that we return a zero so git knows that it should reject the commit.

else
     # The 'echos' will be displayed in the error message. The 'exit 1' tells git this is
     # a failure.
     echo "You must include the Jira issue number in the first line of the commit message."
     echo "The issue number must start with FW- or TD- and must be in all caps."
     exit 1
 fi

All the git hooks work in the same way. They get called by the action indicated in the file name. The script does some work and returns a zero (success) or a one (failure). You can run hooks on both the client-side and the server-side. More info on the hooks can be found in the git documentation.

A few things to note

  • You can, and probably should, include your git hooks in your repo. Put a copy of each into another directory and commit them like you would other files. Then you or another developer will get the copy of every hook they'll need to when working with your repo. They'll still need to put them in the correct location, but they'll have what they need.
  • These are client-side only. This means that you aren't forced to use them. This helps remind a developer that they've made a mistake, but doesn't actually enforce the restriction. In this case, if someone doesn't use the hook, they can commit all they want without including a ticket number. That is where server-side hooks come in. These can be used to enforce the rules. It is probably best to use client-side and server-side scripts together to inform the developer of issues and make sure nothing gets committed that shouldn't.
  • The hook files have no file extension. This is weird for Windows users, but fear not, they are just regular text files and can be edited using any text editor.