Your First Pull Request

Finding a bug in an open source software library or application is frustrating. Finding the error and coming up with a solution is exciting. Submitting a pull request (PR) to the maintainer is daunting.

Setup GitHub

Chances are good that if you are working with an open source project they've hosted it on GitHub. The first thing to do is get yourself a free GitHub account.

Fork the Repository

Go to the GitHub page for the project you want to fix. Once there click the <Fork> button.

/images/fork_button.thumbnail.png

This will give you a copy of the entire source code where you can fix the bugs and run the tests. You will be automatically redirected to your new copy of the repo. You'll notice that it shows you the project your forked from and that you have an exact copy of everything, including the entire history of changes made to the code.

Install Git

You'll need Git installed on your computer to track and commit the changes you make to any of the code. Depending on your OS you may already have it installed. If not, download and install it.

Clone the Repo

Now that you have a copy of the code in your own repo, you need a copy of the repo on your computer where you can edit it, run it, and test it. To do that we need to clone the repository.

First create an empty directory where we'll clone your repository.

In your command prompt or terminal navigate into the empty folder you created and clone your repository:

git clone https://github.com/your_github_username/your_repository_name.git

You'll see something like this:

Cloning into 'your_repo_name'...
remote: Counting objects: 1926, done.
remote: Compressing objects: 100% (68/68), done.
remote: Total 1926 (delta 37), reused 0 (delta 0), pack-reused 1858
Receiving objects: 100% (1926/1926), 1.59 MiB | 667.00 KiB/s, done.
Resolving deltas: 100% (1164/1164), done.

If you haven't yet you will need to set some basic info into your git config. You can't commit changes until you do.

git config --global user.email "your_email"
git config --global user.name "your name"

Make The Fix

Now that you have a local copy of the code you can begin editing it. Go ahead and make the changes you need to make and test those changes. Testing may be as simple as manually checking the thing you noticed was broken, but it may also include running the project's test suite. You may also need to write a few tests to make sure that the thing you just fixed is not broken in future releases. Doing that depends a great deal on the language and testing framework being used. If you aren't sure, reach out to the project maintainer for guidance.

With the code (and possibly test) changes made you need to commit your changes and then push them to the repository you created on GitHub.

Start by adding the files you edited or created. For each file you added or changed run this command.

git add your_file_name

Next commit the changes you've staged. Add a meaningful message that describes the change.

git commit -m "Your commit message goes here"

Now we are ready to push the changes from your local repository to your remote repository on GitHub. You may be asked for your username and password. Enter those at the prompts.

git push https://github.com/your_github_username/your_repository_name.git

Issue a Pull Request

A pull request is a way of asking the project maintainer to add your code changes to the master repository that others will work from and use. It lets you pick just the change set you want.

From GitHub open the project you want where you want your code included. Click the Pull Requests tab and then click the New pull request button.

/images/pull_request.thumbnail.png

Because you want to have your code changes included in another project click the link for 'compare across forks'. This will give you the chance to pick both the original repository your forked from and your repository with changes.

/images/pull_request_across_forks.thumbnail.png

In the base fork pick the project in which you want to include your code change. Chances are you want it in the master fork. For head fork pick your repository and compare against the master. With any luck you'll see that you are 'Able to merge'. This means that you don't have any code changes that conflict with code changes made by someone else in the time between you forking the repository and filing your pull request. It doesn't mean there are guaranteed not to be any problems, but it does mean that you didn't change the same code as someone else.

/images/create_pull_request_after_selecting.thumbnail.png

Now you are ready to finalize your request. Start by clicking Create pull request. Add some information to give the person reviewing your request an explanation as to what you were fixing and how you fixed it. Once you've given the maintainer as much info as you can click Create pull request.

What Happens Now

At this point the project maintainer will get notified that a pull request has been created. They will see the changes you made and can review them to make sure they want to include them in the source code.

In addition to manual review, your version of the code may be run through something like Travis CI. Continuous integration tools like this will run automated tests to determine if your change breaks anything. It may test the code on multiple platforms or multiple versions of programming language looking for problems.

Four things can happen next:

Acceptance

The maintainer may accept and merge your pull request. This might even happen automatically if your change passes the continuous integration testing.

Feedback

Or they may come back to you with questions or comments. They might request that you change the code you submitted. They might want you to follow a specific style or make the change elsewhere in the code base. Perhaps they require documentation to go along with the change you made.

Rejection

Or they may reject your pull request. It might be that you tried to add a feature they don't want included. Or they may want more thorough unit tests. It could be that while you really did find a bug, that bug has now become a 'feature' and fixing it would cause problems to users they aren't ready to deal with yet. If your pull request is rejected it is okay to start a friendly conversation to find out why. You can always continue using your code in your projects.

Nothing

Finally, they may do nothing. Remember that as a rule the maintainer of the project you are trying to fix isn't getting paid to do this work. They have busy lives and other obligations. Be patient.

Using Git Hooks To Help Enforce Our SOP

This week at work I moved all of our git repositories from a GitHub account owned by one of our employees to one owned and managed by the company. That was easy enough, but reminded me that we've been using git commit hooks to make sure we include our ticket/issue number in the commit message. This lets us link Jira with our commits. Trouble is, I couldn't remember how exactly I did it. I should have written it up last time, so I'm writing it up this time.

Git Hooks

Git can run commands at a number of different events using hooks. The 'hook' files are located in your repository's .git/hooks directory. By default there are sample files for each event. These files make good starting place. To use one, just rename the file to remove the '.sample' from the end of the filename.

The bundled example files are Bash shell scripts. They work on Windows as well as Linux machines because the standard Windows installer for git includes a Bash emulator by default. You can write the scripts in any language you like (such as Python) so long as your system can run them as executable files.

Requiring an Issue Number

Our SOP requires us to include the ticket number we are working on as part of each commit message. You could get creative with this using regular expressions. But I wanted to keep it as simple as possible. I'm just doing simple text matching.

My 'commit-msg' looks like this:

 #!/bin/sh
 # This hook gets run after the commit message is entered, but before the commit
 # is made. If this fails, the commit will not take place. We are using it to:
 #    * Check that the first line of the commit message includes the issue number prefix

 # The path to the file with the commit message is passed in as the first argument
 file_path=$1

 # Get the first line of the file
 first_line=$(head -n 1 $file_path)

 # Check that the first line includes the text we want
 if [[ "$first_line" == *FW-* || "$first_line" == *TD-* ]]
 then
     exit 0
 else
     # The 'echos' will be displayed in the error message. The 'exit 1' tells git this is
     # a failure.
     echo "You must include the Jira issue number in the first line of the commit message."
     echo "The issue number must start with FW- or TD- and must be in all caps."
     exit 1
 fi

If you are familiar with Bash scripts this is pretty straight forward. If not, I'll walk through it below. The key is that this will get called before a commit is accepted. If the file returns a zero the event, in this case a commit, is accepted. If it returns a one it is rejected. You can do whatever you like in these scripts.

The breakdown

file_path=$1

The path to the commit message file is being passed in as the first argument to the script. We can use that to get the contents of the file. In this case I want the ticket number in the first line of the commit message. We get that here:

first_line=$(head -n 1 $file_path)

Next we check to see if the first line contains either "FW-" or "TD-". These are the two valid prefixes for tickets in the Jira instance we use at work. You could look for any string or strings. Or get fancier using some regex.

if [[ "$first_line" == *FW-* || "$first_line" == *TD-* ]]

If we find what we are looking for we return a '0'. This tells git all is well and to keep going with the action.

then
     exit 0

If not, we pop up a message to the user so they know their commit is about to be rejected. In this case the user is me, so I make it very specific and clear. After that we return a zero so git knows that it should reject the commit.

else
     # The 'echos' will be displayed in the error message. The 'exit 1' tells git this is
     # a failure.
     echo "You must include the Jira issue number in the first line of the commit message."
     echo "The issue number must start with FW- or TD- and must be in all caps."
     exit 1
 fi

All the git hooks work in the same way. They get called by the action indicated in the file name. The script does some work and returns a zero (success) or a one (failure). You can run hooks on both the client-side and the server-side. More info on the hooks can be found in the git documentation.

A few things to note

  • You can, and probably should, include your git hooks in your repo. Put a copy of each into another directory and commit them like you would other files. Then you or another developer will get the copy of every hook they'll need to when working with your repo. They'll still need to put them in the correct location, but they'll have what they need.
  • These are client-side only. This means that you aren't forced to use them. This helps remind a developer that they've made a mistake, but doesn't actually enforce the restriction. In this case, if someone doesn't use the hook, they can commit all they want without including a ticket number. That is where server-side hooks come in. These can be used to enforce the rules. It is probably best to use client-side and server-side scripts together to inform the developer of issues and make sure nothing gets committed that shouldn't.
  • The hook files have no file extension. This is weird for Windows users, but fear not, they are just regular text files and can be edited using any text editor.

Play Your Way to Flexbox Mastery

Projects at work have brought me back to front end web development. And reminded me what a pain it can be to position elements where you want them. I admit, for just a minute, I was tempted to use a table. Instead I used a Flexbox layout. While Flexbox is still a W3C draft, the CSS commands are supported by most modern browsers (according to Can I Use). If your users aren't using IE or old Android you are probably safe using it.

There are several excellent tutorials on Flexbox, such as this one by css-tricks.com. But if you are feeling a bit more playful check out these games that will keep you entertained while learning the basics of Flexbox layout.

Flexbox Froggy

Flexbox Froggy gives you a pond, some lillypads, and some frogs and ask you to get the frogs on the correctly colored pads. It starts of easy and adds commands as you progress through the 24 levels of play.

For example Level 19 has you using flex-direction and flex-wrap to go from this:

/images/flexbox_froggy_19start.thumbnail.png

to this:

/images/flexbox_froggy_19end.thumbnail.png

Flexbox Defense

If you are feeling a bit more aggressive, or maybe you're colorblind and the frogs all look the same to you, give Flexbox Defense a try. This spin on a classic tower defense game teaches you the key aspects of Flexbox layout while you blast dangerous purple circles with your turrets. Each of the 12 levels starts with a tutorial on the concept you'll need to make it past the level.

/images/flexbox_defense_1end.thumbnail.png

CSS Diner

Enjoying the mix of learning and fun? Need a little CSS selector practice? Checkout CSS Diner. It will teach you to use CSS classes, ids, and pseudo-selectors.

/images/css_diner_13start.thumbnail.png

Missing Dependencies When Installing Python Packages With Pip

Pip is the best way to install Python packages. It handles most of the pain of tracking down dependencies and installing everything. And usually it does it quite well. But some packages require various libraries to be installed and available on the system. When these libraries are missing the installation will fail.

The pip error messages can be overwhelming (there are a lot of them), but the info you need to fix the problem is there if you know where to look for it. In this case let's take a look at the errors I received when trying to install Scrapy into a new virtualenv. I'm on a recent-ish version of Debian Linux. I happen to be in Python 2.7 as Scrapy is still at 95% tests passing for Python 3 at the time of this writing.

Working in my new virtualenv I started by trying to install Scrapy.

pip install scrapy

Things start out well with the usual download, unpack, and install.

Downloading/unpacking scrapy
Downloading Scrapy-1.0.5-py2-none-any.whl (291kB): 291kB downloaded
Downloading/unpacking cssselect>=0.9 (from scrapy)
Downloading cssselect-0.9.1.tar.gz
Running setup.py (path:/tmp/pip-build-pETEoF/cssselect/setup.py) egg_info for package cssselect

And then we hit a snag. After a bit we run into an error with building some of the dependencies. This cascades and we end up with a slew of errors. This is where it gets a bit overwhelming. But we can zero in on the error to see exactly what we're missing.

Building wheels for collected packages: cryptography, cffi
Running setup.py bdist_wheel for cryptography ... error
Complete output from command /home/joshua/.virtualenvs/scrapycars/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-85Mdmv/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmp6bFspgpip-wheel- --python-tag cp27:
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found

This error gets repeated multiple times (I don't know why, but I have it five times in a row) and then gets followed by a bunch of other output.

The key here really is that we are missing the libffi package. This package 'allows code written in one language to call code written in another language'. A search for 'debian libffi' showed me that I could install the libffi-dev package.

sudo apt-get install libffi-dev

With this package installed I then installed Scrapy without errors.

The Fix

  1. Don't panic.
  2. Start at the top-most error (others may be caused by the first error).
  3. Search for and install your missing package.
  4. Pip install again.
  5. Repeat as necessary.

But It Works Outside My VirtualEnv

You may find yourself with dependency errors in a virtualenv, but you didn't get any errors when you pip installed from your system Python installation. How can that be? Well, mostly likely you already had the package installed. When you run pip install again it doesn't throw up errors if the package is already installed.

The package may have been installed as part of the distribution and didn't need the missing package as part of its build. Or you may have had the missing package installed at one time and it was removed in the interim.

Deploying A WSGI Application On WebFaction

I've been using WebFaction since April 2010 as my webhost. Many of those projects are written in Python and use WSGI. Most often they will be a Flask app of some kind.

Creating The Application

WebFaction has their own custom control panel. We'll use the control panel to provision the application on their servers. This part isn't hard, but it is unique to WebFaction. See their docs for how to create an application, connect it to a domain, and create the website.

The key thing when creating the application is to set the App category to mod_wsgi and then pick the appropriate Python version.

/images/webfaction_wsgi_app.thumbnail.png

Using a VirtualEnv

You can use a virtualenv on WebFaction just like you would in your development environment. The benefits are basically the same: separation of package versions between projects, reproducibility between deployments, a general feeling of smugness.

If you are using virtualenvs you'll want virtualenvwrapper. You'll need to install that yourself. I followed the steps in this excellent post.

Once that is done create a new virtualenv:

mkvirtualenv yourvirtualenv

I prefer to use only packages in my virtualenv. You can use global packages if you like, but you run the risk of upgrading a dependency for one project that breaks another. To use only virtualenv packages you create an empty file called sitecustomize.py in your virtualenv's lib/pythonx.y directory.

Next you want to install the packages required by your project. To get that list pip is your friend. From your development environment:

pip freeze > requirements.txt

Now transfer that server to your host and use it to install the packages:

pip install -r requirements.txt

Updating index.py

When your application was created an index.py file was automatically created within the htdocs directory. This file needs to be edited to activate your virtualenv and to set some paths needed by WSGI.

As an example:

import sys

activate_this='/home/yourAccountName/.virtualenvs/yourVirtualEnvName/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))

sys.path.append('/home/yourAccountName/webapps/yourApp/htdocs')

from main import app as application

You will of course replace the paths withe ones to your account's virtualenvs and application directories.

The last line is the statement that starts your application. In my case I have a main.py file that handles starting the app.

Updating httpd.conf

The last file that needs to be edited is the Apache httpd.conf file. It is found in the applications apache2/conf directory. Edit the file to add the following:

WSGIPythonPath /home/yourAccountName/webapps/yourApp/htdocs/
WSGIScriptAlias / /home/yourAccountName/webapps/yourApp/htdocs/index.py

<Directory /home/yourAccountName/webapps/yourApp/htdocs/>
    AddHandler wsgi-script .py
    RewriteEngine on
    RewriteBase /
    WSGIScriptReloading on
</Directory>

This will work if your app is running at the root domain or a subdomain. If you are running in a subdirectory you have some more work to do. But for now, this will remove the .py from the path and give you the kind of URLs you are probably expecting.

Deploy Files and Restart Apache

If you haven't already, now is a good time to move your code to the server. I use mercurial to do this, but you can do it via git, FTP, or whatever method is convenient for you.

Once everything is in place restart Apache so it sees the configuration changes.

apache2/bin/restart

Victory

At this point you should be able to pull up your app in the browser and see it in all its glory.