About a year ago we converted this site to run fully on Jekyll and migrated our blog from Tumblr. When we did that we gained some freedom and the ability to edit our content on GitHub. We’ve received quite a few pull requests from readers to fix typos and optimize images. As our blog got more complicated, we started making mistakes that were hard to catch before publishing. So we came up with a way to catch many of those errors, before they end up in your browser.
Basic content problems are a pain. This is why we run automated tests on 18f.gsa.gov.
One advantage of hosting our site’s code on GitHub is we can integrate with continuous integration (CI) services to check our work before we publish. Many open source projects use a CI to run various tests on their code. We use it to run tests on our content. We use Travis CI but there are many others out there. Until recently, the only test we ran was building the site to know if it was successful.
Just building the site caught a few high-level errors but not all. We added the
html-proofer to our test suite, which checks internal links and validates the HTML on every generated page. This helps us catch more errors but is still kind of like a chainsaw: It tears through the material but misses detail-level problems. If a post was missing the
authors field in the frontmatter, for example, all the tests would pass but the generated page would be missing a byline. This is the kind of error we wouldn’t catch until we merged the content into the site and previewed it on our staging server.
Manually inspecting things is the worst, especially when you can make a computer to do it for you. This is why we wrote the
jekyll_frontmatter_tests gem. It’s pretty simple and helps us catch problems like this before we publish. It works by comparing our posts to a “schema” that describes which fields should exist in the frontmatter and how their values should be formatted. Let’s walk through installation and setup for testing a Jekyll site’s posts.
For these instructions, we recommend typing in each of the commands, not copy-pasting.
Install it by adding
jekyll_frontmatter_tests to your
_config.yml in the
jekyll-plugins section and then
gem install jekyll_frontmatter_tests. Once installed, you should have an extra command called
test when you run
jekyll --help. Here’s what 18f.gsa.gov’s
Gemfile look like.
Next, create a “schema” file in a directory called
deploy/tests/schema. If you don’t have that directory, you can make it by running
mkdir -p deploy/tests/schema from the root of your Jekyll directory. Then create a file called
_posts.yml. You can do this by running
deploy/tests/schema/_posts.yml in your favorite editor.
Let’s assume you expect all your posts to look like this:
--- title: “My fancy blog post” authors: - boone tags: - ruby - gems - jekyll
In that example, each post is required to have a
title that is a string, an array of
authors, and an array of
tags. Each one of those needs a line in the
_posts.yml file, with the expected “type” on the right.
--- title: “String” tags: “Array” authors: “Array”
Once that’s in place, we need to add a
config section at the bottom of the schema file. This will give the plugin a little information it wouldn’t otherwise know. There are two required sections in this part right now:
ignore. The first tells the test where to find the posts, the second tells it to ignore specific files that might also be in that directory. In 18f.gsa.gov, for example, we have a README in our
_posts directory so we add README.md to this section. If you’re using a Mac, yours might look like this:
--- config: path: _posts ignore: - README.md - .. - . - .DS_Store
Altogether, your schema file looks like this:
--- title: “String” tags: “Array” authors: “Array” config: path: _posts ignore: - README.md - .. - . - .DS_Store
Running the tests
Now that you have your schema file in place, run
jekyll test. The command will check each post to make sure it has a title that is a string, an array of tags, and an array of authors. If the test passes, you’ll see the following:
Starting tests Testing posts Finished testing posts Tests finished!
If it fails, you’ll see something like this:
starting tests testing posts The file 2015-09-01-my-fancy-blog-post.md is missing the following keys: * authors Finished testing The test exited with errors: see above.
Integrating with Travis
We use Travis to test 18f.gsa.gov, but these tests will probably run on any continuous integration service. All you need to do is add
bundle exec jekyll test to your Travis configuration. On this site we have a “go script” that runs a couple other things (including a site build) in addition to testing all our blog posts and team member profiles.
That’s it! It was built for Jekyll 3.0 from the start but should work with earlier version. It’ll work for complex sites like ours down to the simplest of blogs. Just make a schema file for each collection and you’re testing.
Update: As a cruel bit of irony we made a couple mistakes with the open graph image in this post. We’re already thinking about ways to write tests to prevent this!