Wednesday, May 13, 2015

How Shift is Using Depicted (dpxdt) and Why You Should Too

I've been meaning to write a post about how we are using Depicted (aka dpxdt) at Shift for awhile now. Since it just saved my arse from pushing a broken release to production I think now is the time for me to both pay homage and share how we're using it in case anyone else might find it useful.

But first .. let me share with you how Depicted saved me today! I'm working on a major refactor change for Shift's URLs, and I expect zero visual change as far as the user is concerned. So I'm intentionally not going to update the URLs in our Depicted config because I expect that everything should be properly redirected. So I pushed my changes to our production server, which can serve both my new version and our current version concurrently (thanks App Engine!). Then I initiated a dpxdt run like so (I'm leaving out some unnecessary args here):

dpxdt/ \
  --test_host "" \ 
  --ref_host ""

Among a few other gems, I got the following diffs:

In this case it's pretty obvious that I forgot to add a redirect for an URL. Whoops! Good thing our users never got this 404 page.

And here in this second diff it's obvious that on of our AJAX requests which loads data about our Car Enthusiasts returned an empty list. I was able to very quickly figure out my bug, and write the unit test for it.

In both of these cases I was able to easily amend my change before I'd already pushed to production. If I had pushed to production I would have had to roll it back and re-push. This is a huge developer time saver and of course provides anyone on the team with insight about how any given change will impact users.

Brett Slatkin, the author of Depicted, describes it as "the ultimate, automated end-to-end test." Depicted allows you to compare screenshots for a new release against screenshots from a "golden" release to catch any visual changes - ideally, before you make the new version live. If you're not using visual diff yet to test your pushes you should be.

As compared with other integration testing tools like Selenium or Casper, it's super easy to debug the results of a Depicted visual diff. Visual diff is also better representation of a bug in the User Experience of your site - it's precisely what your customers see, as opposed to testing state about the underlying DOM.

Adding a Depicted test is also WAY easier than writing another Casper or Selenium test - Depicted uses a YAML configuration file, which we at Shift generate via python script at the end of our CircleCI build. While YAML allows you to do "mix-ins" we wanted to optimize our config to be super easy to add an URL to and also wanted to bake in some special functionality. Ours base config looks like this:

      width: 1280
      height: 1024
      width: 768
      height: 1024
      width: 480
      height: 1024

# It can take a while for all the assets on our page to load.
resourceTimeoutMs: 60000

  - mixpanel
  - adroll
  - adnxs
  - doubleclick
  - perfectaudience
  - getsentry
  - googleadservices
  # Because phantomjs is a little peculiar about webfonts, let's not bother.
  - fonts
  - vimeo

# Attempt to deal with inconsistent font rendering leading to false positives.
injectCss: |
  html > body,
  html > body * {
    font-family: Arial, sans-serif !important;

    url: /404.html

    url: /500.html

    url: /about
    # Because we tend to get false negatives due to the fade-in
    # effect from the G Maps API, we'll just hide the map before 
    # the screenshot.
    injectCss: |
      #about-page .static-map {
        visibility: hidden

    url: /buy

    url: /sell

  SF Cars:
    url: /cars/san-francisco

  Car Tracker:
    url: /my-car/some_key

  Car Tracker:
    url: /my-car/some_other_key

You can tell from that config that it supports the idea that we generate different configs for staging and production, where our datastores contain different entities.

There are also a few scenarios you might notice in our config with regards to Google Maps and font rendering. Fonts are supposed to be better with phantomjs2 so we'll see if we can nuke the font overrides at some point.

We still need to set up some golden data on our servers to guarantee we can test screenshots with stable data, and we also want to make every change to our master branch do a rolling-forward-golden test on our staging server so that it might be more obvious how any change affects master, visually (whether it's a production push or not).

There's still lots to do!

Depicted is amazing. I will make the no-duh prediction here that someone will productize a visual diff service in the next year in the same way that Sentry and NewRelic have done for visualizing and alerting developers on code-level exceptions. Until then, we're really happy deploying our docker VM to Google App Engine and running our own Depicted server.

Depicted has changed how I write code and how I think about production. Production is the best place to do integration testing (ideally before any new version goes live =). There's an old saying about how users don't care about our code, they care about what they see and experience - Depicted reinforces this all the way through a change - I don't care about lines-of-code in a Changelist nearly as much as I care about visual outcome. How many times have you asked someone to send you a screenshot of their changelist or to show you a changelist running on their dev server before you can totally understand it? Depicted obviates the need for this and ultimately can make the code review process much more sane. Imagine if every code review automatically included the screenshot diffs produced by a change? We're going to get there at Shift soon, and I hope that this practice becomes commonplace.

P.S. Thanks to Mikey Lintz for noticing that dpxdt = Delta Pixels Delta Time