Pre-Conj Interview: Ashton Kemerling
Talk: Generative Integration Tests
Background
Ashton Kemerling's talk at the conj is about Generative Testing (also known as Property-based Testing) as applied to integration tests. Generative Testing is a methodology for generating many tests randomly to find failing cases (as opposed to writing each test by hand). It often finds difficult bugs and can present a very small, reproducible failing case.
The best talk I have seen about Generative Testing is John Hughes' talk at Clojure/West. John Hughes wrote Quickcheck for Haskell and Erlang. Reid Draper has recently ported it to Clojure in the form of test.check. He also gave a talk at Clojure/West about test.check.
Why it matters
Generative testing is great for unit testing. It has already found bugs in Clojure itself. What's not obvious is how to apply the technique to integration tests to test the interaction of different systems. This talk appears to be an experience report from a company that has actually used Generative Integration Tests.
About Ashton Kemerling
Introduction
I asked Ashton Kemerling some questions about his talk at Clojure/conj about using generative testing in Clojure. Read the background to his talk.
Interview
PurelyFunctional.tv: How did you get into Clojure?
Ashton Kemerling: The path I took to Clojure is fairly roundabout. During my College years I was very interested in function programming, although ML & Haskell were my favorites back then. I think a large amount of my fascination with them stemmed from the fact that I had more experience than my classmates, my high school had a very rich set of programming courses, and I needed something interesting to play with while taking introduction to OOP courses.
Fast forward a few years and I ended up working at a job using Common Lisp professionally in a legacy web application. The application had a lot of problems due to the condition of CL libraries at the time and we were looking for something to switch to. Clojure was an obvious choice to look at. It's still a lisp, so a lot of our habits would still work, but it fixes the deployment & library issue that CL has while adding a much richer set of data structures. Unfortunately our application relied extensively on mutation (CL doesn't really encourage or discourage any coding style) and OOP, so a conversion was deemed too costly to attempt.
Over the course of the following years I continued to tinker with Clojure on the side. I was convinced it was a great way to do things, but I could never convince anyone to consider using it in their code base. Most places were either too invested emotionally in OOP, or had well tested and working codebases that made a rewrite unwise. So I mostly just hacked on side projects using Clojure over the intervening years without using it professionally. But in the last year or so I realized that Clojure makes a great secondary language for odd-tasks in non-Clojure codebases, which brings me to the current state of using Clojure to test a very large Ruby on Rails and Javascript application.
PF.tv: Can you give a basic outline of how your Clojure program tested the web front end?
AK: We've basically tested in two ways. The first was to simply compile our JS file along with Clojurescript tests and run them in a PhantomJS process similarly to how you'd run Clojurescript or Javascript tests normally. We're still working on this approach, but it's slower to work on that I would have hoped.
The second way was to leverage the fact that Clojure's on the JVM. This involves mixing JDBC, HTTP libraries, and Selenium in novel and exciting ways. This has been the most fruitful way of testing, and what I'll be focusing the most on during my talk. We've used the code I've written as both a tool to hunt down API mistakes, and as a means of narrowing down the reproduction steps on complicated bugs reported by the user.
PF.tv: Can you explain a little more about that? What do you mean by "narrowing down reproduction steps"?
AK: Sure. I'm testing Pivotal Tracker, a very large (~24kloc) Javascript "single page app". With large apps like this there are ways for the application to subtly fall out of sync with the server or process data in a degenerate way.
In particular a lot of times you'll get a report from a user that says something along the lines of "I reloaded and it crashed". You can dig up the logs from that session, but you'll get the last 20 or 30 steps they did before the crash, which doesn't really help you out a whole lot. So we occasionally turn the generative tests towards a type of action (in this case, do things and then refresh) in the hopes that it will help us narrow down to the minimal reproduction steps required to trigger a problem.
In the case that happened last week, we were able to find 2 distinct actions that would provoke the issue reliably after about 20 minutes of test modification and running. Obviously this took hours off of the process required to find the actual root of the bug, since 2 is orders of magnitude easier to work with than 7 or 10 when debugging.
PF.tv: I see. So how is test.check able to reduce the reproduction steps?
AK: Test.Check provides "shrinking". Just as the generators used to create randomized data have the ability to produce more complicated data, they also have the ability to produce less complicated data. Test.Check records all of the failures it finds and attempts to simplify them and find the smallest failing case it can.
PF.tv: A lot of the Clojure/conj participants will be new to Clojure. What resources would you recommend them to make the most of your talk?
AK: They can't go wrong with the test.check README or any of the blog posts mentioned therein. I recommend reading the source directly because it's heavily commented, in particular I recommend the generator source, because the generators represent 90%+ of the test.check API a user will interact with.
Beyond that I have blog posts on my work blog and on my personal blog. Also Reid Draper and I were both on the Cognicast, both of which are based around this subject.
PF.tv: Are there any resources on Selenium and other methods for running frontend tests?
AK: The Clj-Webdriver docs are all I can recommend.
PF.tv: Where can people reach you?
AK: Twitter: @ashton
My blog: ashtonkemerling.com
PF.tv: If Clojure were stranded on a desert island, what one book would it bring?
AK: A boat building book, clearly.
PF.tv: Awesome! Thanks for the interview, it was a pleasure.