Ashton Kemerling’s talk at the conj is about Generative Testing (also known as Property-based Testing) as applied to integration tests. Generative Testing is a methodology for generating many tests randomly to find failing cases (as opposed to writing each test by hand). It often finds difficult bugs and can present a very small, reproducible failing case.
The best talk I have seen about Generative Testing is John Hughes’ talk at Clojure/West. John Hughes wrote Quickcheck for Haskell and Erlang. Reid Draper has recently ported it to Clojure in the form of test.check. He also gave a talk at Clojure/West about test.check.
Why it matters
Generative testing is great for unit testing. It has already found bugs in Clojure itself. What’s not obvious is how to apply the technique to integration tests to test the interaction of different systems. This talk appears to be an experience report from a company that has actually used Generative Integration Tests.
About Ashton Kemerling
PurelyFunctional.tv: How did you get into Clojure?
Ashton Kemerling: The path I took to Clojure is fairly roundabout. During my College years I was very interested in function programming, although ML & Haskell were my favorites back then. I think a large amount of my fascination with them stemmed from the fact that I had more experience than my classmates, my high school had a very rich set of programming courses, and I needed something interesting to play with while taking introduction to OOP courses.
Fast forward a few years and I ended up working at a job using Common Lisp professionally in a legacy web application. The application had a lot of problems due to the condition of CL libraries at the time and we were looking for something to switch to. Clojure was an obvious choice to look at. It’s still a lisp, so a lot of our habits would still work, but it fixes the deployment & library issue that CL has while adding a much richer set of data structures. Unfortunately our application relied extensively on mutation (CL doesn’t really encourage or discourage any coding style) and OOP, so a conversion was deemed too costly to attempt.
PF.tv: Can you give a basic outline of how your Clojure program tested the web front end?
The second way was to leverage the fact that Clojure’s on the JVM. This involves mixing JDBC, HTTP libraries, and Selenium in novel and exciting ways. This has been the most fruitful way of testing, and what I’ll be focusing the most on during my talk. We’ve used the code I’ve written as both a tool to hunt down API mistakes, and as a means of narrowing down the reproduction steps on complicated bugs reported by the user.
PF.tv: Can you explain a little more about that? What do you mean by “narrowing down reproduction steps”?
In particular a lot of times you’ll get a report from a user that says something along the lines of “I reloaded and it crashed”. You can dig up the logs from that session, but you’ll get the last 20 or 30 steps they did before the crash, which doesn’t really help you out a whole lot. So we occasionally turn the generative tests towards a type of action (in this case, do things and then refresh) in the hopes that it will help us narrow down to the minimal reproduction steps required to trigger a problem.
In the case that happened last week, we were able to find 2 distinct actions that would provoke the issue reliably after about 20 minutes of test modification and running. Obviously this took hours off of the process required to find the actual root of the bug, since 2 is orders of magnitude easier to work with than 7 or 10 when debugging.
PF.tv: I see. So how is test.check able to reduce the reproduction steps?
AK: Test.Check provides “shrinking”. Just as the generators used to create randomized data have the ability to produce more complicated data, they also have the ability to produce less complicated data. Test.Check records all of the failures it finds and attempts to simplify them and find the smallest failing case it can.
PF.tv: A lot of the Clojure/conj participants will be new to Clojure. What resources would you recommend them to make the most of your talk?
AK: They can’t go wrong with the test.check README or any of the blog posts mentioned therein. I recommend reading the source directly because it’s heavily commented, in particular I recommend the generator source, because the generators represent 90%+ of the test.check API a user will interact with.
PF.tv: Are there any resources on Selenium and other methods for running frontend tests?
AK: The Clj-Webdriver docs are all I can recommend.
PF.tv: Where can people reach you?
AK: Twitter: @ashton
My blog: ashtonkemerling.com
PF.tv: If Clojure were stranded on a desert island, what one book would it bring?
AK: A boat building book, clearly.
PF.tv: Awesome! Thanks for the interview, it was a pleasure.