Talk: Always Be Composing
Zach Tellman’s talk at the conj is about building composable software.
People often boast of the composability of Clojure programs. Compojure claims that it “allows web application to be composed of small, independent parts.” Ring itself is based on composing middleware and handlers to build a web server. In fact, you can compose pieces easily with Ring, if those pieces were designed to compose with Ring. But not all libraries compose well with the Ring ecosystem. This talk asks “How can we design libraries that compose well in general?” A good intro to composition in Clojure is Rich Hickey’s talk Design, Composition, and Performance.
Why it matters
I often get asked why a certain library in Clojure doesn’t have an easy interface like the one in Ruby. The answer, in many cases, is that the Clojure library was designed to compose with other code, while the Ruby library was meant for programmer convenience.
But if I leave it at that, this answer is unsatisfactory. Behind my answer is years of experience telling me that composition beats convenience in the long run. The asker of that question does not have that experience. I would love to have a way to explain why in an analytical way.
If this talk even so much as opens the conversation up about the design space of composition, it would be extremely helpful. But I’m expecting more from Zach Tellman. His talk could have a profound impact similar to Rich Hickey’s talks on immutable values and simplicity.
About Zach Tellman
PurelyFunctional.tv: Briefly, how did you get into Clojure?
Zach Tellman: At my first job, the main language was C#. However, I worked with Tom Faulhaber, who is a Common Lisp guy from way back, and at the time was just working on the initial implementation of clojure.pprint. I started playing around with Clojure, and started building Penumbra in mid 2009. I haven’t looked back since.
PF.tv: Composition is a very important concept in the Clojure world, yet you claim that many libraries can’t compose cleanly. Why is it such a problem?
ZT: Clojure is a young language, and is fairly unique in the way it tries to couple functional idioms with less-than-functional host environments. It also lacks any real organizational principle: a namespace can encompass an enormous amount of functionality (see clojure.core), or a very small, focused piece of functionality (see most of Ring). Both of these allow us enormous flexibility, but it also means that Clojure programmers are without either precedent or obvious affordances when starting out with Clojure. It’s understandable that we’d make some missteps.
If we look at the first two or three SQL libraries written for Clojure, it’s notable that they all used macros. It became clear after not too long, though, that this was a really limiting approach – the only way to compose with a macro was with more macros. This doesn’t mean that macros shouldn’t be used, of course, just that we should always be aware of how they constrain the surrounding code. Abstractions that are incredibly useful in application-level code might be irritating and arbitrary in a library.
I think a lot of people begin to innately understand these design considerations after a few years of using Clojure, but I haven’t heard anyone try to articulate them in a general form. I guess we’ll see how I do at that.
PF.tv: You mention Ring. I think it’s a great example of a system with very good composability. Middleware written by different programmers can be combined without problems. Ring achieves this by being a standard protocol. So there is coordination between the programmers in that they both agree to the centralized standard. Are there other patterns besides standards that can allow for composition?
ZT: Ring is useful because it’s focused. Modeling HTTP requests as a pure function is a lossy abstraction, but it works for the vast majority of requests. If it wasn’t so simple, I don’t think it would have had nearly as much adoption (see the n-many mostly unused async extensions to Ring that try to accurately model the entire problem space). But by using Ring, we commit ourselves to its simplified view of the world, as well as its thread-per-connection execution model. This is almost always an acceptable simplification, but it transitively limits everything downstream of it. Now we find ourselves in a situation where the entire ecosystem is predicated on this simplified view of things, and anyone who tries to do something more general has to start over from scratch.
So to actually answer your question, composition can’t happen without conventions. But conventions, like macros (which really are just conventions of code structure), constrain and shape the code around them. The tradeoffs and resulting design space are what I want to explore.
PF.tv: You’ve mentioned that Clojure does not constrain the code. One of the most common questions I answer from Clojure beginners is how to structure their code given the freedom allowed in Clojure. Do you think conventions that support composition could help answer that question?
ZT: I use a lot of Java-land libraries, so I spend a lot of time reading Javadocs. The nice thing about Javadocs is that even if the actual docstrings are minimal, it gives you a full, unambiguous dependency graph between all the different pieces of code. There’s no analogue for that in Clojure, which makes reuse a lot harder than it otherwise would be.
PF.tv: You mention dependency graph. Have you had a chance to look at CrossClj? Would that kind of thing be helpful? What would it need to be more helpful?
ZT: I’ve looked at it, it’s a good place to start, but raw dependency data isn’t useful without the tooling to integrate it into our development process. Light Table and others are playing around in this space, but it’s a genuinely hard problem, because it’s not a one-size-fits-all visualization problem. I don’t have any solutions to offer, but I’m certainly paying attention to how things develop.
PF.tv: So there are going to be a lot of beginners at the conj. Are there any resources that could prepare them to make the most of your talk? Any blog posts or videos that would get them ready to participate actively?
ZT: I’ll talk about macros, transducers, and core.async, among other things. Being conversant in each will help, but isn’t necessary.
PF.tv: Where can people follow you and your adventures?
PF.tv: Ok, last question: What is the average airspeed velocity of an unladed Clojure REPL?
ZT: JVM or V8?
PF.tv: I don’t know that!
Thanks for the interview. It was very informative.