How to Program Good is a series focusing on the practice of writing Good Programs.
Navigate to the Series's index if you would like to start from the beginning.
Chapter 2: Data and Interfaces
Chapter 1 discussed the standard by which we will be judging Good Programs: reliability. Reliable programs, given the same inputs, do the same thing everytime. In order for an overall program to be considered reliable, it must be created using pieces which are themselves reliable. It's certainly possible to create an unreliable program from reliable parts, but the inverse is completely impossible.
There are many different kinds of parts we could choose as our fundamental building blocks, but (if you couldn't guess from the chapter title) we will be choosing these: data and interfaces. Why these? Because they work well for our purpose, and they are always available no matter the language or circumstance.
Data - The Inert Material
In the context of a program we generally think in terms of "variables", which name some value in a certain scope, like `A = 6` or `F = open("some file.txt")`. We're going to separate these values into two categories: data and non-data. Our definition of data might be different from another that you've seen, but in the context of this series we will stick with this:
Data is any value which doesn't lose meaning by being moved.
The integer 6 is always 6. No matter if it's moved from stack to heap, memory to disk, my machine to yours, today to tomorrow; it's always just 6. The same can be said for strings, floats, decimals, booleans, and possibly other types specific to certain languages. Arrays, structs, lists, maps, sets, and other data structures retain this property as well.
So what falls into non-data? File handles, closures, sockets, mutexes, process IDs... these are all values which can exist in a program at runtime, and some of them might even be able to be represented as something resembling data (e.g. process IDs), but their actual meaning does not survive being moved from one process to another, from one machine to another, or from one day to the next.
The fact that data can be moved freely is not exactly what makes it useful to us, but rather that free movement necessarily removes two key sources of unreliability: time and place. Non-data is only usable to the extent that it exists in the right place at the right time, and by being tied to these two moving targets an element of uncertainty seeps in. You will notice that usage of non-data is where the vast majority of error handling in programs takes place. Of course we can't avoid the usage of these tools, and we shouldn't be afraid of them, but it's important to recognize that we should prefer data when we can use it.
Interfaces - Describing the Moving Parts
We want our programs to do things, but data is inert and can't do anything on its own. It's the rest of our programs, the functions and classes and methods, that define the behavior of our programs. As a program grows we quickly find the need to organize this behavior in order to make sense of it. The process of organization takes the form of drawing imaginary borders between different sections, each with its own responsibilities; one part of a program does one thing, another part does something else, and neither part knows or cares about the details of what the other is doing.
This concept is called encapsulation. The idea is to construct a blackbox of code, in this series called a "component", which accepts some inputs and returns some outputs accordingly, but whose inards are not known or even important. Each component can be thought of as a small program-within-the-program, and so is judged against the same criteria as our overall program: reliability. Given the same inputs, does the component produce the same outputs every time? This correspondence of inputs to outputs brings us to our second building block: the interface.
An interface is a mechanism which both describes the behavior of a component and disconnects its implementation from that description.
The correspondence of inputs to outputs described by the interface is its "contract", and for a component to implement the interface it must follow that contract. The description of behavior nearly always takes the form of a set of functions or methods implemented together, along with documentation of each function/method and the interface as a whole. Generally a language will consider some type to implement an interface if that type implements every method in the interface. In dynamic languages there might not need to be an explicit definition of an interface in the code, so it will be up to the programmer to document and assert for themselves that a component implements an interface.
An example definition of an interface and its associated types, written in a made-up language, might look like this:
# Contact describes the pieces of information about a person which can be stored # in an addressBook. Contacts are identified by their name field. Contact struct { name String address String phoneNumber String email String } # An addressBook can store Contacts and retrieve them later. addressBook interface { # set stores the given Contact for later retrieval using the list and get # methods. If set has been previously called with a Contact sharing the same # name then that old Contact is replaced by this new one. set func(contact Contact) -> () # list returns all Contacts which have been added to the AddressBook using # set, sorted by their name. list func() -> []Contact # get returns the Contact which was most recently passed to set sharing the # given name, or returns None if no such Contact has ever been passed to # set. get func(name String) -> Maybe<Contact> }
Note the documentation of the interface. While the name and signature of each method provides a pretty good idea of the overall behavior of the interface, the documentation specifies behavior which types can't capture, and is as much a binding part of the contract as they are. What does `set` do if the Contact has been passed to it before? In what order are the Contacts returned from `list`? If some caller of an interface method depends on that method behaving a certain way, that behavior MUST be incorporated into the method's documentation, therefore becoming part of the contract.
A basic component which implements this interface might look like this:
# inMemoryAddressBook implements the addressBook interface using an in-memory # Map type. All Contacts stored in the inMemoryAddressBook will be lost once # it goes out of scope. inMemoryAddressBook Map<String, Contact> (self *inMemoryAddressBook) implements addressBook { set func(contact Contact) -> () { self.store(contact.name, contact) } list func() -> List<Contact> { contacts = self.getAllValues() return contacts.sortedBy(func(c Contact) -> String { return c.name }) } get func(name String) -> Maybe<Contact> { return self.getValue(name) } }
An interface may have multiple implementations which might get used interchangeably depending on configuration or user input. The parts of the code which are relying on an interface only see that there is something which implements it, and that's all they care about. This is the most important property of interfaces, and is what makes them so useful. It allows for different sections of a codebase to expand, contract, and be replaced without affecting any other parts. As long as the same inputs continue to produce the same outputs the rest of the codebase can remain ignorant of what is happening inside.
Here we're going to add another implementation of the `addressBook` interface we defined previously, and then show how some code might use one implementation or the other depending on runtime circumstances:
# kvAddressBook implements the addressBook interface by using a kvDatabase to # store and retrieve Contacts. # # In this example kvDatabase is itself some interface describing a persistent # key-value database which has methods set, get, and getAll. For simplicity this # example ignores error handling both for interactions with the database and # JSON encoding/decoding. kvAddressBook kvDatabase (self kvAddressBook) implements addressBook { set func(contact Contact) -> () { self.set(contact.name, json.encodeToString(contact)) } list func() -> List<Contact> { contacts = self.getAll().map(func(key String, val String) -> Contact { return json.decodeFromString<Contact>(val) }) return contacts.sortedBy(func(c Contact) -> String { return c.name }) } get func(name string) -> Maybe<Contact> { if val = self.get(name); val.some() { return Some(json.decodeFromString<Contact>(val)) } return None } } main func() -> () { # We will initialize thisAddressBook using one of the two possible # implementations, depending on an environment variable. thisAddressBook addressBook if os.getenv("IN_MEMORY") != "" { thisAddressBook = inMemoryAddressBook{} } else { kvDB = # instantiate kvDatabase thisAddressBook = kvAddressBook{kvDB} } # thisAddressBook can now be used as an addressBook, regardless of its # implementation. }
Changing an Interface
Changing the contract of an interface is not something which is done lightly, but will certainly happen, especially in the draft phase of a program where the groundwork is still being laid.
When an interface is changed it is required that all components which implement the interface are carefully checked to be following the new contract. On the other side, all places which are calling methods on the interface must also be checked to ensure they are still using the interface properly.
This checking is not optional, and if it seems tedious then it merely underlines how important simplicity is in interface design, especially as an interface becomes used more and more. On the other hand it's quite easy to identify all places which need to be checked: simply search for all places where the interface is used.
A static type checker can helpful here, but keep in mind that behavioral changes to a contract might not result in changes to the method type signatures. And while data types are not interfaces, changing the definition of a type does constitute a change to the contract of every interface which uses that type.
Sometimes changes to an interface can be made in a way which doesn't require going back and checking all places where the interface is used. These changes are called "backwards-compatible" changes. Perhaps previously a method accepted an integer argument which couldn't be negative, but it now accepts negative values too. As long as the behavior for positive integers remains the same then this is a backwards-compatible change; the previous correspondence of inputs to outputs has not been modified.
Backwards compatibility is always preferred, but often requires undocumented assumptions about the interface. For example, adding a field to a struct is usually backwards-compatible, as long as the field's default value is consistent with the struct's meaning without the field. But what if somewhere in the codebase somebody used a data serializer which assumes the struct's fields don't change? Adding a field to the struct would invalidate all previously serialized forms of it! So do we consider struct field additions to be backwards-incompatible? Usually not, but if you're aware of the usage of such a serializer in your codebase you need to go and double check.
Just Use Data
Designing interfaces which are generic enough to hide their implementation, specific enough to be useful, and simple enough to rarely need changing, is more of an art than a science. As we go on we will encounter different ways to think about the overall structure of a program to help see how components work together to form a larger whole. These different angles on the problem may make it easier to find solutions, but ultimately good interface design is just a matter of practice. This section describes one rule which should be followed when defining an interface, and by reducing the field of possible solutions this rule will also help with tricky design problems.
In almost all cases the inputs and outputs of an interface should be just data. When data crosses the interface boundary (either being given as an input, or being returned as an output) its meaning is not tied to the state of the caller or component, and so can be treated just as it is. When non-data is passed across the interface boundary then so is the implied state of the caller/component associated with that non-data. This implied state then necessarily must become part of the documented interface contract, leaving it more complex to understand and more difficult to correctly implement.
There are rare cases where non-data being passed across the boundary is worth the trade-off. A notable example for me is Go's `Context` type, which allows a caller to cancel a blocking call from outside. This tool simplifies a lot of situations in asynchronous programming which would otherwise require the use of message passing or callbacks. Other such examples may exist, and there may be specific cases where more complex contracts are required to account for performance considerations. But these cases should be met with skepticism, require significant justification, and of course be fully documented in the contract.
Everything is Interfaces
So far we've considered the interface as being a mechanism provided by a programming language which allows us to encapsulate a set of behavior in a useful way. But there are other kinds of interfaces. If you consider every programming-related acronym ending in an "I", the "I" probably stands for "interface": API, CLI, GUI, ABI, TUI, CNI. In all cases the interface is the surface layer of a larger entity, like the remote control of a TV, or the shell of an egg. All the same rules we've been talking about for language-level interfaces apply to these as well: prefer data, prefer backwards-incomptable changes, clearly document all contractual behavior.
Even interfaces which face a human, rather than a machine, like CLIs and GUIs, deserve this treatment. Remember our purpose, which is to create programs which behave so reliably that they become an extension of the user themselves. If the user's interface into the program is constantly changing, like the toy hammer which dents and warps over time, this extension can never happen. Sure, new functionality can be added, but it must be added in a backwards-compatible way, so that the user's existing expectations continue to be met.
The Marble Run
To better understand the relationship between data and interfaces, let's look at a real-world analogy: the marble run. Marble runs are toys which, in their simplest form, consist of a series of rails or tubes into which someone drops marbles, allowing the marbles to run the course from top to bottom. More complex marble runs introduce electric motors to convey the marbles back up to the top of the course, becoming self-driven loops. The artistry of marble runs comes from inventing new and inventive mechanisms for the marbles to run through, from bowls to pendulums, as well as new ways to convey them back to the top, from elevators to ferris wheels.
An example of a complex, self-driven marble run
In our analogy the marble run pieces correspond to components, while the marbles correspond to data. Marbles are inert, like data, and only respond to the outside which forces which move them. Unlike marbles, however, data can also be transformed; we can imagine a marble run with a section that paints all marbles passing through a shade of red, or perhaps smashes marbles into smaller marbles.
The pieces of a marble run are where the interest lies. Each piece has input rails and output rails, and accomplishes a specific task: raise the marble up in the air, convey the marble from A to B, or maybe to act as a staging area for marbles to wait at until a certain number have accumulated. The pieces upstream and downstream of any particular marble run piece don't really care how the task is acccomplished, they only care about the location where their rails meet, and perhaps properties of the marbles passing across those points like speed or quantity. Thus each marble run piece has an interface (the points where its inputs/outputs meet adjecent pieces) and implementation, and so are analogous to components in a program.
Using this analogy we can examine the properties of components a little deeper.
We can think of multi-threaded programs as being analogous to marble runs with multiple marbles in play. As long as each piece only encounters one marble at a time it doesn't need to be designed in any special way, but this restriction must be respected by any pieces upstream of it; in other words, this restriction is part of its interface contract. Other pieces in the contraption may perform synchronization, such as allowing marbles from multiple inputs to build up in a buffer area and only allowing one through at a time. It is up to the designer of the overall run to remain cognizant of the restrictions of each piece, and to position synchronizing mechanisms throughout accordingly.
With respect to passing non-data across interface boundaries, we could imagine a piece of non-data as being two marbles linked together with a piece of string. These connected marbles introduce much more complexity to the interface contract; each marble run piece must no longer rely solely on the state of the marble it has on-hand, but also on the state of a marble it can't even see! It's very possible that some pieces of the marble run will perform just fine in this case. But more complex mechanisms, such has an elevator, must properly handle the case so that the first marble to cross the interface doesn't get lifted before the second arrives. If the second is still tied up in a preceding mechanism the whole run could get locked up and break. And in pieces that handle multiple marbles at once you could end up tying your marbles into knots!
The marble run illustrates these concerns visually, but it also very much illustrates the way we should think of our programs as a whole. Many languages and frameworks misconstrue the relationship between logic and data, attempting to pass what amount to small-scale marble runs around on the rails of larger ones, each piece exhibiting a behavior which is perhaps understandable on the surface but which has too many quirks and edge-cases to be fully understood. We should prefer instead to model our components on the pieces of the marble run: each one with a fixed and simple behavior, ignorant of what occurs before and after it, and only dealing with inert pieces of data whose state is unaffected by any outside forces.
Next Chapter
Now that we have our basic tools in hand it's time to start discussing how to use them in more detail. The next chapter will discuss the composition of components into larger components and ultimately into large, complex programs which nevertheless behave reliably at all stages of their execution (coming soon!)
Hi! I'm available for remote contract work. You can learn more about me and my skillset by browsing around this site, then head over to my resume site to find my work history and professional contact form.
This site is a mirror of my gemini capsule. The equivalent gemini page can be found here, and you can learn more about gemini at my 🚀 What is Gemini? page.