Empowering Users to See the Past

“I know this model used to simulate. What has changed?”

The above quote is something I hear all the time, and I believe it applies to any number of circumstances when a user is sitting in front of an application. Just replace “model” and “simulate” with whatever is appropriate to the application at hand. Try “document”/“contain an image” or “video”/“fade out” or any number of things.

I want to speak to the general case of how to solve this problem for users and of the specific case on how I solved it for theoretical biologists as they incrementally build and simulate models toward a goal of reproducing a biological system.

I want to give the user the appearance of whole versions of their model/document at any point in time without the cost of storing the entire document at every point in time. It’s possible.

The Solution

The solution is simple, yet not trivial to implement. Give the user the ability to see what their model looked like at particular points in time. On the surface that may sound costly in terms of memory and storage. It may also sound costly in terms of time to retrieve the data if it’s coming over a network or needs processing to calculate. I would like to challenge those assumptions. In addition, what “particular points in time” should be used and for how long into the past? So here are the challenges enumerated.

1. Are the space and time costs significant? How are they addressed?
2. What particular points in time should be used for recording historical versions of the model?
3. How long into the past should the historical data be saved?

A quick side note on nomenclature. I’m going to use “model” to refer to the collection of user’s data that the user is working on or with. Sometimes this is called a “document” or “domain model”. I’ll say “record” (emphasis on “cord”, as in the verb), “record into history”, or “create a version” to mean when — by whatever means — a representation of the model at some point in time is stored or saved in a retrievable manner into memory or storage. I’m using “history” here to emphasize that this particular version of the model may be retrieved from the “past” at some future date and it may be different than whatever the current version is. There may be multiple versions of the model recorded into history; each with different values, properties, state, or whatever you would like to call the data in the model. A “version” of the model is a particular set of values, properties, or state at a particular point in time. And last, the current model or current version of the model is the values, properties, and state of the model the user sees in the present moment. Every other version has an earlier date/time and there are no versions with a more recent date/time.

Space and Time

The naïve approach is to store each version of the model each time it is recorded in the history, unmodified (as in, no compression or any other space/time saving actions). This may actually be an okay approach for some applications depending on the size of the model, the frequency of changes, and the amount of resources available. I don’t think it’s likely to work for most enterprise systems and I think this is one of the main reasons most developers never even consider storing model history.

If a developer had to implement some sort of compression or space saving algorithm themself then I think the idea put forth in this article would still fall flat. So what’s the answer? I think there are many solutions out there, but I’ll just share what I used.

I used Clojure and Datomic. Theses tools have already solved the space and time problems of storing complex shared structure of persistent data. I won’t expand on the depth of meaning in the last sentence; let’s just say, they’ll do the hard work for the software developer.

Datomic in particular is a database where nothing can be deleted — basically. In everyday use, data is “retracted”, which means it won’t appear in queries against the database as it appears in the present moment. I’m trying to avoid technical jargon here and explain this in layman terms for those not familiar with Datomic and the layman terms. However, the retracted data is still in Datomic and can be queried if the query supplies a date/time against which the query is to be performed! Voila!

By simply using Datomic as the persistent store for user data we already by default have a history of the user’s data!

I won’t go into detail here on how Datomic manages to not delete anything or into detail on why this is an okay thing to do. I will say two things on why this is ok. 1) “We live in the space age of computing.” — Rich Hickey. There is so much storage space available in the cloud, and it’s growing every day, exponentially. Therefore, it’s reasonable to assume we can create systems that do not delete and will never run out of storage. 2) We do need to be somewhat responsible in how we create those systems, though. User actions happen linearly. The number of user actions coming from a single person does not increase exponentially over their life; so as long as the amount of data generated by a single action is constant then we are operating within what I’m calling reasonable assumptions and responsible software development. Also, it is important that the size of data generated by a single user action, even if constant, is reasonable. We don’t want each user action generating terabytes of data to be stored even if that number is constant. We probably don’t want it to be gigabytes either, but that depends on the frequency of user actions. Anyway, I think you get the point. So if users are generating data linearly and systems capacities are growing exponentially, we’re good. I recognize there are some missing variables in this analysis — like growing the number of users, changes to the application, etc.. I’ll leave this to the reader to consider.

Model Versions

There must be model versions at key points in the history of a model in order for model history to be useful to the user. For me the decision about when and how often to create a version was pretty straight forward. Our application, ProcessDB, has a save/commit button the user clicks to save their work to the cloud. If the user decides their work is worth saving, then I can be sure it’s worth creating a version in the history; so ProcessDB does. There may be more points in time where creating a version would be useful, but for simplicity of development I decided to stop at this level. The user can also save any time they want if they wish to do so specifically to create a version in the history.

In some systems — like Google Docs and Medium (where I’m writing in this moment) — there is no save button and the system automatically saves on the user’s behalf. If your system is like this then you still have to address this question of when to create a version. If your system is automatically saving and you’re using Datomic for persistent storage, then every time you save you have a version, effectively, unless you actively do something to buffer data on the backend, which I wouldn’t recommend.

I can offer a couple suggestions on what to do in situations like this. One suggestion is to give the user the ability the ability to create named or tagged versions. This can happen in real time through some kind of button labelled “Create Version” or “Name Version” or could be done retroactively when the user is viewing a list of versions and selects an old version for tagging or naming. Another suggestion is to allow the user to show only tagged/named versions. This will filter the clutter of all the versions created at potentially hundreds of save points. One last suggestion would be to cluster versions by the hour or day and permit the user to expand those clusters and see the individual changes if they want. This idea is similar to how large search results on a map may be clustered and simply show the number of results at a point instead of showing every pin on the map for every result.

Record History

Forever. Per the arguments and “reasonable and responsible” programming I mentioned at the end of Space and Time (no pun intended), the history of a model should be recorded forever and never deleted.

You can’t know when a user will need some data or for what reason. The user may not have used the application for years or a particular model for years. Then, one day, for a reason completely inexplicable to a software developer, the user wakes up one morning and asks the question, “What did this model look like three years ago?” Or ten years ago! Why not?

In the age of upgrades and progress we have lost something; something precious. History. The Past. We have it in certain areas of our lives, but in our computers we treat the past as meaningless, troublesome… as this difficult thing that stops progress. Instead of creating computers and software that permit users to see into the past we create computers and software that discard the past and require the users to “migrate” their data. Their precious data. The better companies try to help with the migration process. The even better companies make the migration process transparent. The best companies treat user data as an archeological discovery — delicate and fragile — and these companies take the utmost care to be sure the data isn’t lost and remains usable.

I won’t get into all the details here about how to write applications that do this, but the first step is to actually keep the data. When an application saves a user’s model — for whatever reason, by user request or automated save or otherwise — the version saved is precious and not to be discarded, ever.

The Result

ProcessDB, our application, now has a history of models and the user can view the history for the model they’re working on. A model in ProcessDB is displayed and edited in a visual diagram. The user can select a version of the model from the history and it will be displayed in the same diagram as the current version of the model. ProcessDB will highlight the differences where model elements were added and, in a different color, where model elements were removed.

This is an incredibly powerful tool for theoretical biologists akin to a visual version control system. This kind of power is rare in end user space yet common for software developers.

Now, when a theoretical biologist publishes their model in a journal they can also create a version in ProcessDB and that version will be saved for all time. At any future date the user may compare the historical version on the publish date to their most recent version of the model to see what has changed! In the future, ProcessDB will allow branching from these versions. Already, ProcessDB allows creating a copy of an existing model if the user wishes to leave the published version unchanged. A difference can be performed between any two models in ProcessDB — so this powerful diffing feature can be applied between the copy and the unchanged published version of the model. These are powerful features rarely available to end users.

This is exactly the direction I think software development needs to go and it’s where I feel joy when developing software for end users.

A Note About This Article

I wrote this article to celebrate the software engineering accomplishments I’ve made, to promote what’s possible in software engineering, and to shamelessly promote myself and ProcessDB. I really believe there’s untapped potential waiting at our finger tips as software developers. We are taught and directed into paradigms and concepts that often don’t serve us or our customers. This is no one’s fault, in particular, it’s more a collective unchecked momentum. If you find yourself frustrated with programming then I suggest rethinking how you program. Or rather, the environment, paradigm, tools, etc. in which you program. My accomplishments were born out of frustration. I constantly thought, “there must be a better way. There must be something I’m missing.” I searched and I searched. And I discovered. … Search.