Language Log

Executable articles

Here's an idea whose time has come: scientific and technical papers should include an explicit, executable recipe for generating their numbers, tables and graphs from published data.

Traditional scientific and technical journals require authors to specify their materials, methods and analytic techniques precisely enough to permit replication, because replicability is the foundation of the scientific method and the engine of technological progress. Now that the scientific and technical literature has become a networked digital achive, we can do better. We can expect articles to include an executable -- and readable and modifiable -- procedure for turning published data into the numbers, tables and graphs that play a role in their argument.

In a sense, such executable articles are self-replicating. Of course, genuine replication requires application to new data; but executable articles lower the barriers to such generalization. And there's certainly also a benefit to re-implementation of complex algorithms, to avoid the possibility of bugs or perniciously special cases -- but executable articles make this kind of replication more likely as well, just because they make it so much easier to get in the game at some level in the first place.

Among the many good consequences, I'd like to emphasize three:

• Minimizing fraud and error, especially the grayer (and commoner) forms, such as selection of atypical examples, omission of important caveats, and use of inappropriate models;
• Speeding up the virtuous cycle of science and technology, by lowering the barriers to entry into new research areas, and by making larger amounts of data available for re-use;
• Fostering education on a broad front, by letting students learn by doing, especially students who don't happen to be in one of the few places where crucial new data or new techniques are available.

There are many examples where this sort of thing is starting to happen. But it's far from being the norm, and there are plenty of problems in the way of making such practices more general. For example:

1. How can authors specify access to data in a durable way? It's not good enough to say "go to my web site and download the tarball" (though this is better than nothing). We need a set of practices for archiving datasets, and for providing standard, durable identifiers for particular versions. And we need stable ways to refer to an arbitrary piece of such a dataset, which can be executed to create a well-defined and reliable output form.
2. How can authors specify their analysis? There are numerous general- and special-purpose languages that people use in their research. Some are (now) widely available and easy for others to use -- code in Matlab or R or Java or Python -- but even for these, there can be version or operating-system or other contextual problems, and accessing archived data in a portable way poses additional challenges. Should some sort of autoconf system be used? And for how many years do we want to guarantee that an executable analysis should remain executable?
3. How can authors deal deal with issues of privacy and IPR restrictions on data or code? There will often be IRB ("Institutional Review Board") or similar restrictions on open publication of some raw data -- are there ways to anonymize such data that are morally, legally and scientifically acceptable? Do IRB protocols need to be adjusted to make this possible? What about data whose distribution is restricted by copyright issues?

And the biggest problem, of course, is the cultural conservatism of the academy.

As we look across the disciplines of science and engineering, we can see the seeds of plausible solutions to each of the problems, as well as some subdisciplines where moves in this direction are already underway. But I don't know any subdisciplines where executable articles, in the full sense, have become the norm. And I don't know of any complete and general solutions to the problems -- indeed, I doubt that a single solution is appropriate across all the diverse types of data, algorithms and disciplines.

The way to make progress, in my opinion, is for people to start experimenting more widely. This could be done by encouraging experimental but regular publication of executable articles in existing journals, or by starting new journals that specialize in such papers. Scientific and technical societies in relevant areas could also play a useful role in encouraging this development.

And funding agencies could do a lot to foster the development of needed infrastructure, and to encourage its use.

In the language-related fields where I work, executable articles would (in my opinion) be an especially good thing. If you're interested in pursuing the idea, let me know.

Posted by Mark Liberman at January 3, 2007 09:03 AM