Portia: a text processor for literate programs

better ask Wikipedia

Because no literate programming preprocessor allows any document language and any programming language. I, for one, like to embed fragments of OCaml in asciidoc files.

Here are the requirements and short rationals:

R1. Must be able to build itself

That’s the fun part.

R2. Must be bootstrapable with funnelweb

Because funnelweb is quite usable. Especially, it is agnostic with regard to the programming language and almost so regarding the documentation language.

It follows from these requirements that we should either follow funnelweb syntax, which is ugly, or build a tool that’s flexible enough to act like funnelweb (or at least, that can understand a workable subset of funnelweb syntax).

Ideally, all escaping sequences of the macro system can be redefined. When bootstrapping (with the actual funnelweb) we do not mind the quality of the generated documentation since we can produce a better documentation (and source code, thanks to requirement R3 & R4) from recompiling with the bootstrapped processor.

R3. Add correct file and line number informations into generated source code

Whatever the programming language that’s being used of course.

R4. Do not output doc; rather, make code extraction flexible enough for the doc to be written in any documentation language in the first place

The documentation being the important part, do not interfere with it.

R5. Code blocks may be inline

Like short mathematical formulas are better inline.

R6. Must not require to have all the text, nor all the code, present in memory

Funnelweb builds a whole source code representation in memory before outputing anything. This is frightening. Despite I’ve never used literate style for anything but trivial program I believe the technique suits huge programs just as well.

R7. Can split content into several files

Quite obvious.

R8. Named code fragments may be defined from several places (even in different files)

In literate programming, the human reader must remember the main names used by the program as well as the names used to reference important code fragments. In order to limit the number of unessential definitions, Funnelweb allows to build a macro definition incrementally so that you must not introduce temporary names just to insert commentary in between two related code fragments.

R9. Support an include directive

Since we do not care about the order in which the code fragments will be encountered, and we do not care neither in what order we scan the documentation (since we do not produce a documentation according to R4), then we can merely be given a list of files to scan from the command line. We do not need an include directive as funnelweb (and other) does.

Still, many documentation language has an include directive and if we were able to follow it then we could alleviate the user from the need to maintain this list of files (since we could find everything from the root document).

So we do both: we will scan everything from the command line and additionally, if we are taught how to spot an include directive, then we will try to follow it.

For generating source code (remember R4: we do not have to generate the doc), we need a few directives:

We are going to use the OCaml language because it’s compendious yet fast.

The program basically reads its configuration file (or load its compiled configuration file), then proceed with reading the given file, building a dictionary of block definitions and loading additional files along the way, keeping track of code blocks definitions and modification time of the used files, and checking everything.

Then it outputs the code, rereading the files to fetch code blocks so that we do not have to hold in memory a quantity of information equivalent of the resulting source code.

So the basic skeleton, given a set of configuration files plugins and a list of source files to proceed srcfiles, looks like:

Notice that configuration does not appear here, nor does harvested definitions. They lies in global variables, which suits this short lived program just fine. We won’t worry about configuration parameters yet. Suffice to say that all of these global parameters, regular expressions and functions (remember some filters may be functions) are references defined in a module unambiguously named Config.

By loading a conf, we merely want to load a compiled .cmo file:

This is enough for a user to choose between several plugins (funnelweb, …).

So we only need this entry point to parse command line arguments and we are done with the boring work:

So we have the Main module (linked last):

Let’s focus now on our main data type, the code block definition.

It is well known that naming things is the most difficult part in programming. This has to do with the fact that a large part of programming consists in inventing many simple but abstract concepts with no counterpart in actual life or pre-existing abstractions from other intellectual fields. Sometime we can borrow a term or two from mathematics but mostly we have to give names to things that are relevant only to our program, maybe not even for its full lifetime.

So we have to repeatedly look for designations close enough to what we actually mean that, with enough context, it will become clear what concept they refer to. Of course, this context is not easy to acquire without prior knowledge of these somewhat arbitrary definitions, so it takes some time and effort to pull oneself out of this catch-22 situation.

So, after this short introduction have hopefully inclined my reader to leniency, let’s ask ourself what the name of a program fragment should be. Shall we keep calling it a "program fragment"? But this implies that this is part of a program, which is not required. Should it be an "extract", considered this fragment is separated from the rest for display purpose? Or a "phrase", considered it’s part of a larger "discourse" (the program)?

Or, if we forget what we manipulate to consider how we manipulate it, should it be called a "macro body"? Merely a "macro" or "body"? Or a "definition"? This point of view is seducing since it makes our program a more general purpose text processor rather than a specialized tool for literate programming (of course a type or variable name is not really part of the running program and so cannot alter its behavior in any way, but I believe in the power of names to influence our reasoning about abstractions and that giving generic names help building generic programs).

Let’s call these code fragments "definitions", then.

What information is there in a definition? We have already seen that we need its location used both for error reporting (file name, line number and column) and for fetching it quickly on demand (offset and size in bytes).

Also note the recording of the mtime so that we can tell, when fetching the body, that the file have not changed since we collected this definition.

with the convenient convention that line and column numbers are actually offset from the start, thus start at 0, which allows us to add them naturally.

Rather than having a single location, we want to allow for a definition to be split across many locations (the body of the definition is then the concatenation of all fragment in order of appearance - which is specified by the depth first exploration of included files, the ordering of files in command line and finally the order we met definitions in a given file)

The user should be warned about any code fragment that is not, directly or indirectly, referenced from an output definition.

Notice that it’s possible that there is no location at all (empty list), meaning the definition was missing (might be conveniently allowed with ignore_missing flag, to make it possible to generate a valid program and write the extensions later on).

It’s always a good idea to write proper printers for any new type. This may looks fastidious but you are actually doing yourself a favor: better have these printers ready before they are needed than to have to write them quickly while struggling with a bug. Especially when using Batteries which make writing and using such printers so easy.

So here they are:

Now obviously we also want to fetch a definition body from its file (checking mtime):

Then, we will need a way to add definitions to a global registry, and the associated lookup function. Definitions are created from a file, offset and length (line number and column number are not given and will be computed when registering, so that plugins author work is limited to the minimum) and of course the identifier for the definition.

Where location_in_file is responsible to return a correct location (up to proper line and column numbers) from the file name, offset and size:

Now the last part: expansion. Given a function PortiaConfig.find_references (supplied by the configuration) that’s able to spot all expansion points from a non expanded body, and the registry of all known definitions, let’s build a function that will return the complete expanded body (or signal a problem).

Where each substituted definition is properly indented according to its insertion point. We must now complete this module with the functions we used up to here for helping dealing with text files and locations:

Regarding linenum, this function depends on the programming language used. The default implementation from Config will not output linenum directives. But dedicated plugins are easy to write. First, for ocaml:

and for C:

Notice that those directives follow the GNU convention that:

One more word about the linenum directive in OCaml. It is documented in the chapter 6.1 (lexical conventions) of the OCaml manual, and from this documentation it appears that it is not constrained to appear alone on a line. We do make some effort to place these directives on dedicated lines, in order to generate better looking source files.

Also, notice that we must insert a linenum directive at the insertion point of each definition body and after each expansion to return to previous location.

With these functions we are now ready to start the real job of parsing input files(s) and writing output definitions.

Parsing is a pretentious appellation, since we merely need to spot three things in the input files:

For now we do not want to impose any format to these marks so in all generality we are going to read in memory a whole file and ask a configuration provided function to return the list of additional files to scan and the list of definitions that can be found in the file content.

So "parsing" is just:

That we can group, with some helper functions to be defined later, in a parse module:

Also, we want to be able to attach several code fragments to the same name (see R8), with the actual expansion being composed of the concatenation of these fragments. To handle this, we will merely register several definitions with the same name, and when writing the output of a given definition we will append all bodies in order of appearance.

Once all definitions have been gathered we can iterate over all of those which must be written into a file, retrieve their (expanded) body then write it into that file. We will not directly overwrite the destination file, though, rather create a temporary file and replace the older file only if the new one is different. We do this to avoid unnecessary touching files, thus triggering whole rebuilds, each time a single compilation unit is effectively modified.

And that’s all we need in our Output module:

We have seen so far only five parameters taken from the configuration, the first three being references to functions taking a file content as a string and returning substrings of interest:

and the others being simpler function to post-process or beautify the output:

So that our Config module thus far is merely:

Notice that separate compilation of this module imposes that we have to declare the types of these references.

Remember form the Main module that we will load by default the funnelweb plugin, so when running portia without option it will behave (loosely) like funnelweb. This plugins will not implement linenum, though, so no line number directives will be outputted. It would be nice if by default the linenum function was relying on output file name to choose from a set of predefined implementations, though.

We have glossed over many trivial details to get there, but the program would not be complete without those.

A mode in which portia just output Makefile compliant dependencies.

Indenting the generated source code makes column positions reported by compilers (and other tools such as annot for OCaml) different from the one in the source document, which somewhat defeats the line number information feature. Therefore it should be possible to disable indentation altogether for more accurate position reporting.

Add a warning at the beginning of generated files that they are automatically generated and should not be edited manually.

Do not output linenum for shell because they mess with the shebang. tests/*.expected is another case of annoying linenum. Implement per language linenum as suggested at the end of config.fw. In other words, find_definitions should return the language (the file extension is good enough).