On December 14 1998 we released curl 5.2.

The project was still early back then and lots of things had not settled yet. In that release, which came only two weeks after 5.1, we introduced the --manual option, or -M for short.

Long before I started working on curl I learnt to value and appreciate Unix manpages. I more or less learned C programming using them, and I certainly learned my first ways around Unix shells and command lines reading manpages. My first Unix I spent a lot of time on was AIX. It was in the early 1990s, several years before I first used Linux.

Since some systems don’t have the fine concept of manpages, I decided I would help those users by bundling the curl man page into the tool itself. You can ask curl to show the curl manpage, with the -M option. The entire thing, looking very similar – mostly just lacking font details such as bold, italics and underline.

How do you bundle a manpage?

I suppose there are many ways to go about to make such a thing happen. In our case, we were already making and shipping a manpage in the nroff manpage format so it became a question of generating a text version using that page as a source and then convert the text version into C source code.

Converting a manpage to text was done with nroff. nroff is an ancient Unix tool that has been around for a long time and it existed on virtually every Unix flavor already back then. It seemed like a no-brainer to go with that, so that is what the curl build system would use.

Once the build scripts were tweaked it continued to just work. It became problematic only on platforms that lacked nroff – but to help smooth over that obstacle we also shipped the generated source file in distribution tarballs.

nroff really?

nroff is quirky tool. It generates the output differently based on environment details and over the years it would also subtly change its output several times that forced us to adjust the scripts as well.

Still, for as long as the curl manpage was primarily written in nroff format, it was challenging to generate the ASCII version any other way. We stuck with nroff.

Source format change

Earlier this year I blogged about how we finally changed the format of all the curl documentation files that create man pages in the curl project. We switched over to using markdown all over.

Even after that switch we still generated the built-in manual with nroff from the curl.1 manpage, that then was created entirely from a large set of source files written in markdown. The manpage was generated by our own custom tool.

The time was ripe

With firm control of the input file format and generating the output entirely with our own tool, it became a viable (and attractive) option to tweak the tool to offer an alternative output format. Allow it to render the output either as a manpage formatted file, or as an ASCII text file. Without involving or using nroff.

The time had come. We had suffered long enough. It was time to address this friction in the build system.

Yesterday, I merged the pull-request that finally, after 25 years, 2 months, 21 days removed the use of nroff from the curl build scripts.

The curl -M output after this change is not 100% identical, but it is close enough and looks very good and similar in style as before. I did not actually even try to make it a complete clone. In fact, when we generate the output directly from markdown instead of going via the manpage, we can actually make it a better text-only version than we could before.

I opted to still use a justified right margin of the text, because that is what it always used and after some casual initial comparisons I think it looked better than without an aligned right column.

nroff does hyphenation of words, which helps somewhat to make justified text easier and nicer, and our own script does not – at least not until I have figured out a decent way to do it. Like if the word “variable” is the last word on a line, it could be written as “vari-” on the end of the line and “able” could start the next line. I believe doing it badly is worse than not doing it at all.

Building this is easier

It is now (much) easier to build this from source, even on esoteric platforms like Windows.

I don’t think a single person will miss the old way of doing this.

I trust you have figured out already that I have the highest ambitions for the curl documentation. I want everything documented in a clear, easy-to-read and easy-to-find manner. This takes a lot of work and is not something that happens without effort.

Part of providing best-in-class documentation has in my mind always been to provide and ship man pages. And to make sure that those latest up-to-date man pages are always offered to users as good-looking webpages on the curl site.

Over twenty years ago I decided to render the web versions of the man pages by creating my own tool for that purpose: roffit. We still use it on the site today. It is a simple tool that converts nroff formatted man pages to HTML and adds cross-referencing links to other man pages. There is no concept of links in nroff, but I decided to use a combination of text and markup to use as a signal and it has worked out fine since. In part of course because I have then also subsequently made sure to format the curl man pages in this roffit-friendly way.

curl.1

We started writing the curl man pages in the nroff format some time in the beginning of 1999. Mostly then because we did not know or have any tool that would convert nicely into nroff and later on to make sure we got things formatted the way we wanted. This approach works and can be used, but nroff is not a terribly human-friendly format and over time we decided that documenting several hundred command line options in a single nroff file was not ideal.

In November 2016 we revamped the system and started to generate the curl.1 man page at build time from multiple source files. Every command line option would then be documented in its own stand-alone file. Easier to read, easier to manage. Yet we controlled the output nroff format so that the web version would still look excellent. The individual files got .d extensions (for document) but were mostly still using nroff, and were mostly all appended to create the result.

Over time, we introduced more convenience formatting aids to the .d files so that we could write them using less and less nroff, and as a result they also grew more readable in their source format.

The .d file format never had its own name. It is just how we documented 250+ command line options for the curl command line tool.

The libcurl docs however remained authored in nroff. In 2014, the number of libcurl man pages exploded when we started doing a separate document for every libcurl option. We went from 59 man pages in the project to 270 between two releases. Since then we have added many more. Today in early 2024, we have a total of 496 individual man pages in the project. 493 of them concerns libcurl.

Markdown

When curl started, markdown did not exist – that format was created first in 2004. The early documentation we created in the project that was not nroff, we made using plain text files. It was first in 2014 when we slowly started to convert some of our text documents into markdown.

Markdown has several upsides compared to plain old text. In particular it is easier to render good-looking web versions of them, which we do on the curl web site and it offers cross-document linking etc. Over the years we have converted almost all our text documents into markdown. Markdown is also a user-friend format as it is easy to read and easy to edit for contributors, new and old.

curldown

The move away from using nroff for the libcurl man pages came with the introduction of the “curldown” document format in early 2024. Meant to make it easy to generate the same kind of nroff files we previously hand-edited, but using “markdown”. I use quotes around it because it is not a full-fledged markdown support, but it looks similar enough to trick casual users and we decided to use “.md” file extensions to make editors, GitHub and more treat and show them as such.

The file format is inspired by the one we created for the command line tool: with a header holding meta-data and then simplified markdown. Easy to read, easy to edit – for everyone. No more strange nroff syntax to copy and paste. The fact that GitHub render the sources files clean and good-looking is also attractive.

Switching to a markdown lookalike format had some other side-effects: suddenly a lot of CI jobs we have running caught these files and had a go at them: spellchecks, prose checks, checks for capitalized words after periods and more. This improved the language significantly.

Also, generating the nroff files rather than editing them has more benefits than just avoiding the baroque syntax: it also allows us to change styles and escape certain sequences through-out all documentation with just a few fixes in the scripts producing the output.

After this 47,000 lines added and removed patch was merged, we have a completely new source format system for the libcurl docs, that generates the same style of nroff output we had previously. With improved language.

But then why not…

With the huge switch to “curldown” for libcurl options, it felt wrong to keep the old curl .d files for the curl.1 man page, seeing as they were almost there already. I did the work and converted them over to the same markdown lookalike style I did for libcurl: they are now also .md files and now they too get spell-checked and scrutinized much more.

I took this move one step further: previously we generated the single curl.1 man page using a fixed header file and a footer file that we prepended and appended to the nroff generated from the (converted) .d documentation.

Now, we instead have a mainpage.idx file that lists which (markdown) file names to include and convert to nroff. This allows us to split the two big headers and footers into several smaller markdown files, each with its own header. Like “DESCRIPTION”, “URL”, and “GLOBBING”. Separating them makes them easier to manage for contributors and the index file makes it easy to change the order of sections etc.

We end up with a curl.1 man page that looks quite similar to before, but with improve language and easier format to edit the documentation going forward.

When I type this, we have 836 files with .md extensions in the curl git repository. At the time of the most recent release, we had 61.

Future

I’m sure it does not stop here. I already have some ideas on how to improve this setup further.

I’m considering doing something to completely avoid the nroff step for the website HTML versions of the man pages and instead go straight from markdown.

The curl tool has the manpage built-in, (shown with the -M option), and I would like to fix the build to do this without using nroff.

I have some more CI jobs to add to verify and check the language in the markdown files. Mostly to make sure the language is consistent.

Here are six requirements that I have on a project for it to reach my gold approval for stellar docs. Then something about what I’ve done recently to further improve the docs for curl.

Your docs belong in the code repository

It needs to be next to the code so that authors and contributors can update/read the docs while working on the code or docs. Providing it in a separate repository or otherwise separated will undoubtedly lead to discrepancies sooner or later. Similar to how all wikis are always wrong.

Your docs is not extracted from code

Admit it. If you browse around you will realize that the best documented projects you find never provide that docs generated directly from code. Such generated docs can still provide value, but you will not reach gold level without more effort.

Separated docs encourage more writing and writing by people who are otherwise scared of touching the code or thinking that fixing spelling errors in the docs isn’t worth patching the code for. It also allows for other and better formatting on the docs.

Your docs features examples

Users always ask for (more) examples. You can never go wrong by providing examples. If your docs don’t have enough examples, you’re not doing the docs good enough yet.

You document every API call you provide

Fewer things make me sigh more than when I have to dig up the source code and from that try to figure out exactly how an external and publicly provided API works. Yet this still happens regularly even for libraries that have been released and maintained for decades.

In one fairly recent instance I reported such an omission in a popular library. It took them over two years to add it.

Long-living libraries should also provide information about from which versions certain functionality or options exist or can be expected to work or not work.

Your docs is easily accessible and browsed

Good docs means that we can find what we’re looking for and that the documentation flows and is easily read and understood. Ideally, even simple google searches for API details in your library should lead us to suitable entry points.

Preferably, the documentation should also be provided for proper off-line reading, meaning man pages or something similar that can be browsed when disconnected from the Internet.

Your docs should be easy to contribute to

The docs should be easy for contributors to help out with (independently from the code if desired). That also includes that they should be easy for contributors to build and render locally so that they can test and view their updates while working on them.

Documentation in the curl project

I want the documentation for curl and libcurl to be known, recognized and widely admitted to be world-class.

I want the curl documentation to be of a quality and content to make users not able to find competitors or similar projects with better docs.

Documentation in curl is not an after-thought. It is not a second-tier component. It is a crucial and important foundation that allows users to use, trust and rely on our products. We require that new changes or improved functionality are provided with the corresponding updated and accurate documentation.

We also try to verify and check the docs as much as possible with scanners , tests and tools.

Non-stop iterating is key

I maintain that our documentation is as good as it is today a lot thanks to us very rigidly sticking to our guiding principles: compatibility and not breaking existing behavior. Documentation we wrote decades ago is still valid. It gives us plenty of time to keep refining and polishing the documentation of a feature that doesn’t change. No documentation was perfect already at the first attempt, but after numerous iterations and improvements chances are it is better. Time is on our side. And we are never done, documentation can always be improved.

I’m putting in the work

A few days ago I talked documentation with someone and when doing so I thought about what guiding principles I think we should put on project documentation. What I’ve listed above basically.

In then dawned on me that the current man curl.1 man page is actually not featuring that many examples, in spite of it being 3535 lines long. I pondered a little bit on how best add some, and then dove in and extended our system that generates it from the hundreds of individual files that each describe a single command line option. They should of course all offer at least one example!

Having 242 command line options (as of right now, it will be more soon), that sudden idea that seemed simple enough turned into a quite gruesome work and I spent many hours walking over the options to make up examples. I also made sure that our build system now returns an error if there’s a command option without an example in the documentation! This way, we can be sure that also all future command line options will have examples in the man page.

This made the curl.1 man page grew with over 1200 lines!

libcurl options too

A few years ago I did a manual effort and made sure most man pages for libcurl options include examples, but I never made that into a test or anything so there’s nothing that forces us to stick to this.

Having started this journey, I decided now was the time to add that requirement to the scripts. I extended test case 1173, which already scans all man pages to verify some basic syntax, to also check that man pages for options feature an EXAMPLE section.

There are at this moment 374 stand-alone man pages for libcurl options. Only ten of them were detected to not feature good enough examples and it wasn’t very hard to fix this.

Consistency is good

Having just extended the man page checker, another idea came up.

I made the script also check that each man page features the correct set of sections, in the required order! I’m a true believer in consistency and that using the same set in the same order will make the docs easier to read and find information in, and checking for these things will make sure that all future additions will be forced stick to the same.

The mandatory sections for libcurl option man pages are right now, in this order:

NAME
SYNOPSIS
DESCRIPTION
PROTOCOLS
EXAMPLE
AVAILABILITY
RETURN VALUE
SEE ALSO

These man pages are allowed to have other sections as well, and they can be placed anywhere among the mandatory ones, but the eight section headers that has to be there has to be in that order.

Cross-references

While at it, I also extended the man page scanner to check that all references in all curl man pages to libcurl options are verified to actually refer to existing options, to find typos. Ironically this extra check turned out finding exactly no such typos in the current 463 man pages!

Future

The outcome of the work I write about here will of course be merged asap and will be part of future releases and on the website.

We should keep thinking of more ways to improve the documentation and for more ways to verify and cross-reference things mentioned in the docs to increase its accuracy and detect typos.

If you find a problem, an inferior wording or just something you think we should improve in any curl documentation, file a bug or a PR! We also try to make this as easy as possible for users to do directly from the curl website by providing bug-reporting direct links from the documentation pages.

Updates

I added the sixth “rule” days after the initial post after Willy Tarreau’s feedback.

I’ve worked hard on making the presentation I ended up calling libcurl under the hood. A part of that presentation is spent on explaining the main libcurl transfer state machine and here I’ll try to document some of what, in a written form. Understanding the main transfer state machine in libcurl could be valuable and interesting for anyone who wants to work on libcurl internals and maybe improve it.

Background

The state is kept in easy handle in the struct field called mstate. The source file for this state machine is called multi.c.

An easy handle is always in exactly one of these states for as long as it exists.

This transfer state machine is designed to work for all protocols libcurl supports, but basically no protocol will transition through all states. As you can see in the drawing, there are many different possible transitions from a lot of the states.

libcurl transfer state machine

(click the image for a larger version)

Start

A transfer starts up there above the surface in the INIT state. That’s a yellow box next to the little start button. Basically the boat shows how it goes from INIT to the right over to MSGSENT with it’s finish flag, but the real path is all done under the surface.

The yellow boxes (states) are the ones that exist before or when a connection is setup. The striped background is for all states that has a single and specific connectdata struct associated with the transfer.

CONNECT

If there’s a connection limit, either in total or per host etc, the transfer can get sent to the PENDING state to wait for conditions to change. If not, the state probably moves on to one of the blue ones to resolve host name and connect to the server etc. If a connection could be reused, it can shortcut immediately over to the green DO state.

The green states are all about setting up the connection to a state of fully connected, authenticated and logged in. Ready to send the first request.

DO

The green DO states are all about sending the request with one or more commands so that the file transfer can begin. There are several such states to properly support all protocols but also for historical reasons. We could probably remove a state there by some clever reorgs if we wanted.

PERFORMING

When a request has been issued and the transfer starts, it transitions over to PERFORMING. In the white states data is flowing. Potentially a lot. Potentially in both or either direction. If during the transfer curl finds out that the transfer is faster than allowed, it will move into RATELIMITING until it has cooled down a bit.

DONE

All the post-transfer states are red in the picture. The DONE is the first of them and after having done what it needs to round up the transfer, it disassociates with the connection and moves to COMPLETED. There’s no stripes behind that state. Disassociate here means that the connection is returned back to the connection pool for later reuse, or in the worst case if deemed that it can’t be reused or if the application has instructed it so, closed.

As you’ll note, there’s no disconnect anywhere in the state machine. This is simply because the disconnect is not really a part of the transfer at all.

COMPLETED

This is the end of the road. In this state a message will be created and put in the outgoing queue for the application to read, and then as a final last step it moves over to MSGSENT where nothing more happens.

A typical handle remains in this state until the transfer is reused and restarted, in which it will be set back to the INIT state again and the journey begins again. Possibly with other transfer parameters and URL this time. Or perhaps not.

State machines within each state

What this state diagram and explanation doesn’t show is of course that in each of these states, there can be protocol specific handling and each of those functions might in themselves of course have their own state machines to control what to do and how to handle the protocol details.

Each protocol in libcurl has its own “protocol handler” and most of the protocol specific stuff in libcurl is then done by calls from the generic parts to the protocol specific parts with calls like protocol_handler->proto_connect() that calls the protocol specific connection procedure.

This allows the generic state machine described in this blog post to not really know the protocol specifics and yet all the currently support 26 transfer protocols can be supported.

libcurl under the hood – the video

Here’s the full video of libcurl under the hood.

If you want to skip directly to the state machine diagram and the following explanation, go here.

Credits

Image by doria150 from Pixabay