Tag Archives: autotools

Development, Open Source

ptest because “make test” is insufficient

CAUTION: test in progressMuch thanks to autoconf and automake we have an established more or less standardized way to build and install tools, libraries and other software. We build them with ‘make’ and we install them with ‘make install‘. This works great and it works equally fine even when we build stuff cross-compiled.

For testing however, the established concept and procedure is not as good. For testing we have ‘make test’ or ‘make check’ which typically first builds whatever needs to get built for the tests to run and then it runs all tests.

This is not good enough

Why? Because in lots of use-cases we build software using a cross-compiler on a build system that can’t run the executables. Therefore we need to first build the tests, then install the tests (somewhere that is reachable from the target system) and finally execute them. These steps need to be possible to run independently since at least the building and installing will sometimes happen on a different host than the execution of the tests.

yocto-projectIntroducing ptest

Within the yocto project, Björn Stenberg has pushed for ptest to be the basis of this new reform and concept. The responses he’s gotten so far has been positive and there’s a pending updated patch to be posted to the upstream oe-core list soon.

The work does not end there

Even if or when this can be incorporated into OpenEmbedded and Yocto – and I really think it is a matter of when since I believe we can work out all the flaws and quirks until virtually everyone involved is happy. The bulk of the changes however, really should be done upstream, in hundreds and thousands of open source packages. We (as upstream open source projects) need to start doing testing in at least two different steps, where one step build everything that needs to be built for the tests and then a second step that run the suite. The two steps could then in a cross-compiled scenario get executed first on the host system and then on the target system.

I expect that this will mean a whole bunch of patches and scripts to have to be maintained within OpenEmbedded for a while, when things will be tried to get merged into upstream projects and I also foresee that a certain percentage of all projects just won’t accept this new approach and will reject all patches in this vein.

Output format

I think the most controversial part of these suggested “universal” changes is the common test suite output format. The common format is of course required so that we can “supervise” the output and results from any package without having to know any specifics.

While the ptest output format follows the automake test output syntax, I expect many projects that have selected a particular output format to rather stick with that. Hopefully we can then make projects introduce a separate make target or option that runs the test suite with the standard output format.

One little step forward

Building full-fledged Linux distributions cross-compiled that are completely tested on target will remain being hard work for a while more. But we are improving things, one step at a time.

Of course, the name ‘ptest’ is what the system is currently called by Björn within the yocto/OE environment. It is not supposed to be a catchy name for this idea outside of there. The ‘P’ refers to package, as opposed to for example system test and to make it less generic than simply test.

Development, Open Source

Autotool alternatives

7 Comments

Lots of people whine and complain on the set of build tools we often refer to as a collective by the term ‘autotools’. That term tends to include autoconf, libtool and automake.

I think a certain amount of criticism is warranted against this family of aged tools that are unix-centric, have cryptic ways to control them (I think there’s a reason m4 macros  is not widely used…) and they are several independent tools with a tricky mix of cross-breeding.A build tool

The upsides include them being well tested, fairly well known, there’s a wide range of existing tests done for them, they work fine when cross-compiling and they support building out-of-source tree just fine.

But what about the alternatives?

I spend time in projects where the discussion of ditching autoconf come up every once in a while, as sure as that the sun will rise tomorrow. The discussion is always that tool Z is much better and easier to deal with and that everything gets shiny if we just switch. That Z is a lot of different tools that are available today, including CMake, scons, waf or cDetect.

The problem as I always see and why I almost always argue against Z is that autoconf is old, trusty, proven and I know it. The Z tool is often much newer, less proven, less peoeple involved in the project know Z, use Z or know how to customize it (since new tests will be needed and some tests will need to be changed etc). So even though Z is sometimes accepted as a testing ground in my projects, a year or two after the Z was accepted – unless I myself have accepted it and joined its efforts – Z has lagged behind to a point where it isn’t good anymore since I don’t know it and most people are rather fixing the traditional autoconf stuff. So we extract the Z support again.

But if we would never accept new tools we would never evolve, and yes indeed autoconf and friends have their share of flaws.

The question is of course when to switch – what kind of project in what development state etc – and which alternative that is useful for a particular project. Me being a developer primarily working with plain C and working with lowlevel code and libraries mostly will no doubt have a different view than those who use other languages, who do more “apps” or perhaps even GUI programming…

Can you help me point out good build system comparisions and overiews? I’ve tried to find good comparisions but I failed. Just about all of them are written by the authors of one of these tools.

My ambition is to create some sort of comparison document myself. I think the comparison could include autotools, cmake, waf, scons, cdetect, qmake and ant. Any more?

(I got triggered to write this blog post after my post to the trio mailing list on this topic.)