A QNX Porting - Packaging Example
By William Bull and Peter van der Veen, QNX Software Systems Ltd.
Introduction
This article will illustrate how port and package a well-formed GNU source
distribution into a QNX® package. We assume you have installed the
development packages and the new package builder. If you don’t have the
command line packager, click here or connect to the WWW Repository with the
Package Installer.
The steps that follow work for most UNIX software that uses GNU Autoconf and
the related GNU configure scripts. If you’re short on patience, feel free to
skip to then end for the compilation and packaging summary.
Which End is Up?
A long time ago, life was simple for programmers. We all coded to meet the
requirements of our own operating system, our particular machine, or our own
special tools. Eventually, however, computers became more prevalent and
everyone wanted to access everyone else’s software. Regardless of the
motivation-greed or altruism-a demand appeared for compatibility across
architectures and across UNIX variants.
With the creation of computers came the requirement for standards.
Unfortunately, as we have attempted to establish those standards, many of
them have either been ignored, deprecated, or superseded. Today, we have
processors that store numbers differently in memory, leaving us with endian
incompatibilities. We have standards for writing the structure of
executables onto a file system, giving us differing executable formats. We
have shortcomings and missing features in the various POSIX operating system
implementations, stopping our code from compiling.
But fear not! As the confusion of software incompatibilities mounted, the
demand for cross compatibility grew in equal measure. Developers of QNX
Neutrino®, SunOS, Linux, Ultrix, and host of other POSIX operating systems
now have tools at their disposal to make robust self-configuring source code
that, with a little help, can detect the endian flavor of your processor,
the existence of OS-specific system calls, as well as the lack of pertinent
features like threading, memory-map file manipulation, or TCP/IP socket
libraries.
- Autoconf
The Autoconf package contains a collection of scripts that test for the
variations of each UNIX-type operating system. The results of the Autoconf
scripts are used by the software to help with compilation. For example,
Autoconf may make suggestions that may cause additional header files to be
included, or that add specific command-line options to the compiler or
linker.
- The Configure Scripts
There are several standard files used by Autoconf to detect your machine and
its feature set. The first file, configure.in, has the Autoconf macros that
are used to configure the source. The files, config.guess and config.sub,
are used to identify the architecture for your machine. And finally, we have
the configure script itself. Once called, the machine type is detected,
small programs are run to detect your machine’s features, and the software
may then be compiled.
- Automake
The Automake package, although it is similar in name to Autoconf, has a
slightly different purpose. Automake, used in conjunction with its
configuration files (Makefile.in, Makefile.am), helps programmers to build
makefiles. It can be used in conjunction with configure.in files, but this
rarely affects the porting process.
- Libtool
Developers who use Automake sometimes take advantage of the Libtool package.
Libtool simplifies the creation of shared or static libraries. The use of
Libtool is hidden within Autoconf’s makefile macro (Makefile.am) files.
There will be no way for you to tell if your source code needs Libtool by
looking at the files in the source directory.
- xmkmf
Finally, when porting X applications, you should be aware that the
X-Make-Makefile command, xmkmf, is almost always used before compiling the
application. In fact, most people call “xmkmf -a” to skip having to call
“xmkmf; make makefiles; make includes; make depend.”
Starting The Packaging Process
It’s encouraged that you port and contribute to all software projects
(public and private alike). But what is most important is to recognize good
form and coding practice. By understanding how others use cross-platform
configuration tools correctly, you will be able to write better code-and
code that can be used and reused.
Let’s assume we have decided to port some software born out of the GNU
Project and that the software uses a standard configure script. (You can
find more about the GNU Project and GNU software packages at
http://www.gnu.org.)
And let’s pretend you have downloaded version 1.0 of Stream Zinger, a
fictitious web site stripper, crawler, and indexer. At this point, we’re not
so interested in the porting of the product as we are the installation and
packaging of it.
Eight Steps to Stream Zinger (the Best Product on the Planet)
- We’ll pretend that you’ve got the tarball, the tar, gzip version of the
source. First, unzip and untar it and go into the directory:
tar -zxvf streamzinger-1.0.tar.gz
cd streamzinger-1.0
- Let’s assume we’re porting/building Stream Zinger for an x86 machine. The
first thing we’ll do is create a directory to hold the relevant files for
your port.
mkdir x86-pc-nto-qnx
cd x86-pc-nto-qnx
- Now we’re in the x86-pc-nto-qnx directory and we want to tell Configure
to do its job, but we may have to make sure the configure scripts are up to
date. If, for example, Stream Zinger was packaged a year or two ago, it’s
quite possible that it won’t have the proper support files to identify the
QNX realtime platform environment. (You’ll know that’s the case if the
…/configure script complains that it’s unable to detect the host
environment.) There are two ways to circumvent this potential problem:
3 a. The first involves using the Automake utility. When you run Automake in
a directory without a config.sub or config.guess, it will create the latest
configuration files needed to detect our operating system.
cd …
rm config.sub config.guess
automake --add-missing
libtoolize -f
cd x86-pc-nto-qnx
3 b. The alternative approach is to go right to the source and get the
latest config.sub and config.guess files. By ftp’ing to gnu.org, you can
update the files in your package with the latest versions published by those
maintaining the configure scripts.
cd …
rm config.sub config.guess
ftp to ftp.gnu.org
login as anonymous
cd pub/config
mget config.sub config.guess
cd x86-pc-nto-qnx
- We’re all set to call ./configure but, just to make things safer, we’re
going to add two environment variables before our configure script.
Let’s review the command-line options we’re going to use:
–prefix=/opt
All packages should be installed into /opt. That means that if Stream
Zinger has a man page, it should be installed into /opt/share/man…not
/usr/share/man, andnot /usr/local/share/man. These and other file-system
layout issues will be addressed in future articles about the QNX
Hierarchical Filesystem Standard, a derivation of the HFS found at
http://www.pathname.com/fhs/. The prefix option adjusts where the makefiles
for Stream Zinger are placed into the operating system. More importantly, if
Stream Zinger needs files that it normally places in /usr/share or
/usr/include, these will now be adjusted to /opt/share and /opt/include.
–host=x86-pc-nto-qnx
By specifying a host and manufacturer, nto-qnx, the configure script,
will be able to determine the features of the QNX Neutrino environment. The
processor and architecture, x86-pc, will give the configure scripts a means
of determining the endian nature of our environment, and offer Configure the
chance to run any other necessary test scripts.The host option is the key to
Configure-it explicitly tells it how to properly detect a QNX Neutrino
machine. Many source distributions will work without specifying a host, but
only because they look at other aspects of the system to detect the value
(such as the arch command). By providing a host, the configure script can
skip the detection process.It is possible that you’ll be unable to update
the configure files and that the x86-pc-nto-qnx host string won’t work. In a
pinch you can try using --host=i386 to compile the software in a fairly
generic x86 manner.
–src=…
We created the x86-pc-nto-qnx directory inside the Stream Zinger-1.0
source directory, and that’s where we’re running the configure command. We
want the configure script to make sure all the makefiles will work correctly
when we execute the make command from within the x86-pc-nto-qnx directory.
Thus, the source directory is one directory back.
–enable-shared or --enable-static
The command configure and the command libtoolize -f should
automatically take care of the way libraries are built. But you may
explicitly request shared libraries (.so) or static archives (.a) to be
built with these command line options.
So now we actually call the configure script to adjust where the
installation will occur, adjust the makefiles for our operating system, and
define where the source files are located.
…/configure --prefix=/opt --srcdir=… --host=x86-pc-nto-qnx --enable-shared
make
- At this point, we’ve probably had to do some debugging to make sure
everything worked. Occasionally, a header file has a different name or it
may turn out that that the configure script didn’t detect a missing feature
or system call. We’d like to believe that these things never happen…but
occasionally they do. There aren’t a lot of general solutions to these
problems; they’re usually handled on a case-by-case basis.Let’s assume that
our fictitious product, Stream Zinger, has compiled perfectly. We now want
to install it all into a directory for packaging purposes. This directory
can be named anything; we’ll name it installdir. This means that if Stream
Zinger planned on placing a file into /opt/bin, we now want it to install
into installdir/opt/bin. So all we have to do is make the install directory
and convince the makefile to install to the right place.
There are two popular methods of accomplishing this goal:
DESTDIR=$(pwd)/installdir/opt
The DESTDIR environment variable is understood by all makefiles that
are generated by Automake. It is created to provide an intermediate area for
packaging up an install tree. This environment variable, although widely
supported, may not work everywhere, especially on packages that don’t use
Automake.
prefix=$(pwd)/installdir/opt
The prefix environment variable is another variation on the DESTDIR
concept. Again, most makefiles will support either of these, but it’s
crucial to have the installation actually end up in $(pwd)/installdir/opt.
These are the two common means of creating a packaging “staging area.” We’ll
pick one and continue with our port.
mkdir installdir
make prefix=$(pwd)/installdir/opt install
- At this point we’ll use our command-line package building tool, packager.
Its job is to sort the files contained in installdir into a series of
installable packages, and make a package manifest that describes the port of
Stream Zinger.Packager asks a lot of questions, most of which are optional
(but it would be really good for you to answer every one, if you can). But
there are a few that are required that you should take a moment to
understand.
-
Product Identifier: This is a short string that isn’t read by users.
It’s used for things like package dependencies. In our case, we might used
the product identifier of szinger. -
Product Name: This is a name that users will probably see. Try to
capitalize your names as if they are titles, like Stream Zinger. Never
include the version number in your product name. -
Vendor Identifier: This is a single short word used to identify the
person (or organization) releasing, maintaining, or claiming ownership of
the package. Since Stream Zinger is part of the larger body of projects
known as the GNU Project, we’ll give it a vendor identifier of gnu. -
Release Date: This probably isn’t today’s date. This is the date that is
on the streamzinger-1.0.tar.gz file. That date represents the day that the
authors of Stream Zinger released the source code. We’ll pretend that it was
released on 1998/05/11. -
Version: We know that it’s version 1.0. It was in the name of the
original streamzinger-1.0.tar.gz file.
This should give you some idea of what packager is asking for. Future
articles will go into each question in depth. So let’s just start to package
up the files in the installation directory we have created.
packager installdir
…answer a number of questions…
7. After running packager and answering all its questions, we’ll see that it
created a file called szinger-1.0-x86-gnu.qpr. That .qpr file is a web (one
click) package. If the Voyager™ browser sees a link to a .qpr file, it
knows to launch the Package Installer in response to a user’s click.
-u szinger-1.0-x86-gnu.qpr
The -u option for pkg-installer requests to connect directly to a
repository. This can be either a path to a .qpr file or a URL to a remote
installation repository.
Since we’re running the QNX realtime platform, we can use the QNX’s
graphical package installer to test your package.
pkg-installer -u szinger-1.0-x86-gnu.qpr
8. Let’s pretend that we install it and not everything works smoothly.
Perhaps there are spelling errors in our descriptions, or a script doesn’t
run. We’ll just repackage it without rebuilding Stream Zinger.
-m szinger-1.0-x86-gnu.qpr
Of course, we really don’t want to answer the questions all over again.
So we’ll use the -m option to provide the packager with default values. You
may pass any .qpr, .qpk, or .qpm file as a source for default values.
Repackage the installdir using the package we just made.
packager -m szinger-1.0-x86-gnu.qpr installdir
9. In the event that our package does not work due to some compile-time
options, we must do the following:
Go back to the streamzinger-1.0 source directory, change the offending
file(s), recompile, and install like we did in step four. Of course now we
need to rebuild that package, but we want to make sure that .qpr doesn’t
have the same name as last night’s .qpr file. It will be a substantially
different distribution now that those compile-time changes have been added.
Remember, we can’t change the version number since we’re porting public
software. What we can do though is use the -b+ option:
-b+
The package building tool, packager, will increment a build number when
the -b+ option is used. This number will appear in the name of the new
package. For example, 1.0-b1, 1.0-b2, etc. This is especially useful for
automated (or nightly) builds of a source tree.
If you ever distribute a package, recompile and distribute again, always
increase the build number.
make
rm -rf installdir/*
make prefix=$(pwd)/installdir install
packager -b+ -m szinger-1.0-x86-gnu.qpr installdir
Summary
Let’s summarize what we did to build a package, just to keep things clear in
your mind.
(Untar the source.)
tar xz streamzinger-1.0.tar.gz
(Go to the source directory.)
cd streamzinger-1.0
(Make a host directory.)
mkdir x86-pc-nto-qnx
(Go to the host directory. )
cd x86-pc-nto-qnx
Update the config files. (if you have trouble with the configure
script…)
cd …
automake --add-missing
libtoolize -f
or
ftp to ftp.gnu.org/gnu/config
mget *
cd x86-pc-nto-qnx
(Configure the source.)
…/configure --prefix=/opt --srcdir=… --host=x86-pc-nto-qnx --enable-shared
(Make.)
make
(Make install directory.)
mkdir installdir
(Install.)
make prefix=$(pwd)/installdir/opt install
(Build a package.)
packager installdir
(answer a number of questions…)
(Install the package.)
pkg-installer -u szinger-1.0-x86-gnu.qpr
(Remake the package. )
packager -m szinger-1.0-x86-gnu.qpr installdir
If there’s a problem package:
Rebuild.
Clean out the old install.
Reinstall.
Repackage a new build. make
rm -rf installdir/*
make prefix=$(pwd)/installdir install
packager -b+ -m szinger-1.0-x86-gnu.qpr installdir
[Boy that looks a lot better in html than in text…]
Hopefully this article has given you an overview of how to go from
source to compiled package. There will undoubtedly be more articles on
porting, building, and using our development tools, but at least you have a
start! As always, if you have questions, recommendations, or would like to
make comments, feel free to post on the public newsgroups, or network with
other developers on IRC. Above all else, write good clean code.
Cheers!
William Bull