An Introduction to QNX. Packages and Repositorie
By William Bull, QNX Software Systems Ltd.
Good software offers a satisfying user experience - but to deliver that
experience, the software must be easy to install and maintain. To help
developers achieve this goal, we’ve extended the QNX Package Filesystem
and implemented archive repositories, places for distributing packages.
We’ve also implemented utilities to manage and discover packages as well
as to build packages from a set of files. Together, these tools allow
software to be shared more easily, to install without errors, and to
uninstall without a hitch. As you’ll see, all the tools need is a little
information…
The Information Infrastructure
Using the Package Filesystem, you can “union” directories and overlay
files. You can also perform atomic installations, which allow a
collection of files and directories to appear or disappear
simultaneously. All these capabilities paved the way for tools that can
automatically install and uninstall software. But with these tools came
a demand for structure and organization.
For example, to describe a package we need to know:
- what files and directories will be installed on the user’s machine
- where the files should appear, once installed
But to describe software we need to know: - what software this is - its name, version, release date
- who wrote it - the author, the maintainer, their web page
- whether the software needs other software
- whether the software is incompatible with other software
- whether the software forms part of a greater collection of other
software - whether there are scripts or actions that must occur during installation
or uninstallation
Once we collect all this package data, we immediately face two
challenges: how to store it and where to store it.
Naturally, we wanted everyone to use - and reuse - this package data in
the easiest, most effective way possible. That’s why we’ve decided to
store the data in a text file and encode it using a simple XML notation.
The regular (i.e. parseable) format of XML, along with the availability
of tools like the expat library, means developers can manipulate the
data easily; no need for special tools.
To determine where this data should be stored, we first had to determine
how people would want to use it. After much discussion, we worked from
the premise that most software is now shared, traded, or downloaded from
web sites and FTP sites. Consequently, our tools to manipulate packages
would be limited by bandwidth - no one would want to download a complete
package just to retrieve a version number or home page of an author!
Thus, each QNX package may have a separate, external data description
file known as a manifest.
If you place a collection of packages and their external manifests on an
FTP site, you have what we consider an archive repository, a place where
a package administration tool can find and install software.
A Simple Repository
Repositories: Terminology and Features
The term “repository” has different uses, depending on whether you’re
distributing files or installing files. For example, on IRC you might
ask, “Has anyone put up any new repositories?” You would then expect a
response like, "Yes, you can grab my latest ports from
http://207.198.90.123/repository/." Conversely, a tech support
representative may ask you, “Is the package you installed in your local
repository?” In which case, your response might be, “No, I looked on all
of my drives, I don’t think the package was installed correctly.”
Installation repositories
From the perspective of the Package Filesystem, a repository is simply
where it finds the files you’ve installed. On a standard QNX realtime
platform distribution, you have two repositories located in /pkgs:
o /pkgs/base - a read-only repository containing the files that are part
of your original system distribution
o /pkgs/repository - the repository where your packages are installed
To view these repositories, simply click “View My Software” from within
the QNX Package Manager.
If you run out of disk space, you can always mount an extra drive in
/pkgs and tell the QNX Package Installer to install your software in the
new repository. In fact, the mount point of /pkgs is merely a
convenience - an installation repository can exist anywhere on your
disk, even on a floppy. Of course, you need to inform the Package
Filesystem about the new repository and what packages you want
activated. To do this, you can use the QNX Package Installer.
An installation repository has a structure created to avoid software
conflicts. When you build a package, you’ll see a few questions asking
for identifiers and version information. This information forms the
directory structure that appears in the installation repository.
Remember that you don’t have to understand how those directories are
created; we provide package building tools that take care of this
process for you.
Archive repositories
From the Package Installer’s perspective, any directory, web site, or
FTP site that has a package and the package’s manifest is a legal
archive repository. Unfortunately, due to the security restrictions of
some servers, the Package Installer may be unable to read the contents
of a web site or an FTP site. So, before it does anything else, the
Package Installer looks for a repository description file,
repository.qrm, which is also known as the repository manifest. See the
example manifest below:
<RDF:RDF xmlns:RDF=“http://www.w3.org/TR/PR-rdf-syntax”
xmlns:QPM=“http://www.qnx.com”>
<RDF:Description about=“http://www.qnx.com”>
QPM:RepositoryManifest
QPM:RepositoryTypeArchive Repository
</QPM:RepositoryType>
QPM:RepositoryContent
QPM:PackageCount3</QPM:PackageCount>
QPM:LastUpdate20010122</QPM:LastUpdate>
</QPM:RepositoryContent>
QPM:RepositoryDescription
QPM:RepositoryNameWilliam Bull’s
Repository</QPM:RepositoryName>
QPM:RepositoryURLhttp://207.198.90.123/repository</QPM:RepositoryURL>
QPM:RepositoryHomeURLhttp://207.198.90.123/</QPM:RepositoryHomeURL>
QPM:RepositoryEmailgristle@home.com</QPM:RepositoryEmail>
QPM:RepositorySmallIconhttp://207.198.90.123/topics/graphics.gif</QPM:
RepositorySmallIcon>
QPM:RepositoryLargeIconhttp://207.198.90.123/topics/graphics.gif</QPM:
RepositoryLargeIcon>
QPM:PackageListindex</QPM:PackageList>
QPM:SummaryWilliam Bull’s Home
Repository</QPM:Summary>
QPM:SummaryUrlrep://index.html</QPM:SummaryUrl>
QPM:DescriptionThis repository contains
packages which may not be stable. Apologies in
advance.</QPM:Description>
</QPM:RepositoryDescription>
</QPM:RepositoryManifest>
</RDF:Description>
</RDF:RDF>
A Sample Repository Manifest File
(repository.qrm)
With this manifest file in hand, the Package Installer can display
information about the repository and check for updates automatically by
reading the LastUpdate and PackageCount tags.
As mentioned earlier, some FTP sites and web sites won’t allow the
Package Installer to directly check which packages are in the
repository. So, to make maintenance easier, we use a simple text file
that contains the public contents of the repository; this file is
defined by the PackageList tag. To generate this index file, simply type
ls > index.
All the other description and content tags in the repository.qrm file
are optional. To maintain your repository, all you should have to do is
FTP your packages and recreate the index file.
If you look inside the .qrm file, you may notice the use of the rep://
prefix. One of our original design goals was to promote mirroring and
to simplify the distribution of packages and repositories. We wanted to
avoid the requirement for hard-coded URLs, so we created repository
reference URLs. The rep:// prefix simply tells the Package Installer to
use the repository directory as the relative base position for the rest
of this URL. For example, the prefix rep://foo would tell the installer
to look for a directly called foo in the base of the repository. In the
above sample, SummaryURL is located at
http://207.198.90.123/repository/index.html.
A Complete Repository
One-click archive repositories
Having a repository is great for sharing a lot of files that you want
people to browse through. But if you’re running a company or fan web
site, you may want people to install a package by simply clicking on a
hyperlink. We support this with something called the package repository
file (.qpr). This file is just an entire repository tar’d and gzip’d
with a new unique extension. Given that a repository can be as simple as
the .qpm and .qpk file for a package, creating a one-click distribution
is easy!
For instance, if someone using the QNX realtime platform clicks on a
hyperlink ending in the extension .qpr, the web browser will launch the
Package Installer with a command-line option to immediately install the
files.
Packages: Terminology and Features
You may have experienced the benefits of the Red Hat’s packaging format
(RPM) or have read about OS X’s Apple/NeXT-style bundles. QNX packages
offer many of the benefits of these packaging formats and provide
additional features to meet our own enhanced requirements.
Cross-platform issues: host and target
The QNX environment can serve as a cross-platform development
environment as well as a cross-platform server. For instance, you can
work on an x86 computer and generate code for a PowerPC machine. Or you
can install software on an x86 computer that was compiled for a PowerPC
computer. Furthermore, a PowerPC computer can boot and access its
software from the x86 file server - seamlessly.
To support these features, we needed to make a series of distinctions
about software. For instance, it may make perfect sense to separate
PowerPC binaries from x86 binaries, but what about a PowerPC compiler
used on an x86 machine and the support files for compiling PowerPC code?
A situation like this is a bit trickier; we would say that the compiler
is hosted on x86 but that it targets PowerPC.
Here’s a summary of terminology we use when talking about cross-platform
packages:
o Common packages - contain files that have no bearing on the processor.
For example, images and text files.
o Core host packages - contain binaries and files specific to a processor.
For example, utilities like emacs or cat.
o Common target packages - contain files related to building code for a
specific processor, but that can be used on any processor. For example,
header files or documentation specific to targeted development tools.
o Core target packages - Contain binaries and files used to generate code
for a specific processor and that are usable only on that processor.
Using these categories, we can, for example, host emacs for PowerPC,
x86, and MIPS on the same server, without collision or maintenance
hassles. We can also save space by minimizing the need to download
common files that are shared across packages.
Package descriptions: name, version, topic, and (much) more…
One goal for our packaging format was to see the creation of a
long-lasting technology that enables the QNX development community. By
placing a self-organizing, well-described packaging technology into the
hands of the public, we could allow software to be traded, upgraded, and
organized without a “mother company.”
Fully described packages are critical to the long-term success of a
computer platform. So, as you create your first package, you’ll find
that our packaging tools ask a rather lengthy set of questions. These
questions let you fully describe your software and the release you’re
about to distribute. You can then use this information to let users
automatically receive software updates, track the progress of developing
software, and use enhanced tools that make software distribution and
management easier.
By creating a robust, well-documented description set for packages, you
can organize huge amounts of software easily. For example, since a
package is associated with a software topic, tools like the QNX Package
Installer can visually categorize the software for easy access. We’ve
provided several hundred predefined categories, each with its own icon
and description. Users will see these categories as soon as your package
is placed into a repository. For example:
Sample Topic Listing
Relationships: components, dependencies, and bundles
Your software may require the installation of other software: libraries,
utilities, or other packages. Or it may be an optional component of a
larger software project. For situations like these, QNX packages support
several relationship mechanisms. Here are a few of the simplest
relationships supported:
-
Components - A package may be the child of another package. For example,
you may choose to package up the Big Brother Database elisp files for
emacs 19. After the emacs package is published and widely available,
you can distribute your set of elisp files as an optional component.
Your package will be associated with emacs and appear beneath it within
the topic tree:
Big Brother Database as an emacs component, within the emacs topic. -
Dependencies - Packages may require or recommend that another package be
included or excluded. For example, let’s say you find an incompatibility
with a particular package. In that case, you may require that the
offending package be removed, and recommend that it be replaced with one
of three alternate packages. -
Bundles - Collections of unrelated packages, with different versions,
can be automatically installed as a bundle. If the user uninstalls the
bundle, all the associated packages will be automatically uninstalled.
(This is less of a dependency and more a convenience for users.) -
Overlay order - The Package Filesystem allows one package to be
“unioned” over another. As a result, a file in one package can
potentially collide with a file in another package. This kind of
conflict rarely happens, but if it does, you can define an overlay order
to specify whether your package will be installed over or under the
other package. That way, you can ensure that the user will see your
files.
These are but a few of the relationship mechanisms provided by the
Package Manager. By using our package building tools, you can access
rules and qualifications that give you finer control over your product’s
installation. (We’ll discuss these and other packaging features in
future articles.)
Scripts
Once a user has installed your software, you’ll probably want to display
a splash screen or a readme file, or execute some other script to finish
the installation. You may also want to run a script during
uninstallation. The Package Manager handles both these scenarios, but
that’s not the end of the story.
The QNX Package Manager understands when you install, activate,
deactivate, or uninstall a package. You can define a rule to execute a
script whenever one of these events occurs. With this, you can be
notified whenever a package is installed or if a specific package is
uninstalled. For example, if you have an application that caches fonts,
you could conceivably filter all packages being installed or uninstalled
that contain fonts. The Package Manager can do all this because of all
the information that it gathers about your software and places in a
manifest.
Remember that packages may be built for one processor and installed on
another! For example, a PowerPC file server may install your package,
but not have the ability to run the binaries within your package. So
when you build packages, avoid running commands that aren’t shipped with
a base QNX realtime platform installation. (Future articles will provide
an in-depth look on package scripting.)
Summary of File Extensions
Here’s a summary of the file extensions you may encounter when using the
Package Installer and package building tools:
…qpkQNX PackageA .tgz containing your files to be installed.
…qpmQNX Package ManifestAn XML file describing the content of the .qpk
…qprQNX package repositoryA .tgz containing any number of packages and manifests.
…qrmQNX repository manifestAn XML file describing the content of a repository.
More to Come…
We plan to post more articles on QNX package building tools - including
some useful examples - in the near future. Until then, take the time to
explore and experiment. If you have questions, recommendations, or
comments, feel free to post them on the public newsgroups or network
with other developers on IRC.