ARTICLE: The Package Filesystem

QNX Newsgroups (Archives) qdn.public.articles
1

The Package Filesystem
By Thomas (toe-mah) Fletcher, QNX Software Systems Ltd.

In a previous article in this series, I talked about how the
Neutrino filesystem works to resolve names. (See: “Pathname
resolution… with a little on unioning mountpoints”.)

This article describes the package filesystem, which is an
abstraction layer used in the QNX Realtime Platform (QNX RTP).
The package filesystem (fs-pkg in your QNX RTP pidin listing) is
a re-direction layer that sits on top of any other filesystems. It is
composed of three main parts:

\

  1. The packages in their repositories
  2. The spill directory tree
  3. The underlying filesystem(s)



    Terminology

Just to make sure we’re all speaking the same language
here, let’s describe a few terms:

A package is a directory tree of files laid out in a structure
that matches their desired layout if they were to be transposed
to the root of some filesystem.

Packages come in a couple of flavors:

  • installed packages, which are files that have been fully laid
    out in the filesystem (such as those found in
    /pkgs/repository/qnx/os/).
  • download packages, which are files with the extensions of
    “qpr” (tar.gz of mini-repositories suitable for one-click
    downloads) or “qpk” (tar.gz of the installed packages).


    NOTE: A future article will discuss in depth the process of
    turning software components into these packages.

The locations where the packages are installed are called
repositories. These reside on other standard filesystems that
are accessible to the package filesystem. On a typical QNX
RTP system, the main package repositories are /pkgs/base
and /pkgs/repository.

How it works

Now that we know some basic terms, let’s examine in detail
what the package filesystem does. The package filesystem
creates a “virtual” filesystem that re-directs access requests to
a “real” filesystem based on a set of user-selected packages.
The contents of the various packages determine what the virtual
filesystem looks like. An example may help here:

[see image on http://qdn.qnx.com/articles/jan1101/fs-pkg.html]

Core Package (1 platform, 1 general):
/pkgs/base/qnx/core1.0
/pkgs/base/qnx/core1.0/etc/passwd
/pkgs/base/qnx/core1.0/bin/true
/pkgs/base/qnx/core1.0/x86/bin/cat
/pkgs/base/qnx/core1.0/x86/bin/echo

Util package (1 platform):
/pkgs/base/qnx/util1.0/x86/bin/ls

In this example, there are two packages with a variety of
different types of files. Two of the files, passwd and true
(presumably a shell script), are platform-independent. There
are also several files (cat, echo, and ls), which are available
only for the x86 platform.

The lines in the diagram from the virtual filesystem to the real
filesystem indicate the re-direction that is performed.

When a request comes in to the virtual filesystem, a lookup is
performed. The lookup determines if the file is indeed managed
by the package filesystem. If it is, then the system will re-direct
the request to the real location of the file inside the appropriate
package. The real location could be on any filesystem (QNX 4, Linux,
DOS), across the network (via NFS, CIFS), or even cached locally
(a shadow filesystem). The package filesystem doesn’t care at all.

The spill directory

This re-direction to the file’s location in the package continues
as long as the access is read-only. But as soon as any attempt
is made to write to a file that the package filesystem is managing,
then the spill directory comes into play.

Packages are considered immutable, read-only, objects. The intent
here is to ensure that packages can install such that there’s never
a collision between files in the repositories (which is why we use
the convention of vendor/product[/subproduct] with version numbers).
This allows multiple versions of software to co-exist on any one
machine.

[see image on http://qdn.qnx.com/articles/jan1101/fs-pkg.html]

Let’s look at an example of how a file is transfered from the
package repository to the spill directory, an operation known as
spilling.

After installation, the user generally wants to change the default
password. When the password file is opened for writing, it’s copied
to the spill directory, which contains all local modifications that
have been made to used packages (spilled files). From this point
on,the package filesystem will forward all requests to the spill
location rather than to the original file location in the package. If
no spill directory location is specified, then the package
filesystem operates as if it were a read-only filesystem.

The underlying filesystem

The third component of the package filesystem comes from the fact
that the filesystem presented is entirely virtual, composed of
scatter-gather components from various packages.

If the user wants to copy a file to one of the virtual directories
(e.g. /bin), then the package filesystem creates the appropriate
structure on the underlying filesystem and then allows the original
access request to simply fall through to the next server.

If there’s no underlying filesystem, then new files can’t be added
to the virtual filesystem without first generating a package for
them.

Control files

The package filesystem uses two control files:

  • MANIFEST file
  • packages config file


    The MANIFEST files are used by developers who want to put together
    a package for distribution. The purpose of these files, which live at
    the root of every installed package, is to describe which files are
    exported, the services the package provides, the dependencies of
    the package, etc. The format and content of this file will be
    covered in a future article describing how to format and build
    your own packages.

The second control file is the file that’s used to actually
configure the package filesystem itself. The default location for
this file is /etc/system/package/packages.

Like the MANIFEST files, this file is formated in XML. The tags used
are very simple and, hopefully, self-documenting, but we’ll briefly
go over them here:

[see image on http://qdn.qnx.com/articles/jan1101/fs-pkg.html]

<QPF:mapping vfsroot="/" spillroot="/var/pkg/spill">



This tag describes the mapping of the package filesystem.

vfsroot:
Where the package filesystem will attach to in order to create
the root of the virtual filesystem.
spillroot:
Where the directory for the spill files should be located. If
spillroot is omitted, then this is a read-only filesystem.


QPF:repositories
<QPF:item name=“Core Repository” type=“Read-only”>/pkgs/base
</QPF:item>

<QPF:item name=“User Repository”>/pkgs/repository</QPF:item>
</QPF:repositories>



This tag describes the location of the repositories available to the
package filesystem. The package filesystem ignores the name and
type fields, which are used by the package installer. These
repositories are searched in the order listed (top is first, bottom
is last) for the packages.

QPF:packages
QPF:itemqnx/os/core2.1</QPF:item>
QPF:ignoreqnx/os/core2.1/x86</QPF:ignore>
…
</QPF:packages>



This last tag describes the actual packages the package filesystem
should use for the virtual filesystem. Only the base of the packages
needs to be included; the processor components will be picked up
automatically if the package is well-formed.

The package installer uses the QPF:ignore tag for bookkeeping

  • it has no relevance to the package filesystem.

The packages are added in the order of top (first) to bottom (last).
Packages are located by combining the repository stem with the
package stem. So, to locate the above package we would search
for the first instance of the following:

/pkgs/base/qnx/os/core2.1/MANIFEST
/pkgs/repository/qnx/os/core2.1/MANIFEST



Files in the packages occupy slots in the virtual filesystem on a
first-come, first-served basis.

Conclusion

The package filesystem acts as a thin veneer that re-directs user access
to files to their real location. What we haven’t discussed in detail is
that the meaning of “real file” can be re-mapped during runtime, based
on the client attributes.

For example, imagine a PPC client attempting to access an x86 host
over Qnet. The PPC client would want to see only those executables
that it would be able to run when looking at standard file locations.
Running /net/x86host/bin/ls would work, because the request would
be re-mapped to a PPC version of the ls executable. If the x86 file
needed to be accessed (e.g. making an x86 buildfile from a PPC host),
it could still be accessed using /net/x86host/x86/bin/ls. This type of
mapping provides for some very powerful possibilities.

For more information on the package filesystem and its operation,
see the following sources: