qnx4 filesystem actual usage

Is there an easy way to take just the “necessary” part of a qnx4
filesystem, i.e., without the unused blocks?

For production, we’d like to have an “image” that can be copied to
another identical disk to produce a copy, without having to copy the
entire size of the disk.

Any thoughts on a good way to replicate a disk-based system in volume?

lew

Use the “tar” command, including compression if you wish. If the result is
still likely to be bigger than 2GB, pipe tar’s output into “split”.

dB

Lewis Donzis <lew@nospam.donzis.com> wrote in message
news:3E6E9597.D4AFA375@nospam.donzis.com

Is there an easy way to take just the “necessary” part of a qnx4
filesystem, i.e., without the unused blocks?

For production, we’d like to have an “image” that can be copied to
another identical disk to produce a copy, without having to copy the
entire size of the disk.

Any thoughts on a good way to replicate a disk-based system in volume?

lew

David Bacon wrote:

Use the “tar” command, including compression if you wish. If the result is
still likely to be bigger than 2GB, pipe tar’s output into “split”.

tar would be fine, but then how do you un-tar the file on to the new
disk in a production environment? It seems that we’d have to boot up
QNX, run fdisk/dinit, and then un-tar the file. The target system will
not have a CD or even a diskette to boot from.

We could use an external disk duplicator, but it would be nice if it
didn’t have to copy the entire 60GB in order to move about 15MB of
data…

lew

David Bacon wrote:
Use the “tar” command, including compression if you
wish. If the result is
still likely to be bigger than 2GB, pipe tar’s
output into “split”.

tar would be fine, but then how do you un-tar the
file on to the new
disk in a production environment? It seems that we’d
have to boot up
QNX, run fdisk/dinit, and then un-tar the file. The
target system will
not have a CD or even a diskette to boot from.

Does your system have network or USB boot ability. We solved this problem booting from the bootpd server. The only disadvantage of this approach that you need to know MAC address of the network adapter on the target. It would be more elegant to use USB if you have a right device.

If your board cannot do any of this I guess the only way is to take drive off the target and put it to the host system for software install, or connect a floppy to the target for a while.

We could use an external disk duplicator, but it
would be nice if it
didn’t have to copy the entire 60GB in order to move
about 15MB of
data…

lew

David Bacon wrote:
Use the “tar” command, including compression if you wish.
If the result is still likely to be bigger than 2GB, pipe tar’s
output into “split”.

tar would be fine, but then how do you un-tar the file on to the new
disk in a production environment? It seems that we’d have to boot up
QNX, run fdisk/dinit, and then un-tar the file. The target system will
not have a CD or even a diskette to boot from.

Does your system have network or USB boot ability. We solved this problem booting from the bootpd server. The only disadvantage of this approach that you need to know MAC address of the network adapter on the target. It would be more elegant to use USB if you have a right device.

If your board cannot do any of this I guess the only way is to take drive off the target and put it to the host system for software install, or connect a floppy to the target for a while.

We could use an external disk duplicator, but it would be nice if it
didn’t have to copy the entire 60GB in order to move about 15MB of
data…

lew

Lewis Donzis <lew@nospam.donzis.com> wrote:

David Bacon wrote:
Use the “tar” command, including compression if you wish. If the result is
still likely to be bigger than 2GB, pipe tar’s output into “split”.

tar would be fine, but then how do you un-tar the file on to the new
disk in a production environment? It seems that we’d have to boot up
QNX, run fdisk/dinit, and then un-tar the file. The target system will
not have a CD or even a diskette to boot from.

We could use an external disk duplicator, but it would be nice if it
didn’t have to copy the entire 60GB in order to move about 15MB of
data…

First, the disks had better actually be identical. (Exact same
h/t/s etc, otherwise a duplicated partition table could be bad
news.)

Do you need to have it all be one partition?

Create the “constant” part of your file system as one partition,
and mount it as /, maybe even mount it read-only on your target.

Create the “open” part of your file system as a filesystem on
a second partition, and don’t put anything on it.

Then, duplicate the entire constant partition and just the first
bit of the second. (I’m pretty sure that if you create a partition
then just do a “dinit” on it, its only going to write to a bunch
of blocks at the start of the partition, rather than modify blocks all
the way along. You could test it just to make sure.)

Or, you might create this 2nd partition dynamically, by creating a
boot script, that on first boot creates and dinits (and even copies
initial stuff over onto) the 2nd partition, then replaces this
“initializer” boot script with a “normal run” boot script.

-David

QNX Training Services
http://www.qnx.com/support/training/
Please followup in this newsgroup if you have further questions.

Lewis,

Do you have a network, and could you describe your “production environment”
a lttle more clearly? In particular, I don’t understand your “target
system”. Is is supposed to be able to boot from the newly replicated disk,
or is it the one that is supposed to be replicating disks? In the latter
case it needs to be able to boot into something, not necessarily QNX, just
something that supports tar.

dB

David Gibbs <dagibbs@qnx.com> wrote in message
news:b4nls0$fbd$4@nntp.qnx.com

Lewis Donzis <> lew@nospam.donzis.com> > wrote:
David Bacon wrote:
Use the “tar” command, including compression if you wish. If the
result is
still likely to be bigger than 2GB, pipe tar’s output into “split”.

tar would be fine, but then how do you un-tar the file on to the new
disk in a production environment? It seems that we’d have to boot up
QNX, run fdisk/dinit, and then un-tar the file. The target system will
not have a CD or even a diskette to boot from.

We could use an external disk duplicator, but it would be nice if it
didn’t have to copy the entire 60GB in order to move about 15MB of
data…

First, the disks had better actually be identical. (Exact same
h/t/s etc, otherwise a duplicated partition table could be bad
news.)

Do you need to have it all be one partition?

Create the “constant” part of your file system as one partition,
and mount it as /, maybe even mount it read-only on your target.

Create the “open” part of your file system as a filesystem on
a second partition, and don’t put anything on it.

Then, duplicate the entire constant partition and just the first
bit of the second. (I’m pretty sure that if you create a partition
then just do a “dinit” on it, its only going to write to a bunch
of blocks at the start of the partition, rather than modify blocks all
the way along. You could test it just to make sure.)

Or, you might create this 2nd partition dynamically, by creating a
boot script, that on first boot creates and dinits (and even copies
initial stuff over onto) the 2nd partition, then replaces this
“initializer” boot script with a “normal run” boot script.

-David

QNX Training Services
http://www.qnx.com/support/training/
Please followup in this newsgroup if you have further questions.

Create a master disk image by copying the original disk, file by file (use
cp -p or pax) to a newly partitioned and dinit’d filesystem on the master
drive. This will essentially “compact” the disk contents (i.e. eliminating
most blank areas that might have been interleaved with filesystem data on
your original). If you have any really large directories, you might want to
pregrow them on the target disk with mkdir -s size. This will reduce
fragmentation of the directory into multiple extents, which would affect the
runtime performance of your system. Once you’ve created the master, copy the
block special file for the whole master disk to a file. If you copy using
the dd command, you can elect to copy just the starting portion of the disk
that contains your filesystem data (you’ll have to figure out how far the
disk was populated on your own; you should be able to get a rough idea of
the offset relative to the start of the partition by using the ‘dd’ command
to see how much space is consumed in your master disk filesystem). In your
production setup, you would then just copy that file containing the disk
image to your the blank disk.


“Lewis Donzis” <lew@nospam.donzis.com> wrote in message
news:3E6E9597.D4AFA375@nospam.donzis.com

Is there an easy way to take just the “necessary” part of a qnx4
filesystem, i.e., without the unused blocks?

For production, we’d like to have an “image” that can be copied to
another identical disk to produce a copy, without having to copy the
entire size of the disk.

Any thoughts on a good way to replicate a disk-based system in volume?

lew

"> that contains your filesystem data (you’ll have to figure out how far the

disk was populated on your own; you should be able to get a rough idea of
the offset relative to the start of the partition by using the ‘dd’
command

typo - should read ‘df’ not ‘dd’

So why don’t you have the external disk duplicator do the fdisk, dinit and
un-tar the file to the new target drive. There are also a number of
removable hard drive carrier solutions out there. We use Kingston ones.

That’s basically how we’ve been duplicating our production systems for years
now. A typical configuration is to have a system drive with 2 partitions
on it, one for the OS image (200 M) and the remaining disk space/partition
mounted as /tmp. Then there is a second drive or RAID controller for all
our “permanent” data mounted as /u.

FYI… We’re currently in the process of converting over to a bootable CD
master build/install scheme, since most of the newer rackmount chassis come
with a CD drive. Doing a scripted fdisk, dinit & un-pax (in our case)
takes… oh… 20-30 seconds.

“Lewis Donzis” <lew@nospam.donzis.com> wrote in message
news:3E6F2B6A.C3A7E993@nospam.donzis.com

David Bacon wrote:
Use the “tar” command, including compression if you wish. If the result
is
still likely to be bigger than 2GB, pipe tar’s output into “split”.

tar would be fine, but then how do you un-tar the file on to the new
disk in a production environment? It seems that we’d have to boot up
QNX, run fdisk/dinit, and then un-tar the file. The target system will
not have a CD or even a diskette to boot from.

We could use an external disk duplicator, but it would be nice if it
didn’t have to copy the entire 60GB in order to move about 15MB of
data…

lew

Serge Yuschenko wrote:

Does your system have network or USB boot ability. We solved this problem booting from the bootpd server. The only disadvantage of this approach that you need to know MAC address of the network adapter on the target. It would be more elegant to use USB if you have a right device.

Actually, it’s a rack-mount PC and does have PXE boot capability. We
just haven’t had a lot of luck booting QNX through that mechanism.

USB is an interesting idea!

If your board cannot do any of this I guess the only way is to take drive off the target and put it to the host system for software install, or connect a floppy to the target for a while.

That would be another option to consider.

Thanks,

lew

David Gibbs wrote:

First, the disks had better actually be identical. (Exact same
h/t/s etc, otherwise a duplicated partition table could be bad
news.)

Yes, we had considered that. We can generally guarantee identical
disks, but it would, of course, be better not to have to worry about
that.

Do you need to have it all be one partition?

Not absolutely.

Create the “constant” part of your file system as one partition,
and mount it as /, maybe even mount it read-only on your target.

Create the “open” part of your file system as a filesystem on
a second partition, and don’t put anything on it.

Then, duplicate the entire constant partition and just the first
bit of the second. (I’m pretty sure that if you create a partition
then just do a “dinit” on it, its only going to write to a bunch
of blocks at the start of the partition, rather than modify blocks all
the way along. You could test it just to make sure.)

That’s true – but that’s a variant of the original question: how do you
know how many sectors have been used by the filesystem’s own tables and
such.

Or, you might create this 2nd partition dynamically, by creating a
boot script, that on first boot creates and dinits (and even copies
initial stuff over onto) the 2nd partition, then replaces this
“initializer” boot script with a “normal run” boot script.

That’s a good idea.

I was also intrigued with the idea of making a package filesystem, but
wasn’t sure how simple that is to do. That has some nice advantages of
allowing you to spill modified files.

Thanks for the suggestions.

lew

David Bacon wrote:

Do you have a network, and could you describe your “production environment”
a lttle more clearly? In particular, I don’t understand your “target
system”. Is is supposed to be able to boot from the newly replicated disk,
or is it the one that is supposed to be replicating disks? In the latter
case it needs to be able to boot into something, not necessarily QNX, just
something that supports tar.

Well, most anything is possible. The “target” in this case is the final
hardware, which should emerge from the production process with a
bootable hard disk (or CF or DiskOnMondule, but it’s all the same). The
target is a rack-mount PC, and is capable of a PXE boot, but we haven’t
had a whole lot of luck booting QNX via the PXE boot ROMs.

The production environment could be a separate machine to do the
duplication, or a commercial disk duplicator, or it could be the target
itself booting via network or something, and building its own disk.

If the target boots something other than QNX, it would seem a bit tricky
to initialize the disk with a QNX4 filesystem in order to then be able
to un-tar the files?

lew

Rob wrote:

So why don’t you have the external disk duplicator do the fdisk, dinit and
un-tar the file to the new target drive. There are also a number of
removable hard drive carrier solutions out there. We use Kingston ones.

That’s an excellent idea.

FYI… We’re currently in the process of converting over to a bootable CD
master build/install scheme, since most of the newer rackmount chassis come
with a CD drive. Doing a scripted fdisk, dinit & un-pax (in our case)
takes… oh… 20-30 seconds.

That would be great, except we have no CD in our systems. I suppose we
could hook one up just long enough to build the hard drive, but then
that begs the question… how do you make your own bootable QNX CD?

Thanks,

lew

Eric Johnson wrote:

Create a master disk image by copying the original disk, file by file (use
cp -p or pax) to a newly partitioned and dinit’d filesystem on the master
drive. This will essentially “compact” the disk contents (i.e. eliminating
most blank areas that might have been interleaved with filesystem data on
your original). If you have any really large directories, you might want to
pregrow them on the target disk with mkdir -s size. This will reduce
fragmentation of the directory into multiple extents, which would affect the
runtime performance of your system. Once you’ve created the master, copy the
block special file for the whole master disk to a file. If you copy using
the dd command, you can elect to copy just the starting portion of the disk
that contains your filesystem data (you’ll have to figure out how far the
disk was populated on your own; you should be able to get a rough idea of
the offset relative to the start of the partition by using the ‘df’ command
to see how much space is consumed in your master disk filesystem). In your
production setup, you would then just copy that file containing the disk
image to your the blank disk.

Yep, that’s sort of what we had in mind, hence the original question:
how do you know where the last allocated sector is? I hadn’t thought
about just using df (plus maybe a small pad). I wonder… is there an
easy way to read the allocation map and find the last used cluster?

Thanks,

lew

Under ksh, something like this:

echo $(( 8 * $(hd -Ad /.bitmap | tail -n3 |head -n1 | cut -d’ ’ -f1)))

should produce the number of non-zero blocks used by the filesystem (this is
assuming you’re running it under Neutrino). But, I’m totally winging it
here, so you might want to check against ‘df’ and make sure the number looks
right. This number should be more or less similar to that which pops up in
the third column of ‘df /.bitmap’.

“Lewis Donzis” <lew@nospam.donzis.com> wrote in message
news:3E7000AC.799142C7@nospam.donzis.com

Eric Johnson wrote:
Create a master disk image by copying the original disk, file by file
(use
cp -p or pax) to a newly partitioned and dinit’d filesystem on the
master
drive. This will essentially “compact” the disk contents (i.e.
eliminating
most blank areas that might have been interleaved with filesystem data
on
your original). If you have any really large directories, you might want
to
pregrow them on the target disk with mkdir -s size. This will reduce
fragmentation of the directory into multiple extents, which would affect
the
runtime performance of your system. Once you’ve created the master, copy
the
block special file for the whole master disk to a file. If you copy
using
the dd command, you can elect to copy just the starting portion of the
disk
that contains your filesystem data (you’ll have to figure out how far
the
disk was populated on your own; you should be able to get a rough idea
of
the offset relative to the start of the partition by using the ‘df’
command
to see how much space is consumed in your master disk filesystem). In
your
production setup, you would then just copy that file containing the disk
image to your the blank disk.

Yep, that’s sort of what we had in mind, hence the original question:
how do you know where the last allocated sector is? I hadn’t thought
about just using df (plus maybe a small pad). I wonder… is there an
easy way to read the allocation map and find the last used cluster?

Thanks,

lew

“Lewis Donzis” <lew@nospam.donzis.com> wrote in message
news:3E700026.EC805495@nospam.donzis.com

Rob wrote:
So why don’t you have the external disk duplicator do the fdisk, dinit
and
un-tar the file to the new target drive. There are also a number of
removable hard drive carrier solutions out there. We use Kingston ones.

That’s an excellent idea.

Thanks, we thought so too :wink:

FYI… We’re currently in the process of converting over to a bootable
CD
master build/install scheme, since most of the newer rackmount chassis
come
with a CD drive. Doing a scripted fdisk, dinit & un-pax (in our case)
takes… oh… 20-30 seconds.

That would be great, except we have no CD in our systems. I suppose we
could hook one up just long enough to build the hard drive, but then
that begs the question… how do you make your own bootable QNX CD?

Well, you’ll need mkisofsys for Qnx 4 to start with… and knowledge of how
to make a bootable floppy, and a CD burner on another non-qnx machine…
preferably with network connectivity to your Qnx 4 build machine (so you can
easily move your iso images around)… and plenty of time (& extra CDs) to
experiment with, to getting it all perfected.

There was a thread conserning how to do all this a few weeks back, in I
think the qnx4 news group? I participated in it and supplied more
information, but with all do respect, I really don’t have the time to
explain all the details and/or help out with all the little gotchas…
One of my compatiots & I did managed to successfully build a bootable Qnx 4
CD in only 3 trys… So rest assured that it is possible. But, he already
had months worth of experience building bootable CDs for Qnx 6 and I’ve got
years of experience building Qnx 4 boot whatevers on all sorts of different
hardware.

I will warn you that not all the documentation is 100% correct… So be
prepared for a siginificant amount of pain along the way to success.

Good luck :wink:

-Rob

Thanks,

lew

“Rob” <rob@spamyouself.com> wrote in message
news:b4qhpj$ko6$1@inn.qnx.com


I will warn you that not all the documentation is 100% correct… So be
prepared for a siginificant amount of pain along the way to success.

If you can recall what was incorrect, please let us know so we can fix the
problem.

  • Eric

Oops… Sorry… You’re doing this for Qnx 6 systems? Well, that’s
possible too. There’s a mkisofsys for Qnx 6… and it all basically works
the same way as for Qnx 4 or any other OS for that matter. You make a
bootable floppy and use a dd’ed image of it as the boot image for the CD.
As for the rest of it, in short, you just point mkisofsys at a directory
tree, the top level of which is to be the root dir on the CD… and it
bundles it all into a .iso file that can be burned onto a CD. It really
isn’t all that difficult. It’s just that if you do one little thing wrong,
it ain’t gonna work :wink:

You could also combine this with some of the other suggestions that have
been made. We for instance make up different *.pgz disk images for
different flavors of our production machines and put them all on the CD.
The CD itself is a full blown OS image with a ram disk for /tmp “work
space”. We can build boot images using it and/or ‘zcat | pax -w …’ a
specific *.pgz file onto a target production hard drive. Although I
haven’t personally tried it, it should be possible to dd a disk image to the
target drive.

-Rob

“Rob” <rob@spamyouself.com> wrote in message
news:b4qhpj$ko6$1@inn.qnx.com

“Lewis Donzis” <> lew@nospam.donzis.com> > wrote in message
news:> 3E700026.EC805495@nospam.donzis.com> …
Rob wrote:
So why don’t you have the external disk duplicator do the fdisk, dinit
and
un-tar the file to the new target drive. There are also a number of
removable hard drive carrier solutions out there. We use Kingston
ones.

That’s an excellent idea.

Thanks, we thought so too > :wink:


FYI… We’re currently in the process of converting over to a bootable
CD
master build/install scheme, since most of the newer rackmount chassis
come
with a CD drive. Doing a scripted fdisk, dinit & un-pax (in our case)
takes… oh… 20-30 seconds.

That would be great, except we have no CD in our systems. I suppose we
could hook one up just long enough to build the hard drive, but then
that begs the question… how do you make your own bootable QNX CD?


Well, you’ll need mkisofsys for Qnx 4 to start with… and knowledge of
how
to make a bootable floppy, and a CD burner on another non-qnx machine…
preferably with network connectivity to your Qnx 4 build machine (so you
can
easily move your iso images around)… and plenty of time (& extra CDs) to
experiment with, to getting it all perfected.

There was a thread conserning how to do all this a few weeks back, in I
think the qnx4 news group? I participated in it and supplied more
information, but with all do respect, I really don’t have the time to
explain all the details and/or help out with all the little gotchas…
One of my compatiots & I did managed to successfully build a bootable Qnx
4
CD in only 3 trys… So rest assured that it is possible. But, he already
had months worth of experience building bootable CDs for Qnx 6 and I’ve
got
years of experience building Qnx 4 boot whatevers on all sorts of
different
hardware.

I will warn you that not all the documentation is 100% correct… So be
prepared for a siginificant amount of pain along the way to success.

Good luck > :wink:

-Rob

Thanks,

lew

BTW, in my haste I was sloppy in my language. This will not show the number
of used blocks, but rather the offset just past the last used block on the
filesystem (i.e. it should give you the number of consecutive blocks you
need to copy to capture all the filesystem’s data, with respect to the
offset of the partition involved). On a freshly-copied filesystem such as we
had been discussing, this should be very very close to the number of used
blocks reported by ‘df’. Caveat emptor - I haven’t tested this.

“Eric Johnson” <eric@qnx.com> wrote in message
news:b4qd95$cml$1@nntp.qnx.com

Under ksh, something like this:

echo $(( 8 * $(hd -Ad /.bitmap | tail -n3 |head -n1 | cut -d’ ’ -f1)))

should produce the number of non-zero blocks used by the filesystem (this
is
assuming you’re running it under Neutrino). But, I’m totally winging it
here, so you might want to check against ‘df’ and make sure the number
looks
right. This number should be more or less similar to that which pops up in
the third column of ‘df /.bitmap’.

“Lewis Donzis” <> lew@nospam.donzis.com> > wrote in message
news:> 3E7000AC.799142C7@nospam.donzis.com> …
Eric Johnson wrote:
Create a master disk image by copying the original disk, file by file
(use
cp -p or pax) to a newly partitioned and dinit’d filesystem on the
master
drive. This will essentially “compact” the disk contents (i.e.
eliminating
most blank areas that might have been interleaved with filesystem data
on
your original). If you have any really large directories, you might
want
to
pregrow them on the target disk with mkdir -s size. This will reduce
fragmentation of the directory into multiple extents, which would
affect
the
runtime performance of your system. Once you’ve created the master,
copy
the
block special file for the whole master disk to a file. If you copy
using
the dd command, you can elect to copy just the starting portion of the
disk
that contains your filesystem data (you’ll have to figure out how far
the
disk was populated on your own; you should be able to get a rough idea
of
the offset relative to the start of the partition by using the ‘df’
command
to see how much space is consumed in your master disk filesystem). In
your
production setup, you would then just copy that file containing the
disk
image to your the blank disk.

Yep, that’s sort of what we had in mind, hence the original question:
how do you know where the last allocated sector is? I hadn’t thought
about just using df (plus maybe a small pad). I wonder… is there an
easy way to read the allocation map and find the last used cluster?

Thanks,

lew