tape and tar troubles

QNX Newsgroups (Archives) comp.os.qnx
1

I’m having some trouble using the tape and tar commands.

On a daily basis, a script essentially executes a
cd $JOB_ARC
tape eod
tar cvbf 20 /dev/tp0 $files

to archive a months worth of separate daily tar files on one tape, each
morning at 2am by cron job, where $JOB_ARC is an environment variable
that contains the path to an archive directory and $files is a script
variable that contains a list of files to backup.

If you use tar tvf /dev/tp0 to see what is on the tape, you will see the
first tar file, the first time you execute the tar tvf command after a
rewind. But the second time you execute the command (without
rewinding), you will either see an empty tar file (blocksize: 20
message, but no files) or an error tar file (blocksize:20 message, “no
input” message), or possible multiple error or empty files, before you
see a good one again. The difference in tape position between a valid
tar file, and an empty one is usually 2 tape blocks. The difference in
tape position between a valid tar file and an error tar file is usually
0 blocks, although I have seen a case where there was difference of 1
block. When the error tar file comes out, you can echo $? and see a

  1. When a valid tar file or an empty tar file is encountered, $? is 0.

This creates a problem trying to loop and list the contents of all tar
files on the tape, with a varying number of files on the tape, because
of the varying number of days in a month, and a varying gap of bad files
or empty files in between good files.

Currently we use something simple like this:

tape rewind
tar tvf /dev/tp0 # list the contents of the first tar file on the tape

while $? -eq 0
do
tar tvf /dev/tp0 >2 /dev/null # skip the empty tar file
tar tvf /dev/tp0 # list the next good tar
file
done

Problem is, this loop ends when you get a “no input” tar file because $?
returns a 1, and, if there is more than one bad tar file in between,
then it doesn’t work either.

Why are the extra tar files there? I only do one tar cvf command every
night at 2am by cron. Could the tape eod command be adding the extra
gaps? Is this a bug?

The tape forward command is similarly bothered by these empty or error
tar files in between good tar files although its behavior can be
different from doing consecutive tar tvf commands. The tape backup
command doesn’t appear to work at all.

Any suggestions?

Scott

2

Update: I did some test and didn’t use the tape eod comamnd and still got
the problem. It appears the problem is in the pax binary (which process the
tar command). I wrote a simple 1 block file 3 times in a row. then tape
rewind. then tar tvf. There was an error tar file in between each of the
good files. is there any way to write multiple tar files on the tape
without getting error or empty tar files in between?

Scott

“J. Scott Franko” wrote:

I’m having some trouble using the tape and tar commands.

On a daily basis, a script essentially executes a
cd $JOB_ARC
tape eod
tar cvbf 20 /dev/tp0 $files

to archive a months worth of separate daily tar files on one tape, each
morning at 2am by cron job, where $JOB_ARC is an environment variable
that contains the path to an archive directory and $files is a script
variable that contains a list of files to backup.

If you use tar tvf /dev/tp0 to see what is on the tape, you will see the
first tar file, the first time you execute the tar tvf command after a
rewind. But the second time you execute the command (without
rewinding), you will either see an empty tar file (blocksize: 20
message, but no files) or an error tar file (blocksize:20 message, “no
input” message), or possible multiple error or empty files, before you
see a good one again. The difference in tape position between a valid
tar file, and an empty one is usually 2 tape blocks. The difference in
tape position between a valid tar file and an error tar file is usually
0 blocks, although I have seen a case where there was difference of 1
block. When the error tar file comes out, you can echo $? and see a

  1. When a valid tar file or an empty tar file is encountered, $? is 0.

This creates a problem trying to loop and list the contents of all tar
files on the tape, with a varying number of files on the tape, because
of the varying number of days in a month, and a varying gap of bad files
or empty files in between good files.

Currently we use something simple like this:

tape rewind
tar tvf /dev/tp0 # list the contents of the first tar file on the tape

while $? -eq 0
do
tar tvf /dev/tp0 >2 /dev/null # skip the empty tar file
tar tvf /dev/tp0 # list the next good tar
file
done

Problem is, this loop ends when you get a “no input” tar file because $?
returns a 1, and, if there is more than one bad tar file in between,
then it doesn’t work either.

Why are the extra tar files there? I only do one tar cvf command every
night at 2am by cron. Could the tape eod command be adding the extra
gaps? Is this a bug?

The tape forward command is similarly bothered by these empty or error
tar files in between good tar files although its behavior can be
different from doing consecutive tar tvf commands. The tape backup
command doesn’t appear to work at all.

Any suggestions?

Scott

3

Careful with tapes - they sometimes spin past the point you would expect.
When you read or write an archive the tape is not guaranteed to stop exactly
at the end of the archive - there might be some dead space (0 to N blocks).
Some newer drives might do this differently - no guarantee.
The only way to guarantee that you are at the eod is to rewind then do an
eod.
If you tar (and it spins a bit past the end) followed by an eod, you might
actually be locking on to the eod of a partially overwritten previous
archive - and yes you’ll thus have several (zero to size of overwritten old
archive) blocks of junk between your true archive and the supposed eod.

Possibilities: When you archive record the last block used by the archive,
or the number of blocks used. Then use these values to direct an exact tape
position command.
You could even do a tape position back a few blocks (less than size of
archive) and then do an eod.

If you find a repetitive solution that works cleanly please email it to me.
I need to do this for an upcoming project.

Non-Tech:
When using a Tape keep the few buttons on a VCR in mind. Also treat it as if
each command is given to a new person with no knowledge of what previous
people have done with the tape - they’ll default to always running the tape
forward unless you tell them otherwise.

Tech:
Remember that the Tape drive is a sequential device with little memory. It
really only knows the block number it is pointing at. It doesn’t know how
you’ve arranged your data on tape and will happily spin along looking for
then next search item (eod…). Note that an eod command will always spin
the tape further along (never backwards) unless you are exactly at an eod -
and even that I wouldn’t guarantee without some testing or reading. Only a
random access device (RAM, Disk drive, CD, …) can jump to the true eod
whether it be forward or back - tapes are very old unintelligent technology,
even when their medium - DAT - is new.

-Paul
paul@jenosys.com





J. Scott Franko <jsfranko@switch.com> wrote in message
news:3987255A.7B954F90@switch.com…

Update: I did some test and didn’t use the tape eod comamnd and still got
the problem. It appears the problem is in the pax binary (which process
the
tar command). I wrote a simple 1 block file 3 times in a row. then tape
rewind. then tar tvf. There was an error tar file in between each of the
good files. is there any way to write multiple tar files on the tape
without getting error or empty tar files in between?

Scott

“J. Scott Franko” wrote:

I’m having some trouble using the tape and tar commands.

On a daily basis, a script essentially executes a
cd $JOB_ARC
tape eod
tar cvbf 20 /dev/tp0 $files

to archive a months worth of separate daily tar files on one tape, each
morning at 2am by cron job, where $JOB_ARC is an environment variable
that contains the path to an archive directory and $files is a script
variable that contains a list of files to backup.

If you use tar tvf /dev/tp0 to see what is on the tape, you will see the
first tar file, the first time you execute the tar tvf command after a
rewind. But the second time you execute the command (without
rewinding), you will either see an empty tar file (blocksize: 20
message, but no files) or an error tar file (blocksize:20 message, “no
input” message), or possible multiple error or empty files, before you
see a good one again. The difference in tape position between a valid
tar file, and an empty one is usually 2 tape blocks. The difference in
tape position between a valid tar file and an error tar file is usually
0 blocks, although I have seen a case where there was difference of 1
block. When the error tar file comes out, you can echo $? and see a

  1. When a valid tar file or an empty tar file is encountered, $? is 0.

This creates a problem trying to loop and list the contents of all tar
files on the tape, with a varying number of files on the tape, because
of the varying number of days in a month, and a varying gap of bad files
or empty files in between good files.

Currently we use something simple like this:

tape rewind
tar tvf /dev/tp0 # list the contents of the first tar file on the tape

while $? -eq 0
do
tar tvf /dev/tp0 >2 /dev/null # skip the empty tar file
tar tvf /dev/tp0 # list the next good tar
file
done

Problem is, this loop ends when you get a “no input” tar file because $?
returns a 1, and, if there is more than one bad tar file in between,
then it doesn’t work either.

Why are the extra tar files there? I only do one tar cvf command every
night at 2am by cron. Could the tape eod command be adding the extra
gaps? Is this a bug?

The tape forward command is similarly bothered by these empty or error
tar files in between good tar files although its behavior can be
different from doing consecutive tar tvf commands. The tape backup
command doesn’t appear to work at all.

Any suggestions?

Scott

4

Problem is, I don’t recall running into this problem on any other *nix system.
As far as rewinding and eod’ing every time, it would take too long. And there
is not tape argument to position the tape at a specific tape block number. So I
can’t rewind just a bit and go to eod.

I’ve been doing some experimenting, and have come down to a design that should
work in most cases. I tar and output the tar’s printf’s to a file. If I get a
$? of 0, I cat the file to the screen, and zero out the error counter, thereby
listing the contents of the tar file on the tape, but if I get a $? of 1, I
increment an error counter and don’t cat the file to the screen. I loop around
that, ending on an error count of greater than 1, which I only get at the end of
the tape with consecutive “no input” results from tar. The only way this scheme
would fail, would be if I get more than one “no input” in between two good
files. So far I haven’t. When I do get two bad tars between two goods, its
always one empty tar, and one “no input”. Most of the time I get one bad tar
between two goods, and it is a “no input” tar. On the occasion that get an
empty tar between to goods, it should just cat an empty file.

Now I have a new problem. The tar command that works at the command line,
doesn’t work in the script.

tar tvf /dev/tp0 2>tarout

puts both the blocksize message, and the “no input” message, as well as the
listing of the files in the tar archive, into the file tarout, from the command
line.

But when I execute the same command from the script, it either cuts off the
2>tarout, or if I put “” in front of the 2 and >, it puts the blocksize and
“noinput” to the screen and nothing to the tarout file. I’ve tried:

tar 1>tarout 2>>tarout, but that doesn’t work either. I can’t seem to get the
combination that works in the script.

Scott

Paul Russell wrote:

Careful with tapes - they sometimes spin past the point you would expect.
When you read or write an archive the tape is not guaranteed to stop exactly
at the end of the archive - there might be some dead space (0 to N blocks).
Some newer drives might do this differently - no guarantee.
The only way to guarantee that you are at the eod is to rewind then do an
eod.
If you tar (and it spins a bit past the end) followed by an eod, you might
actually be locking on to the eod of a partially overwritten previous
archive - and yes you’ll thus have several (zero to size of overwritten old
archive) blocks of junk between your true archive and the supposed eod.

Possibilities: When you archive record the last block used by the archive,
or the number of blocks used. Then use these values to direct an exact tape
position command.
You could even do a tape position back a few blocks (less than size of
archive) and then do an eod.

If you find a repetitive solution that works cleanly please email it to me.
I need to do this for an upcoming project.

Non-Tech:
When using a Tape keep the few buttons on a VCR in mind. Also treat it as if
each command is given to a new person with no knowledge of what previous
people have done with the tape - they’ll default to always running the tape
forward unless you tell them otherwise.

Tech:
Remember that the Tape drive is a sequential device with little memory. It
really only knows the block number it is pointing at. It doesn’t know how
you’ve arranged your data on tape and will happily spin along looking for
then next search item (eod…). Note that an eod command will always spin
the tape further along (never backwards) unless you are exactly at an eod -
and even that I wouldn’t guarantee without some testing or reading. Only a
random access device (RAM, Disk drive, CD, …) can jump to the true eod
whether it be forward or back - tapes are very old unintelligent technology,
even when their medium - DAT - is new.

-Paul
paul@jenosys.com

J. Scott Franko <> jsfranko@switch.com> > wrote in message
news:> 3987255A.7B954F90@switch.com> …
Update: I did some test and didn’t use the tape eod comamnd and still got
the problem. It appears the problem is in the pax binary (which process
the
tar command). I wrote a simple 1 block file 3 times in a row. then tape
rewind. then tar tvf. There was an error tar file in between each of the
good files. is there any way to write multiple tar files on the tape
without getting error or empty tar files in between?

Scott

“J. Scott Franko” wrote:

I’m having some trouble using the tape and tar commands.

On a daily basis, a script essentially executes a
cd $JOB_ARC
tape eod
tar cvbf 20 /dev/tp0 $files

to archive a months worth of separate daily tar files on one tape, each
morning at 2am by cron job, where $JOB_ARC is an environment variable
that contains the path to an archive directory and $files is a script
variable that contains a list of files to backup.

If you use tar tvf /dev/tp0 to see what is on the tape, you will see the
first tar file, the first time you execute the tar tvf command after a
rewind. But the second time you execute the command (without
rewinding), you will either see an empty tar file (blocksize: 20
message, but no files) or an error tar file (blocksize:20 message, “no
input” message), or possible multiple error or empty files, before you
see a good one again. The difference in tape position between a valid
tar file, and an empty one is usually 2 tape blocks. The difference in
tape position between a valid tar file and an error tar file is usually
0 blocks, although I have seen a case where there was difference of 1
block. When the error tar file comes out, you can echo $? and see a

  1. When a valid tar file or an empty tar file is encountered, $? is 0.

This creates a problem trying to loop and list the contents of all tar
files on the tape, with a varying number of files on the tape, because
of the varying number of days in a month, and a varying gap of bad files
or empty files in between good files.

Currently we use something simple like this:

tape rewind
tar tvf /dev/tp0 # list the contents of the first tar file on the tape

while $? -eq 0
do
tar tvf /dev/tp0 >2 /dev/null # skip the empty tar file
tar tvf /dev/tp0 # list the next good tar
file
done

Problem is, this loop ends when you get a “no input” tar file because $?
returns a 1, and, if there is more than one bad tar file in between,
then it doesn’t work either.

Why are the extra tar files there? I only do one tar cvf command every
night at 2am by cron. Could the tape eod command be adding the extra
gaps? Is this a bug?

The tape forward command is similarly bothered by these empty or error
tar files in between good tar files although its behavior can be
different from doing consecutive tar tvf commands. The tape backup
command doesn’t appear to work at all.

Any suggestions?

Scott

5

Paul Russell wrote:

Careful with tapes - they sometimes spin past the point you would expect.
When you read or write an archive the tape is not guaranteed to stop exactly
at the end of the archive - there might be some dead space (0 to N blocks).
Some newer drives might do this differently - no guarantee.
The only way to guarantee that you are at the eod is to rewind then do an
eod.
If you tar (and it spins a bit past the end) followed by an eod, you might
actually be locking on to the eod of a partially overwritten previous
archive - and yes you’ll thus have several (zero to size of overwritten old
archive) blocks of junk between your true archive and the supposed eod.

Possibilities: When you archive record the last block used by the archive,
or the number of blocks used. Then use these values to direct an exact tape
position command.
You could even do a tape position back a few blocks (less than size of
archive) and then do an eod.

If you find a repetitive solution that works cleanly please email it to me.
I need to do this for an upcoming project.

The following shell script works to the list the contents of a tape with
multiple tar files on it, if the garbage tar files (ones that return “no input”)
don’t occur more than once between good files. You get the “no input” return
from a tar tvf, indefinitely when you reach the end of tape, so I guess the
while test $error_cnt -lt 2 could be changed to be any number above 2 depending
on how many “no input” tar files you get in between good tar files.

#!/bin/sh
let error_cnt=0
tape rewind
while test $error_cnt -lt 2
do
tar tvf /dev/tp0 2>tarout
let result=$?
if test $result -eq 1
then
let error_cnt=error_cnt+1
elif test $result -eq 0
then
cat tarout
let error_cnt=0
fi
done

Non-Tech:
When using a Tape keep the few buttons on a VCR in mind. Also treat it as if
each command is given to a new person with no knowledge of what previous
people have done with the tape - they’ll default to always running the tape
forward unless you tell them otherwise.

Tech:
Remember that the Tape drive is a sequential device with little memory. It
really only knows the block number it is pointing at. It doesn’t know how
you’ve arranged your data on tape and will happily spin along looking for
then next search item (eod…). Note that an eod command will always spin
the tape further along (never backwards) unless you are exactly at an eod -
and even that I wouldn’t guarantee without some testing or reading. Only a
random access device (RAM, Disk drive, CD, …) can jump to the true eod
whether it be forward or back - tapes are very old unintelligent technology,
even when their medium - DAT - is new.

-Paul
paul@jenosys.com

J. Scott Franko <> jsfranko@switch.com> > wrote in message
news:> 3987255A.7B954F90@switch.com> …
Update: I did some test and didn’t use the tape eod comamnd and still got
the problem. It appears the problem is in the pax binary (which process
the
tar command). I wrote a simple 1 block file 3 times in a row. then tape
rewind. then tar tvf. There was an error tar file in between each of the
good files. is there any way to write multiple tar files on the tape
without getting error or empty tar files in between?

Scott

“J. Scott Franko” wrote:

I’m having some trouble using the tape and tar commands.

On a daily basis, a script essentially executes a
cd $JOB_ARC
tape eod
tar cvbf 20 /dev/tp0 $files

to archive a months worth of separate daily tar files on one tape, each
morning at 2am by cron job, where $JOB_ARC is an environment variable
that contains the path to an archive directory and $files is a script
variable that contains a list of files to backup.

If you use tar tvf /dev/tp0 to see what is on the tape, you will see the
first tar file, the first time you execute the tar tvf command after a
rewind. But the second time you execute the command (without
rewinding), you will either see an empty tar file (blocksize: 20
message, but no files) or an error tar file (blocksize:20 message, “no
input” message), or possible multiple error or empty files, before you
see a good one again. The difference in tape position between a valid
tar file, and an empty one is usually 2 tape blocks. The difference in
tape position between a valid tar file and an error tar file is usually
0 blocks, although I have seen a case where there was difference of 1
block. When the error tar file comes out, you can echo $? and see a

  1. When a valid tar file or an empty tar file is encountered, $? is 0.

This creates a problem trying to loop and list the contents of all tar
files on the tape, with a varying number of files on the tape, because
of the varying number of days in a month, and a varying gap of bad files
or empty files in between good files.

Currently we use something simple like this:

tape rewind
tar tvf /dev/tp0 # list the contents of the first tar file on the tape

while $? -eq 0
do
tar tvf /dev/tp0 >2 /dev/null # skip the empty tar file
tar tvf /dev/tp0 # list the next good tar
file
done

Problem is, this loop ends when you get a “no input” tar file because $?
returns a 1, and, if there is more than one bad tar file in between,
then it doesn’t work either.

Why are the extra tar files there? I only do one tar cvf command every
night at 2am by cron. Could the tape eod command be adding the extra
gaps? Is this a bug?

The tape forward command is similarly bothered by these empty or error
tar files in between good tar files although its behavior can be
different from doing consecutive tar tvf commands. The tape backup
command doesn’t appear to work at all.

Any suggestions?

Scott

6

“J. Scott Franko” wrote:

I’m having some trouble using the tape and tar commands.

On a daily basis, a script essentially executes a
cd $JOB_ARC
tape eod
tar cvbf 20 /dev/tp0 $files

to archive a months worth of separate daily tar files on one tape, each
morning at 2am by cron job, where $JOB_ARC is an environment variable
that contains the path to an archive directory and $files is a script
variable that contains a list of files to backup.

If you use tar tvf /dev/tp0 to see what is on the tape, you will see the
first tar file, the first time you execute the tar tvf command after a
rewind. But the second time you execute the command (without
rewinding), you will either see an empty tar file (blocksize: 20
message, but no files) or an error tar file (blocksize:20 message, “no
input” message), or possible multiple error or empty files, before you
see a good one again. The difference in tape position between a valid
tar file, and an empty one is usually 2 tape blocks. The difference in
tape position between a valid tar file and an error tar file is usually
0 blocks, although I have seen a case where there was difference of 1
block. When the error tar file comes out, you can echo $? and see a

  1. When a valid tar file or an empty tar file is encountered, $? is 0.

This creates a problem trying to loop and list the contents of all tar
files on the tape, with a varying number of files on the tape, because
of the varying number of days in a month, and a varying gap of bad files
or empty files in between good files.

Currently we use something simple like this:

tape rewind
tar tvf /dev/tp0 # list the contents of the first tar file on the tape

while $? -eq 0
do
tar tvf /dev/tp0 >2 /dev/null # skip the empty tar file
tar tvf /dev/tp0 # list the next good tar
file
done

Problem is, this loop ends when you get a “no input” tar file because $?
returns a 1, and, if there is more than one bad tar file in between,
then it doesn’t work either.

Why are the extra tar files there? I only do one tar cvf command every
night at 2am by cron. Could the tape eod command be adding the extra
gaps? Is this a bug?

The tape forward command is similarly bothered by these empty or error
tar files in between good tar files although its behavior can be
different from doing consecutive tar tvf commands. The tape backup
command doesn’t appear to work at all.

Any suggestions?

Scott

I’m very new to QNX, but I have been using UNIX for many years, and
tapes for even longer.

What normally happens when you write multiple files to tape (and a
tar file is one tape file) is that you get an empty file between
each tape file. The reason is that when tar completes and closes
the device 2 filemarks are written to mark end of information.
This is what happens on most conventional UNIXs. The QNX tape driver
may be different, but lets assume it isn’t.

So, what you have on your tape is this:

tar-file | filemark | filemark | tar-file | filemark | filemark …

When you read back your first tar it encounters the first filemark
and stops. Now, usually the tape would be left positioned /after/ the
first filemark, that is between two filemarks. Next time you run
tar it has an empty file and reports an error. This should be
consistent,
though and what you report is not consistent. That makes me think
that something else maybe going on.

But to see if what I have described above is what is happening try
this sequence :

tape rewind
tar tvf /dev/tp0
tape forward
tar tvf /dev/tp0

If that still give problems try skipping 2 filemarks:

tape rewind
tar tvf /dev/tp0
tape forward forward
tar tvf /dev/tp0

–

Nigel Wade, System Administrator, Space Plasma Physics Group,
University of Leicester, Leicester, LE1 7RH, UK
E-mail : nmw@ion.le.ac.uk
Phone : +44 (0)116 2523568, Fax : +44 (0)116 2523555

7

Nigel,

I only worked on other Unix for couple years before qnx, but I did work with
Digital Unix, Irix, and Solaris 2.5.1. Before that I new nothing of Unix,
having been a VMS guy for about 12 years. I remember inter record gaps from
vms code I worked with that wrote to tapes, but I didn’t remember running into
this problem on the *nix’s that I used. I’ve only been working with QNX for a
little over 1 year now.

The problem I’m having is that the behavior is not consistent. If you check
my response from Thu, @4:15pm, you can see the script I had to implement to
deal with the inconsitent number of inter tar files. You would think that the
tar executable would deal with the problem if it writes two file marks at the
end of each tar file it writes.

Sometimes the intervening file mark leaves an empty tar file that returns no
file listing and no error, and in that case, the result code ($?) is 0, just
like the return from real tar file that had files in it. But there is also an
inter tar file that returns a tar:“no input” error and a 1 in $?. Most times
there is only one or the other between to good tar files, but what blows up
the code written by my predecessors, is that sometimes two of these inter tar
files or file marks appear. So I had to rewrite the script, and only end if I
get two or more errors in a row. It seems to work so far for July’s backups.
Before I changed the script, to the user, one tape looked like it only had two
tar files on it, when it in fact had 31!.

Try an experiemnt. Tar a couple files to tape. The tar a couple more files
in a second tar file. Repeat if you want to have many tar files. then rewind
and do a tvf and look at the tape position. Then tvf again, and if you get a
“no input” message, look at the tape position. Most times you’ll find it
has not moved. but if you get one of those emtpy files in between, you’d find
that it might move 1 or 2 blocks.

Weird.


Scott

Scott

Nigel Wade wrote:

“J. Scott Franko” wrote:

I’m having some trouble using the tape and tar commands.

On a daily basis, a script essentially executes a
cd $JOB_ARC
tape eod
tar cvbf 20 /dev/tp0 $files

to archive a months worth of separate daily tar files on one tape, each
morning at 2am by cron job, where $JOB_ARC is an environment variable
that contains the path to an archive directory and $files is a script
variable that contains a list of files to backup.

If you use tar tvf /dev/tp0 to see what is on the tape, you will see the
first tar file, the first time you execute the tar tvf command after a
rewind. But the second time you execute the command (without
rewinding), you will either see an empty tar file (blocksize: 20
message, but no files) or an error tar file (blocksize:20 message, “no
input” message), or possible multiple error or empty files, before you
see a good one again. The difference in tape position between a valid
tar file, and an empty one is usually 2 tape blocks. The difference in
tape position between a valid tar file and an error tar file is usually
0 blocks, although I have seen a case where there was difference of 1
block. When the error tar file comes out, you can echo $? and see a

  1. When a valid tar file or an empty tar file is encountered, $? is 0.

This creates a problem trying to loop and list the contents of all tar
files on the tape, with a varying number of files on the tape, because
of the varying number of days in a month, and a varying gap of bad files
or empty files in between good files.

Currently we use something simple like this:

tape rewind
tar tvf /dev/tp0 # list the contents of the first tar file on the tape

while $? -eq 0
do
tar tvf /dev/tp0 >2 /dev/null # skip the empty tar file
tar tvf /dev/tp0 # list the next good tar
file
done

Problem is, this loop ends when you get a “no input” tar file because $?
returns a 1, and, if there is more than one bad tar file in between,
then it doesn’t work either.

Why are the extra tar files there? I only do one tar cvf command every
night at 2am by cron. Could the tape eod command be adding the extra
gaps? Is this a bug?

The tape forward command is similarly bothered by these empty or error
tar files in between good tar files although its behavior can be
different from doing consecutive tar tvf commands. The tape backup
command doesn’t appear to work at all.

Any suggestions?

Scott

I’m very new to QNX, but I have been using UNIX for many years, and
tapes for even longer.

What normally happens when you write multiple files to tape (and a
tar file is one tape file) is that you get an empty file between
each tape file. The reason is that when tar completes and closes
the device 2 filemarks are written to mark end of information.
This is what happens on most conventional UNIXs. The QNX tape driver
may be different, but lets assume it isn’t.

So, what you have on your tape is this:

tar-file | filemark | filemark | tar-file | filemark | filemark …

When you read back your first tar it encounters the first filemark
and stops. Now, usually the tape would be left positioned /after/ the
first filemark, that is between two filemarks. Next time you run
tar it has an empty file and reports an error. This should be
consistent,
though and what you report is not consistent. That makes me think
that something else maybe going on.

But to see if what I have described above is what is happening try
this sequence :

tape rewind
tar tvf /dev/tp0
tape forward
tar tvf /dev/tp0

If that still give problems try skipping 2 filemarks:

tape rewind
tar tvf /dev/tp0
tape forward forward
tar tvf /dev/tp0

–

Nigel Wade, System Administrator, Space Plasma Physics Group,
University of Leicester, Leicester, LE1 7RH, UK
E-mail : > nmw@ion.le.ac.uk
Phone : +44 (0)116 2523568, Fax : +44 (0)116 2523555

8

“J. Scott Franko” wrote:

Nigel,

I only worked on other Unix for couple years before qnx, but I did work with
Digital Unix, Irix, and Solaris 2.5.1. Before that I new nothing of Unix,
having been a VMS guy for about 12 years. I remember inter record gaps from
vms code I worked with that wrote to tapes, but I didn’t remember running into
this problem on the *nix’s that I used. I’ve only been working with QNX for a
little over 1 year now.

The problem I’m having is that the behavior is not consistent. If you check
my response from Thu, @4:15pm, you can see the script I had to implement to
deal with the inconsitent number of inter tar files. You would think that the
tar executable would deal with the problem if it writes two file marks at the
end of each tar file it writes.

It may well be a feature of the device driver. The way filemarks are
handled,
particularly double filemarks is up to the device driver. There may be
POSIX
specs. now, but I remember one flavour of UNIX I used to use where if
you read
two consequtive filemarks the device driver would not attempt to read
any
further info from the device and just return EOF until you re-positioned
the
device.

Sometimes the intervening file mark leaves an empty tar file that returns no
file listing and no error, and in that case, the result code ($?) is 0, just
like the return from real tar file that had files in it. But there is also an
inter tar file that returns a tar:“no input” error and a 1 in $?. Most times
there is only one or the other between to good tar files, but what blows up
the code written by my predecessors, is that sometimes two of these inter tar
files or file marks appear. So I had to rewrite the script, and only end if I
get two or more errors in a row. It seems to work so far for July’s backups.
Before I changed the script, to the user, one tape looked like it only had two
tar files on it, when it in fact had 31!.

Tar is not really the best program for determining what is really on
the tape.
It’s probably doing its best to read the contents, but all sorts of
things may
be going on internally which you are not aware of. E.g. what does tar do
if
the first read returns 0 (i.e. EOF), does it do a re-read or report an
error?
If it does a re-read it may silently skip a filemark and you are none
the
wiser. What does the device driver do if the first read after open
fails, does
it leave the tape positioned before or after the filemark? All these are
variables
which using tar will not alert you to. Most UNIX’s I have used document
these
features in the man pages for the specific device driver e.g. tps and/or
mtio for
IRIX, but I wouldn’t know where to start looking in QNX. Just as an
example here
is what the tps man page under IRIX ways:

  1. If the first read after an open encounters a filemark before
    transferring any data and the tape was not known to be at the
    top of
    a filemark or BOT, the filemark is skipped and the read
    retried. Any
    further errors are reported exactly as they occur. In
    particular, if
    two sequential filemarks are found, the tape is positioned
    between
    them.

  2. On close, if the last tape movement operation was a successful
    write
    (not a write filemark), the following happens:

a) Half-inch tape drives write two filemarks and then
backspace one
file, so that the tape is positioned between the two
filemarks
just written. If, however, the tape is not in ansi mode
and
early warning has been encountered, no write filemark or
backspace is done; filemarks are never written in audio
mode.

b) Other tape drives write one filemark. No backspace is
done.

“Features” like this can be a nightmare when it comes to developing
reliable
and consistent tape reading software.

This is pure speculation on my part, but tar itself could be
inconsistent.
For example, if the tape device is fixed block length tar will pad the
last block of a tar-file with zero bytes up to the block length. When it
reads the data back it may well recognize these zero bytes a the end of
the tar file and not bother to read any further, leaving the tape
positioned
just /before/ the filemark. Another tarfile may fill up the final block
and
not need to pad with zeros. Reading this file back, tar would not know
that
the end of the file had been reached until it read the filemark. This
would leave the tape positioned /after/ the filemark.

Try an experiemnt. Tar a couple files to tape. The tar a couple more files
in a second tar file. Repeat if you want to have many tar files. then rewind
and do a tvf and look at the tape position. Then tvf again, and if you get a
“no input” message, look at the tape position. Most times you’ll find it
has not moved. but if you get one of those emtpy files in between, you’d find
that it might move 1 or 2 blocks.

Sorry, I can’t, we don’t have a tape device on any of our QNX systems.
Even if we did we might need to use a different device driver with
different
characteristics.


I’d begin by trying to determine exactly what is on the tape. Are there
double
filemarks? Is there sometimes really a tar file with no contents, or is
it a
mis-reporting by tar? If it’s a real file with no contents how did it
get there,
something must have created it? Does tar write two filemarks when it
closes the
device after a write? Does “tape eod” position the tape after the first
or
second filemark if there are 2 filemarks at the end?

I’d write a simple C program to read the tape a block at a time and
report
the number of blocks read per file. Then compare that to what tar
reports.

Good luck!

–

Nigel Wade, System Administrator, Space Plasma Physics Group,
University of Leicester, Leicester, LE1 7RH, UK
E-mail : nmw@ion.le.ac.uk
Phone : +44 (0)116 2523568, Fax : +44 (0)116 2523555

9

Thanks for the info! I’ll give it a try (writing a small C program). You’ve got a
wealth of experience!

Scott

Nigel Wade wrote:

“J. Scott Franko” wrote:

Nigel,

I only worked on other Unix for couple years before qnx, but I did work with
Digital Unix, Irix, and Solaris 2.5.1. Before that I new nothing of Unix,
having been a VMS guy for about 12 years. I remember inter record gaps from
vms code I worked with that wrote to tapes, but I didn’t remember running into
this problem on the *nix’s that I used. I’ve only been working with QNX for a
little over 1 year now.

The problem I’m having is that the behavior is not consistent. If you check
my response from Thu, @4:15pm, you can see the script I had to implement to
deal with the inconsitent number of inter tar files. You would think that the
tar executable would deal with the problem if it writes two file marks at the
end of each tar file it writes.

It may well be a feature of the device driver. The way filemarks are
handled,
particularly double filemarks is up to the device driver. There may be
POSIX
specs. now, but I remember one flavour of UNIX I used to use where if
you read
two consequtive filemarks the device driver would not attempt to read
any
further info from the device and just return EOF until you re-positioned
the
device.


Sometimes the intervening file mark leaves an empty tar file that returns no
file listing and no error, and in that case, the result code ($?) is 0, just
like the return from real tar file that had files in it. But there is also an
inter tar file that returns a tar:“no input” error and a 1 in $?. Most times
there is only one or the other between to good tar files, but what blows up
the code written by my predecessors, is that sometimes two of these inter tar
files or file marks appear. So I had to rewrite the script, and only end if I
get two or more errors in a row. It seems to work so far for July’s backups.
Before I changed the script, to the user, one tape looked like it only had two
tar files on it, when it in fact had 31!.

Tar is not really the best program for determining what is really on
the tape.
It’s probably doing its best to read the contents, but all sorts of
things may
be going on internally which you are not aware of. E.g. what does tar do
if
the first read returns 0 (i.e. EOF), does it do a re-read or report an
error?
If it does a re-read it may silently skip a filemark and you are none
the
wiser. What does the device driver do if the first read after open
fails, does
it leave the tape positioned before or after the filemark? All these are
variables
which using tar will not alert you to. Most UNIX’s I have used document
these
features in the man pages for the specific device driver e.g. tps and/or
mtio for
IRIX, but I wouldn’t know where to start looking in QNX. Just as an
example here
is what the tps man page under IRIX ways:

  1. If the first read after an open encounters a filemark before
    transferring any data and the tape was not known to be at the
    top of
    a filemark or BOT, the filemark is skipped and the read
    retried. Any
    further errors are reported exactly as they occur. In
    particular, if
    two sequential filemarks are found, the tape is positioned
    between
    them.

  2. On close, if the last tape movement operation was a successful
    write
    (not a write filemark), the following happens:

a) Half-inch tape drives write two filemarks and then
backspace one
file, so that the tape is positioned between the two
filemarks
just written. If, however, the tape is not in ansi mode
and
early warning has been encountered, no write filemark or
backspace is done; filemarks are never written in audio
mode.

b) Other tape drives write one filemark. No backspace is
done.

“Features” like this can be a nightmare when it comes to developing
reliable
and consistent tape reading software.

This is pure speculation on my part, but tar itself could be
inconsistent.
For example, if the tape device is fixed block length tar will pad the
last block of a tar-file with zero bytes up to the block length. When it
reads the data back it may well recognize these zero bytes a the end of
the tar file and not bother to read any further, leaving the tape
positioned
just /before/ the filemark. Another tarfile may fill up the final block
and
not need to pad with zeros. Reading this file back, tar would not know
that
the end of the file had been reached until it read the filemark. This
would leave the tape positioned /after/ the filemark.


Try an experiemnt. Tar a couple files to tape. The tar a couple more files
in a second tar file. Repeat if you want to have many tar files. then rewind
and do a tvf and look at the tape position. Then tvf again, and if you get a
“no input” message, look at the tape position. Most times you’ll find it
has not moved. but if you get one of those emtpy files in between, you’d find
that it might move 1 or 2 blocks.

Sorry, I can’t, we don’t have a tape device on any of our QNX systems.
Even if we did we might need to use a different device driver with
different
characteristics.

I’d begin by trying to determine exactly what is on the tape. Are there
double
filemarks? Is there sometimes really a tar file with no contents, or is
it a
mis-reporting by tar? If it’s a real file with no contents how did it
get there,
something must have created it? Does tar write two filemarks when it
closes the
device after a write? Does “tape eod” position the tape after the first
or
second filemark if there are 2 filemarks at the end?

I’d write a simple C program to read the tape a block at a time and
report
the number of blocks read per file. Then compare that to what tar
reports.

Good luck!

–

Nigel Wade, System Administrator, Space Plasma Physics Group,
University of Leicester, Leicester, LE1 7RH, UK
E-mail : > nmw@ion.le.ac.uk
Phone : +44 (0)116 2523568, Fax : +44 (0)116 2523555

10

“J. Scott Franko” wrote:

Thanks for the info! I’ll give it a try (writing a small C program). You’ve got a
wealth of experience!

Scott

It’s just a sign of the passing years…

Unfortunately I learned about tapes via a harder route, using FORTRAN
under
PRIMOS and VMS. UNIX/C was so much simpler when we finally migrated.

–

Nigel Wade, System Administrator, Space Plasma Physics Group,
University of Leicester, Leicester, LE1 7RH, UK
E-mail : nmw@ion.le.ac.uk
Phone : +44 (0)116 2523568, Fax : +44 (0)116 2523555

11

“J. Scott Franko” wrote:

Update: I did some test and didn’t use the tape eod comamnd and still got
the problem. It appears the problem is in the pax binary (which process the
tar command). I wrote a simple 1 block file 3 times in a row. then tape
rewind. then tar tvf. There was an error tar file in between each of the
good files. is there any way to write multiple tar files on the tape
without getting error or empty tar files in between?

This “empty” file is actually a marker between output streams
(called a “file mark”) as the tape is just a sequential stream, and needs
some way to separate contents. One is automatically written to the
tape by Fsys when you close() the raw device after doing some number
of write()s; you can also programmatically force them to be issued via
ioctl(). Trying to read a file mark acts as a one-shot EOF. You can
advance over them manually with “tape forward” after processing
each of your pax archives.

12

Yeah, but it isn’t consistent. Sometimes there are two “file marks” in between
and one of returns like and empty tar file and one of them returns and i/o error.
It basically sucks for successive tar files on the same tape. tape forward and
tape backward are inconsistent in this case. You need to read the details in
the previous posts in this thread.

John Garvey wrote:

“J. Scott Franko” wrote:

Update: I did some test and didn’t use the tape eod comamnd and still got
the problem. It appears the problem is in the pax binary (which process the
tar command). I wrote a simple 1 block file 3 times in a row. then tape
rewind. then tar tvf. There was an error tar file in between each of the
good files. is there any way to write multiple tar files on the tape
without getting error or empty tar files in between?

This “empty” file is actually a marker between output streams
(called a “file mark”) as the tape is just a sequential stream, and needs
some way to separate contents. One is automatically written to the
tape by Fsys when you close() the raw device after doing some number
of write()s; you can also programmatically force them to be issued via
ioctl(). Trying to read a file mark acts as a one-shot EOF. You can
advance over them manually with “tape forward” after processing
each of your pax archives.