File Operations

I’m curious about how QNX 6.1.0 handles file I/O. Specifically, when can I
be sure a file is completely written to the disk? I have a file that I want
to back up, but I don’t want to copy the original to the backup until the
original is completely written to disk. I think the driver, devb-doc in this
case, may still be writing to a disk-on-chip, even after the fclose command
returns.

The only evidence I have of this situation is the state of the original file
and the backup. When I need to save the original file, I write it out and
then insert a line count at the head of the file. Then I fflush and fclose
the file. After the fclose returns, I use a system call to “cp” the original
file to the backup. On one occaision, I had an incomplete original file and
an identical backup file. So somehow, the “cp” was made before the file was
completely written to the disk. Or so it seems.

Any comments or questions?

David Kuechenmeister

“David Kuechenmeister” <david.kuechenmeister@viasat.com> wrote in message
news:bjdddg$56e$1@inn.qnx.com

I’m curious about how QNX 6.1.0 handles file I/O. Specifically, when can I
be sure a file is completely written to the disk? I have a file that I
want
to back up, but I don’t want to copy the original to the backup until the
original is completely written to disk. I think the driver, devb-doc in
this
case, may still be writing to a disk-on-chip, even after the fclose
command
returns.

It probably does. Thanks to the cache…

The only evidence I have of this situation is the state of the original
file
and the backup. When I need to save the original file, I write it out and
then insert a line count at the head of the file. Then I fflush and fclose
the file. After the fclose returns, I use a system call to “cp” the
original
file to the backup. On one occaision, I had an incomplete original file
and
an identical backup file. So somehow, the “cp” was made before the file
was
completely written to the disk. Or so it seems.

If you started ‘cp’ after fflush(), the cp should get the data up to the
point where it was flushed. The fflush() does not guarantee completion, but
it is supposed to guarantee ordering. After all, that’s why it exists - to
allow intermixed read() and write() calls. If cp really can read incomplete
data after flushing, that would be a bug.

Any comments or questions?

I don’t know of any way to ensure the data is commited to disk, short of
sending SIGTERM to the driver. But that would also kill it of course :wink:
In QNX4 there was SIGPWR for that, but afaik this convention was dropped.
You can mount filesystem in ‘synchronous mode’ which means all writes are
synchronous but it makes things quite slow. You can call sync() but it also
returns before data is commited - it just initiates cache flush. If you wait
sufficient time after the sync(), you should be ok. Not quite elegant, but
better than nothing.

– igor

“Igor Kovalenko” <kovalenko@attbi.com> wrote in message
news:bje3d0$imi$1@inn.qnx.com

“David Kuechenmeister” <> david.kuechenmeister@viasat.com> > wrote in message
news:bjdddg$56e$> 1@inn.qnx.com> …
I’m curious about how QNX 6.1.0 handles file I/O. Specifically, when can
I
be sure a file is completely written to the disk? I have a file that I
want
to back up, but I don’t want to copy the original to the backup until
the
original is completely written to disk. I think the driver, devb-doc in
this
case, may still be writing to a disk-on-chip, even after the fclose
command
returns.


It probably does. Thanks to the cache…

The only evidence I have of this situation is the state of the original
file
and the backup. When I need to save the original file, I write it out
and
then insert a line count at the head of the file. Then I fflush and
fclose
the file. After the fclose returns, I use a system call to “cp” the
original
file to the backup. On one occaision, I had an incomplete original file
and
an identical backup file. So somehow, the “cp” was made before the file
was
completely written to the disk. Or so it seems.


If you started ‘cp’ after fflush(), the cp should get the data up to the
point where it was flushed. The fflush() does not guarantee completion,
but
it is supposed to guarantee ordering. After all, that’s why it exists - to
allow intermixed read() and write() calls. If cp really can read
incomplete
data after flushing, that would be a bug.

Any comments or questions?


I don’t know of any way to ensure the data is commited to disk, short of
sending SIGTERM to the driver. But that would also kill it of course > :wink:
In QNX4 there was SIGPWR for that, but afaik this convention was dropped.
You can mount filesystem in ‘synchronous mode’ which means all writes are
synchronous but it makes things quite slow. You can call sync() but it
also
returns before data is commited - it just initiates cache flush. If you
wait
sufficient time after the sync(), you should be ok. Not quite elegant, but
better than nothing.

Isn’t strange given the nature of the operating system that no
services/method are availble to ensure data written is commited.


– igor

David Kuechenmeister <david.kuechenmeister@viasat.com> wrote:

I’m curious about how QNX 6.1.0 handles file I/O. Specifically, when can I
be sure a file is completely written to the disk? I have a file that I want

The commit policy is controllable on a per-mount basis (commit=, can set
to always force synchronous writes or always allow asynchronous metadata
or a combination in-between), or on a per-file basis (when open using
O_SYNC flag or using fsync()), plus of course the “sync” utility.

original is completely written to disk. I think the driver, devb-doc in this
case, may still be writing to a disk-on-chip, even after the fclose command
returns.

Yes, quite possibly, close() does not force a sync/flush, any written
data may still be in cache or being written to disk.

then insert a line count at the head of the file. Then I fflush and fclose
the file. After the fclose returns, I use a system call to “cp” the original

No, fflush() is a stdio level operation, and nothing to do with the disk
subsystem (well, it will initiate a write if necessary but not a sync).
Try the following sequence:

fflush(fp);
fsync(fileno(fp));
fclose(fp);

Note that “cp” also does not do any form of syncing either (so the
destination file may not be fully on-disk when the utility returns).
I’ve always felt that would be a useful cp flag, but you may have to
write your own read/write utility …

Any comments or questions?

Typically the bias of performance/robustness must be selected to match
the system usage (for high-reliability you would mount “commit=high” to
force every write to go immediately to disk, with obvious performance
impact; for higher performance you’d let it linger longer in cache, but
need to be cautious during shutdown/power-loss; in other cases you
do it on a per-file, and use open(O_SYNC) or fsync() to force out data
from critical files and leaves others to the default caching policy).

david.kuechenmeister@viasat.com sed in <bjdddg$56e$1@inn.qnx.com>:

and the backup. When I need to save the original file, I write it out and
then insert a line count at the head of the file. Then I fflush and fclose
the file. After the fclose returns, I use a system call to “cp” the original

Alarm rings here;

Have you properly fseek()ed between read/write transitions?
POSIX/stdio is picky on this when intermixing read and write.
(I’ve bitten hard when using perl)

Dig documents for fseek(), fopen(), fflush() for relevant warnings (if any)

kabe