Non blocking file system writes

Hi All,

One of our real-time processes is responsible for writing log data
to the hard disk. It is currently using fopen, fwrite (append mode),
and then fclose for each message it receives. We also have the
capability
of downloading a large tar file to our system. First it goes to a
ramdisk
and is then copied (using cp within a C system call) to the hard disk.
I think that our cache is being filled during this copy and the 1st
process mentioned above is getting starved although it is a higher
priority (possibly because the cache is full). FYI, our Fsys cache
is set to 100K with the default delay (3sec).

Soooo, my question is should the higher priority process use open
instead with the O_NONBLOCK flag? Will this skip the Fsys cache and
go directly from the process cache (if one exists) to the disk?

Any other suggestions like changing the cache size or delay time?

Thanks,
Rob

Previously, Rob Davidson wrote in qdn.public.qnx4:

Soooo, my question is should the higher priority process use open
instead with the O_NONBLOCK flag? Will this skip the Fsys cache and
go directly from the process cache (if one exists) to the disk?

All disk writes go through the Fsys cache. You can get around this
problem by writing a set of intermediate processes as follows:

OLD Application -------------> Fsys


New Application ----> Cache-Process <---- Helper -----> Fsys

The Cache process would never block for very long, and would always
reply immediately to your application. The helper would get data from
the Cache-Process and write it to disk. Now assuming that your
Cache-Process has enough memory, should be able to make your own non-blocking
I/O.

Mitchell Schoenbrun --------- maschoen@pobox.com

“Rob Davidson” <rdavidson@SoftwareRemodeling.com> wrote in message
news:3AD5F266.88C3E736@SoftwareRemodeling.com

Hi All,

One of our real-time processes is responsible for writing log data
to the hard disk. It is currently using fopen, fwrite (append mode),
and then fclose for each message it receives. We also have the
capability
of downloading a large tar file to our system. First it goes to a
ramdisk
and is then copied (using cp within a C system call) to the hard disk.
I think that our cache is being filled during this copy and the 1st
process mentioned above is getting starved although it is a higher
priority (possibly because the cache is full). FYI, our Fsys cache
is set to 100K with the default delay (3sec).

Soooo, my question is should the higher priority process use open
instead with the O_NONBLOCK flag? Will this skip the Fsys cache and
go directly from the process cache (if one exists) to the disk?

Any other suggestions like changing the cache size or delay time?

I’m not sure I understand this 100%. I 'm confuse by the “the 1st process
mentioned above is getting starved”. Does it mean it starved CPU
time because it blocks on the write?

By the way you should use open/read/write/ instead of fopen if you
write are > 512 bytes. They have less overehead.

I don’t think it’s possible to do async operation with select or O_NONBLOCK.
Maybe FSYS supports O_NONBLOCK and the write operation would
return with -1 (can’t write) not sure.

If you generate data faster then the HD can handle them, there is no
other solution then to buffer the data. Personnaly I always deal with
this with a process with two threads (or two processes and share memory
if you don’t feel comfortablee with threads). Once process receives
the data and puts it in a circular queue, this is very fast. The client
would block only if the queue is full. The other process (running at low
priority)
flushes the queue to the disk.

This works very well and is very real-time friendly. Every time I had
to do this I wrote the “logger” as an IO/manager so client can
use open/read/write/close. The client think he’s writting to disk
while in fact he’s writting in the logger’s cache.

If you are interested in this I can check if i’m allow to make the
code public (it doesn’t belong to me)

Thanks,
Rob