real-time data archiving on hard disk

QNX Newsgroups (Archives) qdn.public.qnx4
1

Hi,

I’m new to QNX4, and I’m writing C programs using Watcom 10.6.

I need to store data on disk at high rate (256 bytes / msec).
Every millisecond I have a new 256 bytes data packet to write on disk.
I need to program a fast process (< 1 msec) that will do this task.

I did some tests to see how much time it takes to write a 256 bytes packet
to the disk. I changed the ticksize to 0.25 msec and I’m reading the clock
to get a timestamp. So I get a first timestamp, I write to disk and I get a
second timestamp. I put that sequence in a loop. Most of the time the
difference between the 2 timestamps is 0 but once a while I can see a
difference of 1 to 5 msec (often after 64536 bytes have been written) which
is my worst case. I feel like the Fsys.eide driver wait for his buffer to
be full before initiating a write to disk and when the write sequence start,
it last 1 to 5 msec. My process can’t be that long, my worst case must last
under 1 msec.

Instead of 1 long writing sequence, I would prefer severals small writing
sequence

I did try to change my process’ priority for a higher priority (29) but it
did not change anything.
I did try to do a fflush() after a fwrite() of 256 bytes but it didn’t
change anything.
I did try to use unbuffered file I/O → use open() with O_DSYNC option
instead of fopen but it was worse, without O_DSYNC option it was the same as
buffered file I/O.
Should I use 2 process to achieve this task? Shared Mem? Proxies?
Should I try to reduce to Fsys.eide buffer using the -m option?

Do you have any suggestions?

Thanks for your help!

Benoit

2

Previously, Benoit Adam wrote in qdn.public.qnx4:

I need to store data on disk at high rate (256 bytes / msec).
Every millisecond I have a new 256 bytes data packet to write on disk.
I need to program a fast process (< 1 msec) that will do this task.

Does it actually have to be on the media within 1msec, or do you just
want the write to take that long?

I did some tests to see how much time it takes to write a 256 bytes packet
to the disk. I changed the ticksize to 0.25 msec and I’m reading the clock
to get a timestamp. So I get a first timestamp, I write to disk and I get a
second timestamp. I put that sequence in a loop. Most of the time the
difference between the 2 timestamps is 0

That’s because the data just went into the Fsys cache.

but once a while I can see a
difference of 1 to 5 msec (often after 64536 bytes have been written) which
is my worst case.

What kind of hard drive are you using. It could be that the driver was
moving data through the io port when you tried to do your write, keeping
the system busy. Or it could be some other interruption. The beta program
DeJa View would be mighty handy about right now.

I feel like the Fsys.eide driver wait for his buffer to
be full before initiating a write to disk and when the write sequence start,
it last 1 to 5 msec.

Something like that’s probably about right. There are at least two
places that buffering might be occuring. If you are using fwrite(),
it might be in your application program. Otherwise it might be in
Fsys.

You could fool around with fflush() and the fcntl()
parameters to Fsys to try to get Fsys to behave the way you want, but
here is a failsafe method.

Instead of doing writes from your program, create to new processes in this
configuration:


YOUR-PROCESS —(Send)–> BUFFER-PROCESS <–(Send)— IO-PROCESS -(write)----> Fsys

When your process wants to do a write, it sends your data to the BUFFER process.
The BUFFER process puts the data in a circular buffer. The IO-PROCESS sends a
message to the BUFFER-PROCESS asking for data to write. If there is none, it gets
pended until there is some. If there is some, it gets replied back. The IO-PROCESS
then does the writes to disk.

Run YOUR-PROCESS and the BUFFER-PROCESS at a higher priority
than Fsys, and give BUFFER-PROCESS enough room, and you will
meet your deadlines, unless something else of higher priority is
in the way.


Mitchell Schoenbrun --------- maschoen@pobox.com

3

What kind of hard drive are you using. It could be that the driver
was
moving data through the io port when you tried to do your write,
keeping
the system busy. Or it could be some other interruption. The beta
program
DeJa View would be mighty handy about right now.

To add to this: you should investigate getting an AV drive (this
actually might not be necessary since 256K/sec really isn’t that hard to
achieve unless you have very limited buffer space - i.e. RAM). I have
seen 400K/sec sustained on a 486 with a SCSI drive (with ample buffering

  • to cover drive recalibrations).
4

Previously, Rennie Allen wrote in qdn.public.qnx4:

To add to this: you should investigate getting an AV drive

I’m not standing on this statement or anything, but the last
time I asked about whether a particular new drive was AV I
was told that they all are now. I’m curious if anyone else
has hear pro or con on this.

Mitchell Schoenbrun --------- maschoen@pobox.com

5

I wouldn’t be at all surprised (they weren’t when I was working with
486’s though :slight_smile:

-----Original Message-----
From: Mitchell Schoenbrun [mailto:maschoen@pobox.com]
Posted At: Thursday, April 26, 2001 1:36 PM
Posted To: qnx4
Conversation: real-time data archiving on hard disk
Subject: Re: real-time data archiving on hard disk


Previously, Rennie Allen wrote in qdn.public.qnx4:

To add to this: you should investigate getting an AV drive

I’m not standing on this statement or anything, but the last
time I asked about whether a particular new drive was AV I
was told that they all are now. I’m curious if anyone else
has hear pro or con on this.

Mitchell Schoenbrun --------- maschoen@pobox.com

6

what is AV ?

Previously, Mitchell Schoenbrun wrote in qdn.public.qnx4:

Previously, Rennie Allen wrote in qdn.public.qnx4:

To add to this: you should investigate getting an AV drive

I’m not standing on this statement or anything, but the last
time I asked about whether a particular new drive was AV I
was told that they all are now. I’m curious if anyone else
has hear pro or con on this.

Mitchell Schoenbrun --------- > maschoen@pobox.com
\

7

Previously, Ian Cannon wrote in qdn.public.qnx4:

what is AV ?

Audio/Video? Audio-Visual? Something like that. Basically a
drive that needs to provide a reliable predictable maximum
latency for streaming applications. Some drives were prone
to recalibrating themselfs whenever they wanted to due to
thermal changes. This could put a sizable glitch in their
performance for a very small amount of time. Unfortunately
this amount of time was large enough that you could lose a
video frame.


Previously, Mitchell Schoenbrun wrote in qdn.public.qnx4:
Previously, Rennie Allen wrote in qdn.public.qnx4:

To add to this: you should investigate getting an AV drive

I’m not standing on this statement or anything, but the last
time I asked about whether a particular new drive was AV I
was told that they all are now. I’m curious if anyone else
has hear pro or con on this.

Mitchell Schoenbrun --------- > maschoen@pobox.com



\

–
Mitchell Schoenbrun --------- maschoen@pobox.com

8

I think you need to take Mitchell’s suggestion and create a separate process
to do the writing to disk. However, you might do it a bit more simply using
only one extra process and “pending reply” messaging with your process.
Borrowing from Mitchell:

YOUR-PROCESS <–(Send)— IO-PROCESS -(write)----> Fsys

YOUR-PROCESS buffers the data until it receives a message from IO-PROCESS,
at which point it Reply()s with the buffered data (which may consist of more
than 1 data frame). IO-PROCESS then leisurely writes the data to disk and
Send()s another message to YOUR-PROCESS. If YOUR-PROCESS doesn’t have any
data yet, it doesn’t Reply(), but just sets a flag to remember that
IO-PROCESS is awaiting a Reply(). Then when YOUR-PROCESS has more data, it
can Reply(), and on it goes.

Regards,

Bert Menkveld
Engineer
Corman Technologies Inc.
bert@cormantech.com

“Mitchell Schoenbrun” <maschoen@pobox.com> wrote in message
news:Voyager.010426003839.5730A@schoenbrun.com…

Previously, Benoit Adam wrote in qdn.public.qnx4:

I need to store data on disk at high rate (256 bytes / msec).
Every millisecond I have a new 256 bytes data packet to write on disk.
I need to program a fast process (< 1 msec) that will do this task.

Does it actually have to be on the media within 1msec, or do you just
want the write to take that long?

I did some tests to see how much time it takes to write a 256 bytes
packet
to the disk. I changed the ticksize to 0.25 msec and I’m reading the
clock
to get a timestamp. So I get a first timestamp, I write to disk and I
get a
second timestamp. I put that sequence in a loop. Most of the time the
difference between the 2 timestamps is 0

That’s because the data just went into the Fsys cache.

but once a while I can see a
difference of 1 to 5 msec (often after 64536 bytes have been written)
which
is my worst case.

What kind of hard drive are you using. It could be that the driver was
moving data through the io port when you tried to do your write, keeping
the system busy. Or it could be some other interruption. The beta
program
DeJa View would be mighty handy about right now.

I feel like the Fsys.eide driver wait for his buffer to
be full before initiating a write to disk and when the write sequence
start,
it last 1 to 5 msec.

Something like that’s probably about right. There are at least two
places that buffering might be occuring. If you are using fwrite(),
it might be in your application program. Otherwise it might be in
Fsys.

You could fool around with fflush() and the fcntl()
parameters to Fsys to try to get Fsys to behave the way you want, but
here is a failsafe method.

Instead of doing writes from your program, create to new processes in this
configuration:


YOUR-PROCESS —(Send)–> BUFFER-PROCESS <–(Send)—
IO-PROCESS -(write)----> Fsys

When your process wants to do a write, it sends your data to the BUFFER
process.
The BUFFER process puts the data in a circular buffer. The IO-PROCESS
sends a
message to the BUFFER-PROCESS asking for data to write. If there is none,
it gets
pended until there is some. If there is some, it gets replied back. The
IO-PROCESS
then does the writes to disk.

Run YOUR-PROCESS and the BUFFER-PROCESS at a higher priority
than Fsys, and give BUFFER-PROCESS enough room, and you will
meet your deadlines, unless something else of higher priority is
in the way.


Mitchell Schoenbrun --------- > maschoen@pobox.com

9

Previously, Bert Menkveld wrote in qdn.public.qnx4:

I think you need to take Mitchell’s suggestion and create a separate process
to do the writing to disk. However, you might do it a bit more simply using
only one extra process and “pending reply” messaging with your process.
Borrowing from Mitchell:

YOUR-PROCESS <–(Send)— IO-PROCESS -(write)----> Fsys

This is a good simplification if your process can go to sleep
in RECEIVE mode. You cut down on movement of the data as well
as the extra message passing context switches. The advantage
to my approach is that you build it as a drop in black box.
That is you substitute your write() or fwrite() call with
a routine that does send.

That brings up an interesting possibility. You could write an
intermediary admin between your process and Fsys that would
buffer. In that mode, you would not change your code at all,
but just open a different filename. For example instead of

fopen("/mydir/myfile",“w”);

you would open

fopen("/fastwrite/mydir/myfile",“w”);





Mitchell Schoenbrun --------- maschoen@pobox.com