Storing data at 10 Mb/s

In my application i need to store approximately 10 Mbytes of data
every second. What is the best solution to achive that?
My OS is QNX4.25D on P-III 500MHz.

I’ve tried to just write the data to a file, but i only managed to
get 3-4 Mb/s. And i have a strong suspicion that Fsys.eide doesn’t
take advantage of UltraDMA modes. (However i didn’t change
the Fsys/Fsys.eide defaults.)

Another attempt was done to stream the data thru tcp/ip socket and
have another OS (more skilled with hdds) to store it.
It gave me 4-5Mb/s on ethernet segment (not a cross-over cable).

Please input your ideas.
Thanks in advance
Dmitri

Dmitri Ivanov <ivdal@yahoo.com> wrote:

In my application i need to store approximately 10 Mbytes of data
every second. What is the best solution to achive that?

That is probably beyond repIO capabilities. Is SCSI an option for you?
That should provide 20-30MB/sec throughput. Or you could contact your
customer/sales representative about a potential EIDE solution.

And i have a strong suspicion that Fsys.eide doesn’t
take advantage of UltraDMA modes.

No, Fsys.eide does not officially support any DMA modes (PIO only).

Thanks a lot!

In a local vendor’s catalogue i came across an Ultra 2 SCSI controller
card with Symbios SYM53C895 chipset. (Btw, its Advantech
MIC-3640 cPCI card (i need cPCI)). They promise 80MB/s.
It seems that Fsys.sym8scsi driver supports that chipset.
A question for everybody: Can you say anything good/bad from your experience
about this card/chipset/driver?

Dmitri Ivanov <> ivdal@yahoo.com> > wrote:
In my application i need to store approximately 10 Mbytes of data
every second. What is the best solution to achive that?

That is probably beyond repIO capabilities. Is SCSI an option for you?
That should provide 20-30MB/sec throughput. Or you could contact your
customer/sales representative about a potential EIDE solution.

And i have a strong suspicion that Fsys.eide doesn’t
take advantage of UltraDMA modes.

No, Fsys.eide does not officially support any DMA modes (PIO only).

“Dmitri Ivanov” <ivdal@yahoo.com> wrote in message
news:ap5hg0$c37$1@inn.qnx.com

Thanks a lot!

In a local vendor’s catalogue i came across an Ultra 2 SCSI controller
card with Symbios SYM53C895 chipset. (Btw, its Advantech
MIC-3640 cPCI card (i need cPCI)). They promise 80MB/s.

Well that’s easier said then done, (what ever the OS). SCSI transfert rate
is one thing, but then you have LOTS of other factor to consider. CPU
speed, RAM speep, DISK speed, Filesystem caracteristic, disk fragmentation,
etc. IHMO 80 Mb/s is impossible under any OS mostly when it’s write
speed…

It seems that Fsys.sym8scsi driver supports that chipset.
A question for everybody: Can you say anything good/bad from your
experience
about this card/chipset/driver?

Dmitri Ivanov <> ivdal@yahoo.com> > wrote:
In my application i need to store approximately 10 Mbytes of data
every second. What is the best solution to achive that?

That is probably beyond repIO capabilities. Is SCSI an option for you?
That should provide 20-30MB/sec throughput. Or you could contact your
customer/sales representative about a potential EIDE solution.

And i have a strong suspicion that Fsys.eide doesn’t
take advantage of UltraDMA modes.

No, Fsys.eide does not officially support any DMA modes (PIO only).

Dmitri Ivanov wrote:

Thanks a lot!

In a local vendor’s catalogue i came across an Ultra 2 SCSI controller
card with Symbios SYM53C895 chipset. (Btw, its Advantech
MIC-3640 cPCI card (i need cPCI)). They promise 80MB/s.
It seems that Fsys.sym8scsi driver supports that chipset.
A question for everybody: Can you say anything good/bad from your
experience
about this card/chipset/driver?

We have had no problems with this chip except that for some reason we
need to be very careful how we access the tape drives. It’s possible to
make them work OK, but if the first command to a drive after booting is
“tape bot”, for example, the tape can become inaccessible until the
driver is restarted. This holds true for 895 and 896 chips on addon
cards and on HP Netserver MBs. The disks work fine.

Richard

Dmitri Ivanov wrote:

In my application i need to store approximately 10 Mbytes of data
every second. What is the best solution to achive that?

That is probably beyond repIO capabilities. Is SCSI an option for you?
That should provide 20-30MB/sec throughput. Or you could contact your
customer/sales representative about a potential EIDE solution.


And i have a strong suspicion that Fsys.eide doesn’t
take advantage of UltraDMA modes.

No, Fsys.eide does not officially support any DMA modes (PIO only).
\

In a local vendor’s catalogue i came across an Ultra 2 SCSI controller
card with Symbios SYM53C895 chipset. (Btw, its Advantech
MIC-3640 cPCI card (i need cPCI)). They promise 80MB/s.

Well that’s easier said then done, (what ever the OS). SCSI transfert
rate
is one thing, but then you have LOTS of other factor to consider. CPU
speed, RAM speep, DISK speed, Filesystem caracteristic, disk
fragmentation,
etc. IHMO 80 Mb/s is impossible under any OS mostly when it’s write
speed…

Well, actually I need only 10MB/s. I don’t think things can be
so bad? :wink:

Dmitri Ivanov wrote:

Thanks a lot!

In a local vendor’s catalogue i came across an Ultra 2 SCSI controller
card with Symbios SYM53C895 chipset. (Btw, its Advantech
MIC-3640 cPCI card (i need cPCI)). They promise 80MB/s.

The limiting factor here is how fast the disk can actually write the
bits - sorry I don’t remember what that specification was called.
A couple of years ago I managed nearly 10Mb/sec with a P2/233 and an
Adaptec 2940 Ultra card and a 10K rpm Cheetah drive that said it could
write 15Mb/sec to the disk. Mind you I had to use some undocumented
pass-thru features to bypass almost everything the AHA driver wanted to
do for/to me… And I dedicated that drive to receiving my data… no
filesystem… might have had a partition table, but it was up to me to
avoid clobbering the other partitions!
But Ultra 2 should do it. How much other PCI bus activity is there?
Oh and don’t forget the fan for the disk drive! They do get hot at 10K
and faster.

Phil Olynyk
OBT Software Corp.

It seems that Fsys.sym8scsi driver supports that chipset.
A question for everybody: Can you say anything good/bad from your experience
about this card/chipset/driver?

Dmitri Ivanov <> ivdal@yahoo.com> > wrote:
In my application i need to store approximately 10 Mbytes of data
every second. What is the best solution to achive that?

That is probably beyond repIO capabilities. Is SCSI an option for you?
That should provide 20-30MB/sec throughput. Or you could contact your
customer/sales representative about a potential EIDE solution.

And i have a strong suspicion that Fsys.eide doesn’t
take advantage of UltraDMA modes.



No, Fsys.eide does not officially support any DMA modes (PIO only).
Hmmmm, you say “officially”… Is there hope for the future, then?

:slight_smile:


\

The limiting factor here is how fast the disk can actually write the
bits - sorry I don’t remember what that specification was called.

What are typical hd rates for Ultra 2 SCSI?

A couple of years ago I managed nearly 10Mb/sec with a P2/233 and an
Adaptec 2940 Ultra card and a 10K rpm Cheetah drive that said it could
write 15Mb/sec to the disk. Mind you I had to use some undocumented
pass-thru features to bypass almost everything the AHA driver wanted to
do for/to me… And I dedicated that drive to receiving my data… no
filesystem… might have had a partition table, but it was up to me to
avoid clobbering the other partitions!

Actually, I would like to avoid any programming on my side with this task, I
just
want to open() a file and write() data periodically. Let alone any driver
overrides :wink:
Is that possible?

But Ultra 2 should do it. How much other PCI bus activity is there?

Well, before I can write 10M/s I have to get that data from
a DSP card (based on Texas Instruments TMS320C67x).
I guess this is the major activity. So I have about (10+10)M/s,
which should be nothing compared to 132M/s troughput
that PCI bus is capable of.

Oh and don’t forget the fan for the disk drive! They do get hot at 10K
and faster.

10K ???

Phil Olynyk
OBT Software Corp.

Thanks a lot!

Dmitri Ivanov wrote:

The limiting factor here is how fast the disk can actually write the
bits - sorry I don’t remember what that specification was called.

What are typical hd rates for Ultra 2 SCSI?

Check your favourite manufacturers web page :slight_smile: Seagate doesn’t admit
to any Ultra 2 drives,
but the ST318418N Barracuda 36ES2 is a 50 pin 18G drive which gives
better than 25 MBytes/sec
Formatted Internal Transfer rate, only 20 MBytes/sec External because of
the narrow bus.
The Ultra 160 ST31806LW Cheetah 36ES gives 49 MBytes/sec Internal, and
160 MBytes/sec external.

A couple of years ago I managed nearly 10Mb/sec with a P2/233 and an
Adaptec 2940 Ultra card and a 10K rpm Cheetah drive that said it could
write 15Mb/sec to the disk. Mind you I had to use some undocumented
pass-thru features to bypass almost everything the AHA driver wanted to
do for/to me… And I dedicated that drive to receiving my data… no
filesystem… might have had a partition table, but it was up to me to
avoid clobbering the other partitions!

Actually, I would like to avoid any programming on my side with this task, I
just
want to open() a file and write() data periodically. Let alone any driver
overrides > :wink:
Is that possible?

I doubt it. Perhaps Bill Flowers or one of the QNX filesystem gurus
could comment. I couldn’t get over
3 or 4 MBytes/sec sustained, even using the blockwrite function.
On the other hand, one could write functions that look like open and
write and close (more or less)
that used the passthru capability. More or less, a high speed library.

But Ultra 2 should do it. How much other PCI bus activity is there?

Well, before I can write 10M/s I have to get that data from
a DSP card (based on Texas Instruments TMS320C67x).
I guess this is the major activity. So I have about (10+10)M/s,
which should be nothing compared to 132M/s troughput
that PCI bus is capable of.

Oh and don’t forget the fan for the disk drive! They do get hot at 10K
and faster.


10K ???
10,000 revolutions per minute. The faster, the hotter > :slight_smile:

Phil Olynyk
OBT Software Corp.

Thanks a lot!

Actually, I would like to avoid any programming on my side with this
task, I
just
want to open() a file and write() data periodically. Let alone any
driver
overrides > :wink:
Is that possible?

I doubt it. Perhaps Bill Flowers or one of the QNX filesystem gurus
could comment. I couldn’t get over
3 or 4 MBytes/sec sustained, even using the blockwrite function.
On the other hand, one could write functions that look like open and
write and close (more or less)
that used the passthru capability. More or less, a high speed library.

Do you think writing to a raw partition can help?

10K ???
10,000 revolutions per minute. The faster, the hotter > :slight_smile:

Ok, i got it. At some appalling moment i thought about 10Kbytes/s :wink:

Thanks again

Dmitri Ivanov wrote:

Actually, I would like to avoid any programming on my side with this
task, I
just
want to open() a file and write() data periodically. Let alone any
driver
overrides > :wink:
Is that possible?

I doubt it. Perhaps Bill Flowers or one of the QNX filesystem gurus
could comment. I couldn’t get over
3 or 4 MBytes/sec sustained, even using the blockwrite function.
On the other hand, one could write functions that look like open and
write and close (more or less)
that used the passthru capability. More or less, a high speed library.


Do you think writing to a raw partition can help?

Sorry for the delay in follow-up…
I had tried using the blockwrite function, writing to a raw partition,
but it didn’t help. It still goes through the driver’s buffering scheme.
The only
thing that got me going was using the raw scsi pass-through function.
This is tricky
because you have to figure out the difference between the virtual buffer
addresses
used by your program, and the actual physical addresses used by your pci
scsi card
for its dma access. Then you can fill your buffer and have the scsi card
write directly
out of that buffer without intermediate copying and handshaking. It can
get pretty
interesting if you mess up, especially if you are reading… into the
wrong memory!
I have some information on the scsi pass-through that used to work with
an older
Adaptec driver. I don’t know if it will work with the driver for your
card, and I don’t
know if I am free to pass it on to you. I can check on the last part, if
your sales contact
can’t help you get scsi pass-through information for your driver.
Of course, if you really don’t want to do any programming on this, I’m
sure that
QSSL software people need work as much as I do… or maybe they are busy
with 6.?

Phil Olynyk
OBT Software Corp.

10K ???
10,000 revolutions per minute. The faster, the hotter > :slight_smile:


Ok, i got it. At some appalling moment i thought about 10Kbytes/s > :wink:

Thanks again

Do you think writing to a raw partition can help?

Sorry for the delay in follow-up…
I had tried using the blockwrite function, writing to a raw partition,
but it didn’t help. It still goes through the driver’s buffering scheme.
The only
thing that got me going was using the raw scsi pass-through function.
This is tricky
because you have to figure out the difference between the virtual buffer
addresses
used by your program, and the actual physical addresses used by your pci
scsi card
for its dma access. Then you can fill your buffer and have the scsi card
write directly
out of that buffer without intermediate copying and handshaking. It can
get pretty
interesting if you mess up, especially if you are reading… into the
wrong memory!
I have some information on the scsi pass-through that used to work with
an older
Adaptec driver. I don’t know if it will work with the driver for your
card, and I don’t
know if I am free to pass it on to you. I can check on the last part, if
your sales contact
can’t help you get scsi pass-through information for your driver.

Thanks a lot. I really appreciate your help.
I’ll see what I can do about my sales contact. Actually they’ve just
faxed me another offer. It’s PEP CP365 Ultra160 SCSI adapter (i didn’t
check the chipset yet). As it turned out it takes more than just picking up
a SCSI controller - since SCSI disks need power and fanning I’ll need
a box for it, possibly with it’s own bus. I guess I need a little time
(and maybe some web links as well) to learn about those SCSI things
a bit more…

Of course, if you really don’t want to do any programming on this, I’m
sure that
QSSL software people need work as much as I do… or maybe they are busy
with 6.?

That is not about not wanting to do my job. :wink:
The reason behind it is that this task is considered to be a simple
extention feature.
I mean neither I (in terms of work hours) nor my program (in terms of CPU
consumption) are given much time for it. The program has to
suck some data from DSP cards, process it and pass to the operator
workstation, meanwhile storing the archive on the disk.
That’s why I wanted to avoid any complication, possibly at the expense
of more expensive device.