Erratic interrupt latency

Hi Ken;
can you tap into the smi or smi-act pin on the cpu? Make sure that legacy mode
usb support is turned off as well.

With legacy mode usb support turned on in the bios every now and then the hardware gets yanked out from under our feet! The way legacy mode usb seem to be implimented is that the system routinely enters system management mode. We have no way to delay or even know if that has happened at the os level.


Previously, Ken Schumm wrote in qdn.public.qnx4:
{ We’re having some trouble getting consistent interrupt latency on
{ a QNX 4.25 embedded PC/104 product of ours. Usually the latency is
{ around 10-15 us as measured on a dso. Sometimes, though, maybe
{ .001% of the time, this latency stretches out to as much as 120 us.
{ We need a reliable 4 kHz collection rate, so these 120 us
{ disturbances often cause data to be lost.
{
{ Our product is based around an Ampro 4DXi board (486/133), with an
{ Ampro MiniModule/LCD II vga card and a custom data acquisition card
{ of our own design.
{
{ I think we have eliminated photon and the graphics subsystem
{ from consideration. We removed the video card and ran the system
{ without photon and our gui software with no improvement.
{
{ We thought perhaps that the cpu was being put into low power mode
{ by the idle process, so we wrote our own idle, that spins at
{ priority one. This made no difference.
{
{ Our data collection process is running at priority 29, and proc
{ at 28. We have rotated the data collection interrupt, using the
{ -i 7 option to proc, to make it the highest priority interrupt
{ in the box. We tried both -zn and -zN options to proc to no
{ avail.
{
{ We have also tried killing off most everything except the
{ main data collection process (supervisor) and this made no
{ difference.
{
{ Ideas are appreciated, sin info is below.
{
{ --------------------- sin ------------------------------------
{ SID PID PROGRAM PRI STATE BLK CODE DATA
{ – – Microkernel — ----- — 10448 0
{ 0 1 sys/Proc32 28f READY — 118k 811k
{ 0 2 sys/Slib32 10r RECV 0 53k 4096
{ 0 4 /bin/Fsys 10r RECV 0 77k 409k
{ 0 5 usr/local/tffs/Fsys.doc 10r RECV 0 36k 53k
{ 0 8 /bin/Dev32 24f RECV 0 32k 90k
{ 0 9 /bin/Dev32.ansi 20r RECV 0 40k 49k
{ 0 10 idle 0r READY — 0 0
{ 0 19 //2/bin/nameloc 20o RECV 0 6144 20k
{ 0 20 //2/bin/nameloc 20o REPLY 0 6144 16k
{ 0 24 //2/minya/bin/speaker 10o RECV 0 4096 12k
{ 0 27 //2/minya/bin/dbmgr 10o RECV 0 20k 28k
{ 0 28 //2//bin/supervisor 29f RECV 0 53k 24k
{ 0 35 //2/
/photon/bin/Photon 10r READY — 57k 28k
{ 0 37 //2//vesabios.ms 10o RECV 0 40k 28k
{ 0 39 //2/
/drivers/Pg.vga4 12r REPLY 35 118k 118k
{ 0 42 //2//bin/phfontall 10r RECV 0 159k 442k
{ 0 50 //2/
/bin/Input.mdi8000 12o RECV 0 26k 28k
{ 0 51 //2/minya/bin/gui 10o READY — 122k 114k
{ 0 54 //2//bin/Input.mdi8000 10o RECV 0 26k 28k
{ 0 58 //2/bin/Dev32.ser 20r RECV 0 16k 32k
{ 0 64 //2/bin/tinit 10o WAIT -1 16k 28k
{ 0 66 //2/minya/bin/hostif 10o REPLY 8 57k 32k
{ 0 70 //2/bin/dumper 28o RECV 0 16k 20k
{ 1 71 //2/bin/login 10o REPLY 8 24k 20k
{ 2 72 //2/bin/sh 10o WAIT -1 94k 45k
{ 2 84 //2/bin/sin 10o REPLY 1 45k 49k
{
{ -------------------- sin ver ---------------------------------
{ PROGRAM NAME VERSION DATE
{ sys/Proc32 Proc 4.25I Nov 25 1998
{ sys/Proc32 Slib16 4.23G Oct 04 1996
{ sys/Slib32 Slib32 4.24B Aug 12 1997
{ /bin/Fsys Fsys32 4.24S Jul 16 1998
{ /bin/Fsys Floppy 4.24B Aug 19 1997
{ /bin/Fsys.aha7scsi scsi 4.24M Mar 23 1998
{ //1/bin/Dev32 Dev32 4.23G Oct 04 1996
{ //1/bin/Dev32.ansi Dev32.ansi 4.23H Nov 21 1996
{ //1/bin/Dev32.ser Dev32.ser 4.23I Jun 27 1997
{ //1/bin/Dev32.par Dev32.par 4.23G Oct 04 1996
{ //1/bin/Dev32.pty Dev32.pty 4.23G Oct 04 1996
{ //1/bin/Mouse Mouse 4.24A Aug 22 1997
{ //1/bin/Iso9660fsys Iso9660fsys 4.23B Jun 10 1998
{ //1/bin/Pipe Pipe 4.23A Feb 26 1996
{ //1/bin/Net Net 4.25B Jul 27 1998
{ //1/bin/Net.ether1000 Net.ether100 4.24B Jul 24 1998
{ //1/usr/bin/lpsrvr lpsrvr 4.24A Jun 26 1997
{ //1/
/bin/phfontpfr Photon Font 1.13A Jul 07 1998
{ //1//usr/ucb/Socklet Socklet 4.25G Dec 08 1998
{ //1/
/photon/bin/Photon Photon 1.13D Sep 03 1998
{ //1//4.25/usr/ucb/Dns Dns 1.00C Jun 30 1998
{
{ ------------------ sin info -----------------------------------
{ Node CPU Machine Speed Memory Ticksize Display Flags
{ 2 486/487 AT 9708 4161k/7962k 10.0ms VGA Color -3-±---------8P
{
{ Heapp Heapf Heapl Heapn Hands Names Sessions Procs Timers Nodes Virtual
{ 0 0 23256 0 64 100 64 500 125 4 6M/ 25M
{
{ Boot from Hard at — – --:-- Locators: 2
{
{ ------------------ sin irq ------------------------------------
{ IRQ PID PROGRAM CS:IP DS
{ -1 8 /bin/Dev32 0005:005760 000D
{ -1 9 /bin/Dev32.ansi 0005:005DC0 000D
{ -1 58 //2/bin/Dev32.ser 0005:0024FC 000D
{ 0 1 sys/Proc32 00F0:004CB3 00F8
{ 1 9 /bin/Dev32.ansi 0005:00690C 000D
{ 3 58 //2/bin/Dev32.ser 0005:00177C 000D
{ 4 58 //2/bin/Dev32.ser 0005:0017A4 000D
{ 7 28 //2/
/bin/supervisor 0005:00AE70 000D
{ 13 1 sys/Proc32 00F0:004C77 00F8
{
{
{
{
{
{


Pat Ford email: pford@qnx.com
QNX Software Systems, Ltd. WWW: http://www.qnx.com
(613) 591-0931 (voice) mail: 175 Terence Matthews
(613) 591-3579 (fax) Kanata, Ontario, Canada K2M 1W8

“Armin Steinhoff” <A-Steinhoff@web_.de> wrote in message
news:3AAFD755.891E0829@web_.de…

… and at the end: the AMD CPU at Ken’s PC/104 board has SMM > :slight_smile:


http://www.amd.com/products/epd/processors/6.32bitproc/11.am486fami/12.am486

dx/index.html

Armin

Is there any way to disable SMM? Some bit in control register or sth
similiar?

Pavol Kycina

Previously, Armin Steinhoff wrote in qdn.public.qnx4:

… and at the end: the AMD CPU at Ken’s PC/104 board has SMM > :slight_smile:

http://www.amd.com/products/epd/processors/6.32bitproc/11.am486fami/12.am486dx/index.html

Armin

Thanks, Armin! Ampro told me their board didn’t use SMM. I’ll follow up
on it.

Hi Pat,

This board doesn’t have any usb support, so I don’t think that’s the
problem.

I’ll see about tapping into the smi/smi-act pins on the cpu, but they
are REALLY tiny. I need to get a pin-point soldering iron and a
magnifying glass and try to solder a small lead on it to clip on to.

Thanks,
Ken

Previously, Pat Ford wrote in qdn.public.qnx4:

Hi Ken;
can you tap into the smi or smi-act pin on the cpu? Make sure that legacy mode
usb support is turned off as well.

With legacy mode usb support turned on in the bios every now and then the hardware gets yanked out from under our feet! The way legacy mode usb seem to be implimented is that the system routinely enters system management mode. We have no way to delay or even know if that has happened at the os level.


Previously, Ken Schumm wrote in qdn.public.qnx4:
{ We’re having some trouble getting consistent interrupt latency on
{ a QNX 4.25 embedded PC/104 product of ours. Usually the latency is
{ around 10-15 us as measured on a dso. Sometimes, though, maybe
{ .001% of the time, this latency stretches out to as much as 120 us.
{ We need a reliable 4 kHz collection rate, so these 120 us
{ disturbances often cause data to be lost.
{
{ Our product is based around an Ampro 4DXi board (486/133), with an
{ Ampro MiniModule/LCD II vga card and a custom data acquisition card
{ of our own design.
{
{ I think we have eliminated photon and the graphics subsystem
{ from consideration. We removed the video card and ran the system
{ without photon and our gui software with no improvement.
{
{ We thought perhaps that the cpu was being put into low power mode
{ by the idle process, so we wrote our own idle, that spins at
{ priority one. This made no difference.
{
{ Our data collection process is running at priority 29, and proc
{ at 28. We have rotated the data collection interrupt, using the
{ -i 7 option to proc, to make it the highest priority interrupt
{ in the box. We tried both -zn and -zN options to proc to no
{ avail.
{
{ We have also tried killing off most everything except the
{ main data collection process (supervisor) and this made no
{ difference.
{
{ Ideas are appreciated, sin info is below.
{
{ --------------------- sin ------------------------------------
{ SID PID PROGRAM PRI STATE BLK CODE DATA
{ – – Microkernel — ----- — 10448 0
{ 0 1 sys/Proc32 28f READY — 118k 811k
{ 0 2 sys/Slib32 10r RECV 0 53k 4096
{ 0 4 /bin/Fsys 10r RECV 0 77k 409k
{ 0 5 usr/local/tffs/Fsys.doc 10r RECV 0 36k 53k
{ 0 8 /bin/Dev32 24f RECV 0 32k 90k
{ 0 9 /bin/Dev32.ansi 20r RECV 0 40k 49k
{ 0 10 idle 0r READY — 0 0
{ 0 19 //2/bin/nameloc 20o RECV 0 6144 20k
{ 0 20 //2/bin/nameloc 20o REPLY 0 6144 16k
{ 0 24 //2/minya/bin/speaker 10o RECV 0 4096 12k
{ 0 27 //2/minya/bin/dbmgr 10o RECV 0 20k 28k
{ 0 28 //2//bin/supervisor 29f RECV 0 53k 24k
{ 0 35 //2/
/photon/bin/Photon 10r READY — 57k 28k
{ 0 37 //2//vesabios.ms 10o RECV 0 40k 28k
{ 0 39 //2/
/drivers/Pg.vga4 12r REPLY 35 118k 118k
{ 0 42 //2//bin/phfontall 10r RECV 0 159k 442k
{ 0 50 //2/
/bin/Input.mdi8000 12o RECV 0 26k 28k
{ 0 51 //2/minya/bin/gui 10o READY — 122k 114k
{ 0 54 //2//bin/Input.mdi8000 10o RECV 0 26k 28k
{ 0 58 //2/bin/Dev32.ser 20r RECV 0 16k 32k
{ 0 64 //2/bin/tinit 10o WAIT -1 16k 28k
{ 0 66 //2/minya/bin/hostif 10o REPLY 8 57k 32k
{ 0 70 //2/bin/dumper 28o RECV 0 16k 20k
{ 1 71 //2/bin/login 10o REPLY 8 24k 20k
{ 2 72 //2/bin/sh 10o WAIT -1 94k 45k
{ 2 84 //2/bin/sin 10o REPLY 1 45k 49k
{
{ -------------------- sin ver ---------------------------------
{ PROGRAM NAME VERSION DATE
{ sys/Proc32 Proc 4.25I Nov 25 1998
{ sys/Proc32 Slib16 4.23G Oct 04 1996
{ sys/Slib32 Slib32 4.24B Aug 12 1997
{ /bin/Fsys Fsys32 4.24S Jul 16 1998
{ /bin/Fsys Floppy 4.24B Aug 19 1997
{ /bin/Fsys.aha7scsi scsi 4.24M Mar 23 1998
{ //1/bin/Dev32 Dev32 4.23G Oct 04 1996
{ //1/bin/Dev32.ansi Dev32.ansi 4.23H Nov 21 1996
{ //1/bin/Dev32.ser Dev32.ser 4.23I Jun 27 1997
{ //1/bin/Dev32.par Dev32.par 4.23G Oct 04 1996
{ //1/bin/Dev32.pty Dev32.pty 4.23G Oct 04 1996
{ //1/bin/Mouse Mouse 4.24A Aug 22 1997
{ //1/bin/Iso9660fsys Iso9660fsys 4.23B Jun 10 1998
{ //1/bin/Pipe Pipe 4.23A Feb 26 1996
{ //1/bin/Net Net 4.25B Jul 27 1998
{ //1/bin/Net.ether1000 Net.ether100 4.24B Jul 24 1998
{ //1/usr/bin/lpsrvr lpsrvr 4.24A Jun 26 1997
{ //1/
/bin/phfontpfr Photon Font 1.13A Jul 07 1998
{ //1//usr/ucb/Socklet Socklet 4.25G Dec 08 1998
{ //1/
/photon/bin/Photon Photon 1.13D Sep 03 1998
{ //1//4.25/usr/ucb/Dns Dns 1.00C Jun 30 1998
{
{ ------------------ sin info -----------------------------------
{ Node CPU Machine Speed Memory Ticksize Display Flags
{ 2 486/487 AT 9708 4161k/7962k 10.0ms VGA Color -3-±---------8P
{
{ Heapp Heapf Heapl Heapn Hands Names Sessions Procs Timers Nodes Virtual
{ 0 0 23256 0 64 100 64 500 125 4 6M/ 25M
{
{ Boot from Hard at — – --:-- Locators: 2
{
{ ------------------ sin irq ------------------------------------
{ IRQ PID PROGRAM CS:IP DS
{ -1 8 /bin/Dev32 0005:005760 000D
{ -1 9 /bin/Dev32.ansi 0005:005DC0 000D
{ -1 58 //2/bin/Dev32.ser 0005:0024FC 000D
{ 0 1 sys/Proc32 00F0:004CB3 00F8
{ 1 9 /bin/Dev32.ansi 0005:00690C 000D
{ 3 58 //2/bin/Dev32.ser 0005:00177C 000D
{ 4 58 //2/bin/Dev32.ser 0005:0017A4 000D
{ 7 28 //2/
/bin/supervisor 0005:00AE70 000D
{ 13 1 sys/Proc32 00F0:004C77 00F8
{
{
{
{
{
{


Pat Ford email: > pford@qnx.com
QNX Software Systems, Ltd. WWW: > http://www.qnx.com
(613) 591-0931 (voice) mail: 175 Terence Matthews
(613) 591-3579 (fax) Kanata, Ontario, Canada K2M 1W8

Thats what I did here, don’t have any caffeine before trying 8*()

Previously, Ken Schumm wrote in qdn.public.qnx4:
{ Hi Pat,
{
{ This board doesn’t have any usb support, so I don’t think that’s the
{ problem.

doesn’t mean that the core bios wasn’t used in a usb board

{
{ I’ll see about tapping into the smi/smi-act pins on the cpu, but they
{ are REALLY tiny. I need to get a pin-point soldering iron and a
{ magnifying glass and try to solder a small lead on it to clip on to.
{
{ Thanks,
{ Ken
{
{ Previously, Pat Ford wrote in qdn.public.qnx4:
{ > Hi Ken;
{ > can you tap into the smi or smi-act pin on the cpu? Make sure that legacy mode
{ > usb support is turned off as well.
{ >

\

Pat Ford email: pford@qnx.com
QNX Software Systems, Ltd. WWW: http://www.qnx.com
(613) 591-0931 (voice) mail: 175 Terence Matthews
(613) 591-3579 (fax) Kanata, Ontario, Canada K2M 1W8

Previously, Pat Ford wrote in qdn.public.qnx4:

Thats what I did here, don’t have any caffeine before trying 8*()

OK. I’ll have a few beers instead :slight_smile:

Previously, Ken Schumm wrote in qdn.public.qnx4:
{ Hi Pat,
{
{ This board doesn’t have any usb support, so I don’t think that’s the
{ problem.

doesn’t mean that the core bios wasn’t used in a usb board

Yes, but there is no option to turn off legacy mode USB support :frowning:
So if it’s there, we’re screwed and we’ll have to look at other
vendors.

{
{ I’ll see about tapping into the smi/smi-act pins on the cpu, but they
{ are REALLY tiny. I need to get a pin-point soldering iron and a
{ magnifying glass and try to solder a small lead on it to clip on to.
{
{ Thanks,
{ Ken
{
{ Previously, Pat Ford wrote in qdn.public.qnx4:
{ > Hi Ken;
{ > can you tap into the smi or smi-act pin on the cpu? Make sure that legacy mode
{ > usb support is turned off as well.
{

\

Pat Ford email: > pford@qnx.com
QNX Software Systems, Ltd. WWW: > http://www.qnx.com
(613) 591-0931 (voice) mail: 175 Terence Matthews
(613) 591-3579 (fax) Kanata, Ontario, Canada K2M 1W8

Previously, Pat Ford wrote in qdn.public.qnx4:

Hi Ken;
can you tap into the smi or smi-act pin on the cpu? Make sure that legacy mode
usb support is turned off as well.

With legacy mode usb support turned on in the bios every now and then the
hardware gets yanked out from under our feet! The way legacy mode usb seem
to be implimented is that the system routinely enters system management mode.
We have no way to delay or even know if that has happened at the os level.

Ampro reports that they make absolutely no use of SMI on their CM/4DXe
board. We have confirmed this by monitoring SMI and SMIACT with a scope.

We are still looking into legacy mode USB.

Ken Schumm <kwschumm@qsolv.com> wrote:

Previously, Pat Ford wrote in qdn.public.qnx4:
Hi Ken;
can you tap into the smi or smi-act pin on the cpu? Make sure that legacy mode
usb support is turned off as well.

With legacy mode usb support turned on in the bios every now and then the
hardware gets yanked out from under our feet! The way legacy mode usb seem
to be implimented is that the system routinely enters system management mode.
We have no way to delay or even know if that has happened at the os level.

Ampro reports that they make absolutely no use of SMI on their CM/4DXe
board. We have confirmed this by monitoring SMI and SMIACT with a scope.

We are still looking into legacy mode USB.

Hmmm… it doesn’t have SMM, but it does have legacy USB support?

How does that work?

legacy USB keyboard and mouse support is an awfully neat trick, but it’s
pretty ugly. It relies on the host computer running a USB pseudo stack
to get info from USB peripherals, then it jams them into the backside
of the PS/2 controller using a mechanism built into PS/2 hardware.

This `backside jamming’ is usually done by some BIOS code that is
executed in SMM.

It’s all supposed to be transparent to the OS, but good luck.

HWSUPPORT - Please FAQ this under SMM and under `legacy USB’

Previously, pete@spam-yourself.com wrote in qdn.public.qnx4:

Ken Schumm <> kwschumm@qsolv.com> > wrote:
Previously, Pat Ford wrote in qdn.public.qnx4:
Hi Ken;
can you tap into the smi or smi-act pin on the cpu? Make sure that legacy mode
usb support is turned off as well.

With legacy mode usb support turned on in the bios every now and then the
hardware gets yanked out from under our feet! The way legacy mode usb seem
to be implimented is that the system routinely enters system management mode.
We have no way to delay or even know if that has happened at the os level.

Ampro reports that they make absolutely no use of SMI on their CM/4DXe
board. We have confirmed this by monitoring SMI and SMIACT with a scope.

We are still looking into legacy mode USB.

Hmmm… it doesn’t have SMM, but it does have legacy USB support?

How does that work?

It doesn’t. According to Ampro, there is no SMM or USB support at
all in the hardware or BIOS.

We are still having the original problem, but have gotten around it
temporarily by implementing a circular buffer which the ISR loads data
into. This lets us “ride out” the excess latency periods. We eventually
need to get to the bottom of this problem, but it is working for now.

pete@spam-yourself.com wrote:

How does that work?

legacy USB keyboard and mouse support is an awfully neat trick, but it’s
pretty ugly. It relies on the host computer running a USB pseudo stack
to get info from USB peripherals, then it jams them into the backside
of the PS/2 controller using a mechanism built into PS/2 hardware.

This `backside jamming’ is usually done by some BIOS code that is
executed in SMM.

It’s all supposed to be transparent to the OS, but good luck.

HWSUPPORT - Please FAQ this under SMM and under `legacy USB’

We had an interesting case recently with this issue.
A QNX4 program would often run reply blocked indefinitely
on Dev for no apparent good reason.
Customer reduced the problem to:
loop forever
write to /dev/null
end loop

This little program would hang without fail after some hours or
days. The board in question uses the Pentium 233 MMX CPU.
(Don’t know if that has SMM or not?)
It appears that the program was still waiting for Dev to respond,
and Dev thought it had already responded. Dead lock.

After some months of agony both sides of the Atlantic (thanks &
kudos to QSSL International FAE), this was finally traced
to… Keyboard support being enabled in the BIOS setup.
After disabling this, the problem was solved.

I’m not sure what can be done to solve this and prvent it in
future versions including RTP, but it seems that OS design
has to take extra special care these days with these “features”.

Between BIOS & CPU, a lot is being done to usurp functions
previously performed either by real hardware or the actual
OS, and being done in a way that’s not all that transparent
to upper layer software after all…

After some months of agony both sides of the Atlantic (thanks &
kudos to QSSL International FAE), this was finally traced
to… Keyboard support being enabled in the BIOS setup.
After disabling this, the problem was solved.

What do you mean by Keyboard support being enabled in the BIOS ? Do you
mean USB keyboard legacy support ? Also, I am completely at a loss to
understand how this could cause Dev to “think that it had replied”; I
would be very interested in hearing more about this.

I’m not sure what can be done to solve this and prvent it in
future versions including RTP, but it seems that OS design
has to take extra special care these days with these “features”.

If this is USB keyboard legacy support, then there is nothing the OS
can do to work around the latencies induced by these features (it is
impossible for the OS to detect what is going on - which is precisely
the point of the operation).

Between BIOS & CPU, a lot is being done to usurp functions
previously performed either by real hardware or the actual
OS, and being done in a way that’s not all that transparent
to upper layer software after all…

If this is USB keyboard legacy emulation, then it really should be
totally transparent (outside the temporal domain), hence my stated
confusion above.

Rennie Allen wrote:

After some months of agony both sides of the Atlantic (thanks &
kudos to QSSL International FAE), this was finally traced
to… Keyboard support being enabled in the BIOS setup.
After disabling this, the problem was solved.

What do you mean by Keyboard support being enabled in the BIOS ? Do you
mean USB keyboard legacy support ? Also, I am completely at a loss to
understand how this could cause Dev to “think that it had replied”; I
would be very interested in hearing more about this.

I’m not sure what can be done to solve this and prvent it in
future versions including RTP, but it seems that OS design
has to take extra special care these days with these “features”.

If this is USB keyboard legacy support, then there is nothing the OS
can do to work around the latencies induced by these features (it is
impossible for the OS to detect what is going on - which is precisely
the point of the operation).

Between BIOS & CPU, a lot is being done to usurp functions
previously performed either by real hardware or the actual
OS, and being done in a way that’s not all that transparent
to upper layer software after all…

If this is USB keyboard legacy emulation, then it really should be
totally transparent (outside the temporal domain), hence my stated
confusion above.

I don’t have the exact wording of the BIOS-I suppose it is as
you say, legacy USB keyboard.

I would like to know too how this can cause this sequence of events.
The fact that the simple application is reply blocked on Dev,
leads to the deduction that the message between them was missed.
To trace it will take access to the source, so we have no way
of discovering the mechanism 100%.

I can tell you it was totally unexpected
Fact is, enable it, and the situation recurs rapidly.
Disable, and the application runs indefinitely.
So this feature is not transparent to the OS.