mouse interrupt causes disruption of real-time robot control

QNX Newsgroups (Archives) qnx.ddk
1

Hello,

currently, I am trying to stabilize system behavior within a high-speed
robot control done on a standard PC under QNX Neutrino (currently v6.2.). It
seems to be connected with sporadic mouse movements and window refreshments
on the photon screen. Here is the scene:

Upon reception of a cyclic, hardware-generated interrupt (exactly every 1 ms
from the communication system), some calculations are made in a high
priority thread in order to set the contents of an outgoing data telegram.
After composition, the data telegram is broadcasted on the communication
bus. The external components receiving the broadcast, each evaluate the data
and send back an answer telegram. For some components, it is necessary that
the time intervals between the incoming data telegrams are always the same
(there is a small tolerable jitter).
When I move the mouse pointer on the PC the open windows on the photon
screen are activated and deactivated. This internally leads to the behaviour
that the time between reception of an interrupt and the actual appearance of
the data telegram on the communication bus varies so much, that the
sensitive external components run out of synchronisation.

Here are some questions:

A. Is it possible to set the priorities of these automatic mouse and screen
manipulations to a low priority, so that the control procedure is not
disturbed anymore?

B. Is it possible to switch off mouse and keyboard activities by software?

C. Are there maybe some other hidden places for configuration of these
effects except the BIOS settings? In the BIOS I have already disabled
everything that looked like having something to do with mouse and screen
interrupts.

I would be happy if someone could help me with this.

Thank you.

Nnamdi

2

Hi…

Some pointers…

  1. How about if you use an embedded pc104 or any other type of single
    board computer (SBC) form factor to act as, well, an embedded system? In
    this system you can disable the mouse and other such things, and like
    things, etc. Also, if an embedded system is not an option, you may
    want to add a DAQ board to your current system that has real-time
    timers. You can then use the timer to trigger 1000 Hz interrupts very
    accurately, and in this case, a mouse movement may not be an issue.

  2. When you service an interrupt every 1ms, there is not much more that
    you can do in that computer other than service the 1ms interrupt
    (depending on how fast is your CPU of course). You can generate a strobe
    signal from the 1ms interrupt cycle to see how much time the interrupt
    service actually takes. Then you can see how much time is left to
    service the mouse, or for that matter, to do any thing else.

  3. If you try, you may find that when servicing interrupts at 1000 Hz,
    an embedded system with 150 MHz CPU may do better than a 2 GHz system
    with a mouse attached to it.

  4. (on the side) why do you need to have 1000 hz loop to stabilize a
    robot? Is this a land robot? Are you stabilizing a subsystem within your
    robot? Even aerial robots operate very well at 400 Hz and less. I have
    not seen many robotic systems that need 1000 Hz stabilization loop,
    aerial or otherwise. Any way, if you need 1000 Hz to stabilize any
    thing, I would take the mouse away from this system just the same (go
    back to point 1).


    Regards…

Miguel.


Nnamdi Kohn wrote:

Hello,

currently, I am trying to stabilize system behavior within a high-speed
robot control done on a standard PC under QNX Neutrino (currently v6.2.). It
seems to be connected with sporadic mouse movements and window refreshments
on the photon screen. Here is the scene:

Upon reception of a cyclic, hardware-generated interrupt (exactly every 1 ms
from the communication system), some calculations are made in a high
priority thread in order to set the contents of an outgoing data telegram.
After composition, the data telegram is broadcasted on the communication
bus. The external components receiving the broadcast, each evaluate the data
and send back an answer telegram. For some components, it is necessary that
the time intervals between the incoming data telegrams are always the same
(there is a small tolerable jitter).
When I move the mouse pointer on the PC the open windows on the photon
screen are activated and deactivated. This internally leads to the behaviour
that the time between reception of an interrupt and the actual appearance of
the data telegram on the communication bus varies so much, that the
sensitive external components run out of synchronisation.

Here are some questions:

A. Is it possible to set the priorities of these automatic mouse and screen
manipulations to a low priority, so that the control procedure is not
disturbed anymore?

B. Is it possible to switch off mouse and keyboard activities by software?

C. Are there maybe some other hidden places for configuration of these
effects except the BIOS settings? In the BIOS I have already disabled
everything that looked like having something to do with mouse and screen
interrupts.

I would be happy if someone could help me with this.

Thank you.

Nnamdi

3

Nnamdi Kohn wrote:

When I move the mouse pointer on the PC the open windows on the photon
screen are activated and deactivated. This internally leads to the behaviour
that the time between reception of an interrupt and the actual appearance of
the data telegram on the communication bus varies so much, that the
sensitive external components run out of synchronisation.

I’ll take a stab and say you’ve got legacy USB support turned on in the BIOS, try disabling that and any powersave like features too.


A. Is it possible to set the priorities of these automatic mouse and screen
manipulations to a low priority, so that the control procedure is not
disturbed anymore?

Not going to help if you are are already at high priority. The only thing more precise is to use InterruptAttach().


B. Is it possible to switch off mouse and keyboard activities by software?

Not a good idea and they really don’t use large amount of CPU anyway.


C. Are there maybe some other hidden places for configuration of these
effects except the BIOS settings? In the BIOS I have already disabled
everything that looked like having something to do with mouse and screen
interrupts.

You do want the mouse to work I presume, after all you do have a GUI to manage.


Evan

4

Sorry if this comes across a little rude but you are dissing something that really isn’t a tough job for modern processors providing the OS doesn’t get in your face.


Miguel Simon wrote:

  1. How about if you use an embedded pc104 or any other type of single
    board computer (SBC) form factor to act as, well, an embedded system? In
    this system you can disable the mouse and other such things, and like

I don’t think Nnamdi wants them disabled permanently.


things, etc. Also, if an embedded system is not an option, you may
want to add a DAQ board to your current system that has real-time
timers. You can then use the timer to trigger 1000 Hz interrupts very
accurately, and in this case, a mouse movement may not be an issue.

I believe that’s pretty much what Nnamdi’s already doing.


  1. When you service an interrupt every 1ms, there is not much more that
    you can do in that computer other than service the 1ms interrupt

Bullshit. There is plenty of time for lots to happen unless the CPU is hitting the hardware too much.


  1. If you try, you may find that when servicing interrupts at 1000 Hz,
    an embedded system with 150 MHz CPU may do better than a 2 GHz system
    with a mouse attached to it.

Please explain? IO port accesses do suck but they suck just as bad on a 150 MHz Pentium as they do on a 2 GHz Athlon64.


  1. (on the side) why do you need to have 1000 hz loop to stabilize a
    robot? Is this a land robot? Are you stabilizing a subsystem within your

It’s certainly handy for tight non-linear profiles.


Evan

5

Evan Hillas wrote:

Hi Evan…

Not that I care or that it matters a single bit, but you are right, you
are indeed rude dude! Relax, take a brake, or do something. Somewhere
you missed the point that any and all comments are meant to help…



Sorry if this comes across a little rude but you are dissing something
that really isn’t a tough job for modern processors providing the OS
doesn’t get in your face.

I am not sure if you are making my point for me…

Miguel Simon wrote:

  1. How about if you use an embedded pc104 or any other type of single
    board computer (SBC) form factor to act as, well, an embedded system?
    In this system you can disable the mouse and other such things, and like



    I don’t think Nnamdi wants them disabled permanently.

Let Nnamdi speak for himself.

things, etc. Also, if an embedded system is not an option, you may
want to add a DAQ board to your current system that has real-time
timers. You can then use the timer to trigger 1000 Hz interrupts very
accurately, and in this case, a mouse movement may not be an issue.


I believe that’s pretty much what Nnamdi’s already doing.

Again, let the original poster speak for himself so that we can help.
Your answers are not friendly, and your post may intimidate the original
poster.

  1. When you service an interrupt every 1ms, there is not much more
    that you can do in that computer other than service the 1ms interrupt


    Bullshit.

Nice… When and how did I offend you so that you answer in this way?


There is plenty of time for lots to happen unless the CPU is

hitting the hardware too much.


3. If you try, you may find that when servicing interrupts at 1000 Hz,
an embedded system with 150 MHz CPU may do better than a 2 GHz system
with a mouse attached to it.


Please explain? IO port accesses do suck but they suck just as bad on a
150 MHz Pentium as they do on a 2 GHz Athlon64.

The original poster is mixing controls with OS, two related, but
separate issues that perhaps need two different answers. That is what I
had in mind and was trying to convey, but now, you are getting in the
way. Nice again.

Do you have experience with embedded systems? Yes, or not? Do not care,
will not waste my time any more.

Regards…

Miguel.

  1. (on the side) why do you need to have 1000 hz loop to stabilize a
    robot? Is this a land robot? Are you stabilizing a subsystem within your



    It’s certainly handy for tight non-linear profiles.


    Evan
6

Things to check

  1. make sure that you don’t have any legacy usb emulation enabled (as noted by others)
  2. make sure that your processing thread is running at a higher priority than all
    other processes (especially the graphics driver), and that it is not blocked on any resources
    held by other processes which could result in priority inversion (such as semaphores)
  3. check to make sure that the priority of your interrupt is higher than that of the mouse/graphics interrupt.
    It sounds like it’s not so much the mouse movements but the graphics activity of bringing the windows
    to the front. You could test this by disabling cursor focus. Do you still have a jitter problem?
  4. if you have momentics PE then I would take a kernel trace and check it out, you will see if any threads
    are preempting you.

Nnamdi Kohn wrote:

Hello,

currently, I am trying to stabilize system behavior within a high-speed
robot control done on a standard PC under QNX Neutrino (currently v6.2.). It
seems to be connected with sporadic mouse movements and window refreshments
on the photon screen. Here is the scene:

Upon reception of a cyclic, hardware-generated interrupt (exactly every 1 ms
from the communication system), some calculations are made in a high
priority thread in order to set the contents of an outgoing data telegram.
After composition, the data telegram is broadcasted on the communication
bus. The external components receiving the broadcast, each evaluate the data
and send back an answer telegram. For some components, it is necessary that
the time intervals between the incoming data telegrams are always the same
(there is a small tolerable jitter).
When I move the mouse pointer on the PC the open windows on the photon
screen are activated and deactivated. This internally leads to the behaviour
that the time between reception of an interrupt and the actual appearance of
the data telegram on the communication bus varies so much, that the
sensitive external components run out of synchronisation.

Here are some questions:

A. Is it possible to set the priorities of these automatic mouse and screen
manipulations to a low priority, so that the control procedure is not
disturbed anymore?

B. Is it possible to switch off mouse and keyboard activities by software?

C. Are there maybe some other hidden places for configuration of these
effects except the BIOS settings? In the BIOS I have already disabled
everything that looked like having something to do with mouse and screen
interrupts.

I would be happy if someone could help me with this.

Thank you.

Nnamdi

7

Hello Evan, thanks for your answer.

When I move the mouse pointer on the PC the open windows on the photon
screen are activated and deactivated. This internally leads to the
behaviour
that the time between reception of an interrupt and the actual appearance
of
the data telegram on the communication bus varies so much, that the
sensitive external components run out of synchronisation.

I’ll take a stab and say you’ve got legacy USB support turned on in the
BIOS, try disabling that and any powersave like features too.

I already disabled all “Integrated Peripherals” in the BIOS setup, except
for “OnChip IDE” channel 0 and 1. And in the “Power Management Setup” there
is nothing enabled, either.

A. Is it possible to set the priorities of these automatic mouse and
screen
manipulations to a low priority, so that the control procedure is not
disturbed anymore?

Not going to help if you are are already at high priority. The only thing
more
precise is to use InterruptAttach().

instead of InterruptAttachEvent()?

B. Is it possible to switch off mouse and keyboard activities by
software?

Not a good idea and they really don’t use large amount of CPU anyway.

It seems to be rather the activations and deactivations of photon windows
(as the mouse hovers over them in cursor focus mode) that steals too much
time between the hardware interrupt and the finish of the respective
interrupt service routine. Not the mouse movements themselves.

C. Are there maybe some other hidden places for configuration of these
effects except the BIOS settings? In the BIOS I have already disabled
everything that looked like having something to do with mouse and screen
interrupts.

You do want the mouse to work I presume, after all you do have a GUI to
manage.

We are thinking of using a separate computer providing the GUI and connected
to the control PC via an Ethernet adapter and sockets. Hopefully, the
interrupt generated from the Ethernet adapter will not disturb the control.

Nnamdi

8

Hi Colin, thanks for your suggestions.

  1. make sure that you don’t have any legacy usb emulation enabled (as
    noted by others)

as far as I can see, there is nothing USB-like enabled in the BIOS, anymore.
There is just the keyboard and a radio mouse, both PS2 connected.

  1. make sure that your processing thread is running at a higher priority
    than all
    other processes (especially the graphics driver), and that it is not
    blocked on any resources
    held by other processes which could result in priority inversion (such
    as semaphores)

I can’t see any other system thread (except for one procnto thread) above
priority 21. The application threads triggered by communication interrupt
all reside between 30 and 35. So, there should be no problem from here.

  1. check to make sure that the priority of your interrupt is higher
    than that of the mouse/graphics interrupt. It sounds like it’s not
    so much the mouse movements but the graphics activity of
    bringing the windows to the front. You could test this by disabling
    cursor focus. Do you still have a jitter problem?

The jitter problem still exists even after disabling cursor focus. Even when
I don’t use the mouse and the keyboard at all, the jitter hits from time to
time. This behavior is sporadic after some seconds of time and thus
unbelievably disturbing in the overall development process.

  1. if you have momentics PE then I would take a kernel trace and check
    it out, you will see if any threads are preempting you.

unfortunately we still use Neutrino v6.2. So there is no tool like the
Tracelogger available for tracing the kernel and user threads. Is there
maybe such an application for 6.2 as well?

Nnamdi

9

Hi Miguel,

  1. How about if you use an embedded pc104 or any other type of single
    board computer (SBC) form factor to act as, well, an embedded system? In
    this system you can disable the mouse and other such things, and like
    things, etc. Also, if an embedded system is not an option, you may want
    to add a DAQ board to your current system that has real-time timers. You
    can then use the timer to trigger 1000 Hz interrupts very accurately, and
    in this case, a mouse movement may not be an issue.

This will not be possible in our current project state anymore. It is a BIG
system with various different application functionalities. We have to use
standard PC hardware to be able to cope with the development overhead.

  1. When you service an interrupt every 1ms, there is not much more that
    you can do in that computer other than service the 1ms interrupt
    (depending on how fast is your CPU of course). You can generate a strobe
    signal from the 1ms interrupt cycle to see how much time the interrupt
    service actually takes. Then you can see how much time is left to service
    the mouse, or for that matter, to do any thing else.

there seems to be enough time for servicing everything necessary. The
critical point for our system is the jitter. If we don’t get along with 1ms
we just go to 10ms. Our system runs fine and there are just the control
parameters to be changed. But still there remains the jitter problem. We
have even tried it with a 1s (one second) cycle.

  1. If you try, you may find that when servicing interrupts at 1000 Hz, an
    embedded system with 150 MHz CPU may do better than a 2 GHz system with a
    mouse attached to it.

I was wondering if there was a systematic way to determine the jitter source
within the system without the necessity of trying different systems. As QNX
is a very good REAL-TIME operating system, there should be a systematic
reason for every kind of behavior, I think.

  1. (on the side) why do you need to have 1000 hz loop to stabilize a
    robot? Is this a land robot? Are you stabilizing a subsystem within your
    robot? Even aerial robots operate very well at 400 Hz and less. I have not
    seen many robotic systems that need 1000 Hz stabilization loop, aerial or
    otherwise. Any way, if you need 1000 Hz to stabilize any thing, I would
    take the mouse away from this system just the same (go back to point 1).

We operate a parallel robot system where cycles of up to 8 kHz (eight kilo
hertz) are common. These are very complex, strong and fast robot systems.
The robot structure is fragile (due to light weight) and the kinematics are
coupled over closed loops. So every slight deviation time or set value might
break the robot structure. We are trying to go the first steps into a new
and demanding robotics area using a central control approach.

Nnamdi

10

Nnamdi Kohn wrote:

Hello,

currently, I am trying to stabilize system behavior within a high-speed
robot control done on a standard PC under QNX Neutrino (currently v6.2.). It
seems to be connected with sporadic mouse movements and window refreshments
on the photon screen. Here is the scene:

Upon reception of a cyclic, hardware-generated interrupt (exactly every 1 ms
from the communication system), some calculations are made in a high
priority thread in order to set the contents of an outgoing data telegram.
After composition, the data telegram is broadcasted on the communication
bus. The external components receiving the broadcast, each evaluate the data
and send back an answer telegram. For some components, it is necessary that
the time intervals between the incoming data telegrams are always the same
(there is a small tolerable jitter).
When I move the mouse pointer on the PC the open windows on the photon
screen are activated and deactivated. This internally leads to the behaviour
that the time between reception of an interrupt and the actual appearance of
the data telegram on the communication bus varies so much, that the
sensitive external components run out of synchronisation.

Here are some questions:

A. Is it possible to set the priorities of these automatic mouse and screen
manipulations to a low priority, so that the control procedure is not
disturbed anymore?

I believe the screen manipulation is here the point. Is the graphic
interface a bus master?

B. Is it possible to switch off mouse and keyboard activities by software?

Makes no sense, IMHO. Mouse and keyboard doesn’t create significant CPU
or bus load.

C. Are there maybe some other hidden places for configuration of these
effects except the BIOS settings? In the BIOS I have already disabled
everything that looked like having something to do with mouse and screen
interrupts.

Bus traffic of the PCI or AGP bus can steal a lot of CPU cycles by DMA.

So check out if the delays are related to changes of the screen contents.


–Armin



I would be happy if someone could help me with this.

Thank you.

Nnamdi

11

Nnamdi Kohn wrote:

Hello,

currently, I am trying to stabilize system behavior within a high-speed
robot control done on a standard PC under QNX Neutrino (currently v6.2.). It
seems to be connected with sporadic mouse movements and window refreshments
on the photon screen.

Here is a part of a message from the standard news group:

You need to ensure that you never fill the register FIFO completely.
If you write to a FIFO’s (draw-engine) register when the FIFO is
already full, the chip will insert wait-states on the PCI bus,
effectively locking the system bus until there is a free entry in
the FIFO to allow the PCI write cycle to complete.

What graphic driver you are using??

–Armin

12

Armin Steinhoff wrote:

You need to ensure that you never fill the register FIFO completely.
If you write to a FIFO’s (draw-engine) register when the FIFO is
already full, the chip will insert wait-states on the PCI bus,
effectively locking the system bus until there is a free entry in
the FIFO to allow the PCI write cycle to complete.

What graphic driver you are using??

Oh lovely, I can see many drivers getting caught on that one. And the author also saying that it’s a perfectly ok thing to hit. My personal opinion is the hardware FIFO should not stall, discarding the excess data, but that’s not something that will be settable.


Evan

13

Nnamdi Kohn wrote:

I’ll take a stab and say you’ve got legacy USB support turned on in the
BIOS, try disabling that and any powersave like features too.


I already disabled all “Integrated Peripherals” in the BIOS setup, except
for “OnChip IDE” channel 0 and 1. And in the “Power Management Setup” there
is nothing enabled, either.

Double check for the “Legacy support” option. It may not be under Peripherals as its a BIOS extension thingy that provides emulation.


Not going to help if you are are already at high priority. The only thing
more
precise is to use InterruptAttach().


instead of InterruptAttachEvent()?

Yup, you are bypassing all thread level priority handling with InterruptAttach(). The only things that can interfere with this is IRQ masking and bus contention, eg: the graphics card is locking the CPU out.


You do want the mouse to work I presume, after all you do have a GUI to
manage.


We are thinking of using a separate computer providing the GUI and connected

I personally feel that is overkill. But it will likely solve your problem.


Evan

14

Evan Hillas wrote:

instead of InterruptAttachEvent()?

Yup, you are bypassing all thread level priority handling with
InterruptAttach(). The only things that can interfere with this is IRQ
masking and bus contention, eg: the graphics card is locking the CPU out.

And IRQ sharing with other device drivers using InterruptAttachEvent(),
of course!

Even if the other driver thread-level work is designed for very very low
priority, it can keep the shared IRQ masked until it has finished
dealing with the pulse coming from its InterruptAttachEvent()…

Davide

–
/* Ancri Davide - */

15

Nnamdi Kohn wrote:

Hello,

currently, I am trying to stabilize system behavior within a high-speed
robot control done on a standard PC under QNX Neutrino (currently v6.2.). It
seems to be connected with sporadic mouse movements and window refreshments
on the photon screen. Here is the scene:

Upon reception of a cyclic, hardware-generated interrupt (exactly every 1 ms
from the communication system), some calculations are made in a high
priority thread in order to set the contents of an outgoing data telegram.
After composition, the data telegram is broadcasted on the communication
bus. The external components receiving the broadcast, each evaluate the data
and send back an answer telegram. For some components, it is necessary that
the time intervals between the incoming data telegrams are always the same
(there is a small tolerable jitter).
When I move the mouse pointer on the PC the open windows on the photon
screen are activated and deactivated. This internally leads to the behaviour
that the time between reception of an interrupt and the actual appearance of
the data telegram on the communication bus varies so much, that the
sensitive external components run out of synchronisation.

This sounds like IRQ masking. We have an application that gets an
interrupt every 256 microseconds from an A/D card. If the A/D card
shares an IRQ line with anything else, there can be jitter in the
interrupt intervals, even when the thread that handles the A/D
interrupts has the highest priority. In our experience, ethernet
controllers are particularly bad, but I wouldn’t be surprised if
graphics controllers or serial drivers could also cause problems.
Our solution is this. In the BIOS, we assign an IRQ level of 9 to one
PCI slot, and we plug the A/D card into this slot. We then ensure that
no other devices are using IRQ 9. We also use InterruptAttach rather
than InterruptAttachEvent.

Here are some questions:

A. Is it possible to set the priorities of these automatic mouse and screen
manipulations to a low priority, so that the control procedure is not
disturbed anymore?

B. Is it possible to switch off mouse and keyboard activities by software?

C. Are there maybe some other hidden places for configuration of these
effects except the BIOS settings? In the BIOS I have already disabled
everything that looked like having something to do with mouse and screen
interrupts.

I would be happy if someone could help me with this.

Thank you.

Nnamdi

16

Davide Ancri wrote:

And IRQ sharing with other device drivers using InterruptAttachEvent(),
of course!

Good point.

Nnamdi, is there other devices sharing the same IRQ with your 1kHz interrupt? The graphics chip for example?


Evan

17

Evan Hillas wrote:

Nnamdi, is there other devices sharing the same IRQ with your 1kHz
interrupt? The graphics chip for example?

Err, ignore that. I see John has a good writeup on it.


Evan

18

JohnMcClurkin wrote:

In our experience, ethernet controllers are particularly bad, but I
wouldn’t be surprised if graphics controllers or serial drivers could
also cause problems.

There was a very interesting thread some month ago about this:

news://inn.qnx.com:119/d8716f$4so$1@inn.qnx.com

InterruptAttachEvent() is a wonderful way to play with interrupts, but
ONLY for non-shared IRQ scenarios!!

In tipical x86 PC-like system it’s nearly impossible to install a couple
of custom PCI devices and getting their IRQ not being shared with some
ethernet card: IRQs dedicated to PCI are tipically 3, and most
industrial PC boards mount 3 ethernet controllers (and we need all of
them, often!).

I think implementing ethernet drivers with InterruptAttachEvent() for
the x86 PC architecture is the negation of QNX realtime strength itself,
but QSSL seems not to think so!


Our solution is this. In the BIOS, we assign an IRQ level of 9 to one
PCI slot, and we plug the A/D card into this slot. We then ensure
that no other devices are using IRQ 9.

Unfortunately, not every BIOS let you “isolate” an IRQ for a particular
slot.


We also use InterruptAttach rather than InterruptAttachEvent.

When you’re sure that you IRS is not shared, InterruptAttachEvent()
gives bettere real-time behaviour of you overall system.

Davide

–
/* Ancri Davide - */