strange preemption behavior caused by photon

QNX Newsgroups (Archives) qnx.newuser
1

Hello,

I am currently programming a real-time application at high cycle frequencies
(up to 8kHz, 125µs). Every cycle, my computer gets an external hardware
INTERRUPT to trigger the application driver’s service routine. The
application consists of several (5) different threads interoperating with
each other, all operating at a QNX priority above 30. Running the
application, as long as nothing happens on the photon desktop (no printf(),
no mouse movement, no screen switching,…), the operation is fine. As soon
as I switch screens on the desktop, the application is not able to follow
the high frequency cycle anymore (indicated by internal measurements with
ClockCycles() or by counting the cycles passed). This should not happen
because of the fact that photon itself resides at a much lower QNX priority
than the high speed application.

Are there any suggestions, why photon activities are able to preempt an
application running at a priority above 30?

Nnamdi

2

Nnamdi Kohn <nnamdi.kohn@tu-bs.de> wrote:

Hello,

I am currently programming a real-time application at high cycle frequencies
(up to 8kHz, 125 us). Every cycle, my computer gets an external hardware
INTERRUPT to trigger the application driver’s service routine. The
application consists of several (5) different threads interoperating with
each other, all operating at a QNX priority above 30. Running the
application, as long as nothing happens on the photon desktop (no printf(),
no mouse movement, no screen switching,…), the operation is fine. As soon
as I switch screens on the desktop, the application is not able to follow
the high frequency cycle anymore (indicated by internal measurements with
ClockCycles() or by counting the cycles passed). This should not happen
because of the fact that photon itself resides at a much lower QNX priority
than the high speed application.

Are there any suggestions, why photon activities are able to preempt an
application running at a priority above 30?

Hardware.

By that, I mean that when you do that stuff, it will cause the graphics
driver to engage the hardware for, probably, a bunch of DMA transfers to
move information from system RAM to graphics engine. This will delay
bus access for every other device at the hardware bus level. It is
quite possibly you’re not seeing OS based process/thread preemption,
but hardware contention that is below the level that the OS can control.

-David

David Gibbs
QNX Training Services
dagibbs@qnx.com

3

Hardware.

By that, I mean that when you do that stuff, it will cause the graphics
driver to engage the hardware for, probably, a bunch of DMA transfers to
move information from system RAM to graphics engine. This will delay
bus access for every other device at the hardware bus level. It is
quite possibly you’re not seeing OS based process/thread preemption,
but hardware contention that is below the level that the OS can control.

your explanation sounds good. Thank you.

  1. As we have to ensure 100 percent time determinism (real-time) while
    running our application, I’d like to know if there is any COMPLETE list of
    possible other hardware effects invisible to the operating system like the
    described one. I guess this even depends on the actual computer hardware
    facilities and bios configuration settings, right?

  2. When these not-controllable effects influence the applications running on
    the system, is there maybe a way to suppress them to take effect within the
    system (locking the keyboard, …)?

Nnamdi

4

Nnamdi Kohn wrote:

  1. As we have to ensure 100 percent time determinism (real-time) while
    running our application, I’d like to know if there is any COMPLETE list of
    possible other hardware effects invisible to the operating system like the
    described one. I guess this even depends on the actual computer hardware
    facilities and bios configuration settings, right?

Yeah, inappropriate hardware can very much get in the way. Any big DMAs are to be considered and maybe flawed design too. There doesn’t have to be any interference, it’s all dependent on the chipset and it’s config.


  1. When these not-controllable effects influence the applications running on
    the system, is there maybe a way to suppress them to take effect within the
    system (locking the keyboard, …)?

If it’s, say, a unified memory system then reducing your graphics resolution and bit-depth should help.

Exactly what sort of jitter level is the concern here?
Is info being missed or are you needing to update some I/O device at 8 kHz?
Can you get away with counting and buffering to catch up when the CPU is freed up?
Maybe an interrupt handler would help? They are quite easy to do in QNX.


Evan

5

Nnamdi Kohn <nnamdi.kohn@tu-bs.de> wrote:

Hardware.

By that, I mean that when you do that stuff, it will cause the graphics
driver to engage the hardware for, probably, a bunch of DMA transfers to
move information from system RAM to graphics engine. This will delay
bus access for every other device at the hardware bus level. It is
quite possibly you’re not seeing OS based process/thread preemption,
but hardware contention that is below the level that the OS can control.

your explanation sounds good. Thank you.

  1. As we have to ensure 100 percent time determinism (real-time) while
    running our application, I’d like to know if there is any COMPLETE list of
    possible other hardware effects invisible to the operating system like the
    described one. I guess this even depends on the actual computer hardware
    facilities and bios configuration settings, right?

COMPLETE list is tricky. It really, and I mean REALLY depends on
the hardware. One of the nastiest ones is SMM (System Management
Mode) on intel (and other x86) hardware. SMM is a mode that,
essentially, on interrupt preempts the processor in a way that is
transparent to the OS running on it, for unpredictable periods of
time. In the worst case, I’ve heard of “hardware” devices that
were actually done by emulation in the BIOS at the SMM level. When
the SMI (System Management Interrupt) kicks in, there is nothing the
OS can do, and my understanding is that this is a completely non
maskable, non-preventable interrupt.

I don’t know enough of the possible bad things hardware could do
to give you a complete list.

  1. When these not-controllable effects influence the applications running on
    the system, is there maybe a way to suppress them to take effect within the
    system (locking the keyboard, …)?

A more rigidly controlled access can reduce the unexpected load a
user can inflict. e.g. the -S option to pwm will disable console
switching.

-David

David Gibbs
QNX Training Services
dagibbs@qnx.com

6

“Nnamdi Kohn” <nnamdi.kohn@tu-bs.de> wrote in message
news:dcv427$5g7$1@inn.qnx.com…

Hardware.

By that, I mean that when you do that stuff, it will cause the graphics
driver to engage the hardware for, probably, a bunch of DMA transfers to
move information from system RAM to graphics engine. This will delay
bus access for every other device at the hardware bus level. It is
quite possibly you’re not seeing OS based process/thread preemption,
but hardware contention that is below the level that the OS can control.

your explanation sounds good. Thank you.

  1. As we have to ensure 100 percent time determinism (real-time) while
    running our application, I’d like to know if there is any COMPLETE list of
    possible other hardware effects invisible to the operating system like the
    described one. I guess this even depends on the actual computer hardware
    facilities and bios configuration settings, right?

A nasty one is CPU throttling. You might expect this on laptop but most P4

will
throttle down if they get too hot.

  1. When these not-controllable effects influence the applications running
    on
    the system, is there maybe a way to suppress them to take effect within
    the
    system (locking the keyboard, …)?

Nnamdi

7

Let me offer a quick opinion here.

Your real-time application should be on a computer by itself. There should
be NO GRAPHICS capibility on that computer. Have a second computer
networked to the first that does any grasphics rendering that you may need
to do. Problem solved.


“Evan Hillas” <evanh@clear.net.nz> wrote in message
news:dcv6vu$7fd$1@inn.qnx.com…

Nnamdi Kohn wrote:

  1. As we have to ensure 100 percent time determinism (real-time) while
    running our application, I’d like to know if there is any COMPLETE list
    of
    possible other hardware effects invisible to the operating system like
    the
    described one. I guess this even depends on the actual computer hardware
    facilities and bios configuration settings, right?


    Yeah, inappropriate hardware can very much get in the way. Any big DMAs
    are to be considered and maybe flawed design too. There doesn’t have to be

any interference, it’s all dependent on the chipset and it’s config.

  1. When these not-controllable effects influence the applications
    running on
    the system, is there maybe a way to suppress them to take effect within
    the
    system (locking the keyboard, …)?


    If it’s, say, a unified memory system then reducing your graphics
    resolution and bit-depth should help.

Exactly what sort of jitter level is the concern here?
Is info being missed or are you needing to update some I/O device at 8
kHz?
Can you get away with counting and buffering to catch up when the CPU is
freed up?
Maybe an interrupt handler would help? They are quite easy to do in QNX.


Evan

8

Bill Caroselli wrote:

Let me offer a quick opinion here.

Your real-time application should be on a computer by itself. There should
be NO GRAPHICS capibility on that computer. Have a second computer
networked to the first that does any grasphics rendering that you may need
to do. Problem solved.

Phooey! What’s the point of QNX? May as well go to some simpler OS/Executive.


Evan

9

Exactly what sort of jitter level is the concern here?

a maximum of approximately 10 µs could be tolerated.

Is info being missed or are you needing to update some I/O device at 8
kHz?

information seems to be missing, because the right software modules involved
could not be executed within the right time. There is no direct I/O except
for the communication which is done at cycle start (READ) and somewhere in
the cycle middle (WRITE).

Can you get away with counting and buffering to catch
up when the CPU is freed up?

No. Unfortunately, the application controlls the parallel drives (motors) of
a high-speed robot. Every input data (coming from sensors and resolvers) is
necessary within the right 8 kHz cycle in order to calculate the right
output data. Received in an other cycle, the data would be worthless.

Maybe an interrupt handler would help? They are quite easy to do in QNX.

that’s what we do. But the complex calculations are divided into different
modules. So there is not ONE interrupt handler able to do all the stuff
necessary. It would collapse our modular architecture.

Nnamdi

→ UNFORTUNATELY I WILL BE OFF FOR THE NEXT TWO WEEKS,
SO MY ANSWERS WILL CONTINUE AFTER MY RETURN. Thankx

10

“Evan Hillas” <evanh@clear.net.nz> wrote in message
news:dd5pqd$n4b$1@inn.qnx.com…

Bill Caroselli wrote:
Let me offer a quick opinion here.

Your real-time application should be on a computer by itself. There
should
be NO GRAPHICS capibility on that computer. Have a second computer
networked to the first that does any grasphics rendering that you may
need
to do. Problem solved.


Phooey! What’s the point of QNX? May as well go to some simpler
OS/Executive.

The point is that if you are using message based (directly or via resmgr
framework) you can easly move the GUI to another machine.

Simpler OS/Executive??? heu how much simpler could it be??

Evan

11

Mario Charest wrote:

The point is that if you are using message based (directly or via resmgr
framework) you can easly move the GUI to another machine.

Simpler OS/Executive??? heu how much simpler could it be??

:slight_smile:

Okay, fine, if that’s the prefered physical placements of equipment, but not fine if it’s just to work around misunderstand timing problems.


Evan

12

Nnamdi Kohn wrote:

Exactly what sort of jitter level is the concern here?


a maximum of approximately 10 µs could be tolerated.

Less than 10us jitter is a tight requirement, I think you are being a bit demanding for anything outside of a single ISR.

Is info being missed or are you needing to update some I/O device at 8 kHz?

information seems to be missing, because the right software modules involved
could not be executed within the right time. There is no direct I/O except
for the communication which is done at cycle start (READ) and somewhere in
the cycle middle (WRITE).

Communication? This is the message passing right? Your ISRs do the direct I/O accesses I presume.

Can you get away with counting and buffering to catch
up when the CPU is freed up?


No. Unfortunately, the application controlls the parallel drives (motors) of
a high-speed robot. Every input data (coming from sensors and resolvers) is
necessary within the right 8 kHz cycle in order to calculate the right
output data. Received in an other cycle, the data would be worthless.

Well, ideally, to get the lowest level servo loop to execute on the most recent data acquired on this cycle you would read the input positions, calculate ideal positions, compare them, and write the outputs all in a single sequence. All that should be a complete module that sits in a single ISR and therefore you won’t be inserting extra lag into the servo system.

That is unless you don’t actually have any servo code of your own. Some systems use independent smart servo controllers that all you have to do is feed the profile to. In this case response time looses it’s importance and my outline looses it’s relevance.

Maybe an interrupt handler would help? They are quite easy to do in QNX.


that’s what we do. But the complex calculations are divided into different

Good.


modules. So there is not ONE interrupt handler able to do all the stuff
necessary. It would collapse our modular architecture.

Build a callback system for your servo ISR or have a wrapper API for accessing the servo data from outside of the servo code.

→ UNFORTUNATELY I WILL BE OFF FOR THE NEXT TWO WEEKS,
SO MY ANSWERS WILL CONTINUE AFTER MY RETURN. Thankx

Okay.


Evan

13

If you have real-time applications and user interface application s running
on the same CPU there will always be instances where the lower priority user
applications will start an operations that MUST finish before the higher
priority real-time application can continue. Like a disk I/O.

With multiple CPUs you can have as many user interface applications as you
want and still have them all appear pretty snappy without affecting the
deterministic qualities of the real-time application.


“Mario Charest” postmaster@127.0.0.1 wrote in message
news:dd87tf$eji$1@inn.qnx.com…

“Evan Hillas” <> evanh@clear.net.nz> > wrote in message
news:dd5pqd$n4b$> 1@inn.qnx.com> …
Bill Caroselli wrote:
Let me offer a quick opinion here.

Your real-time application should be on a computer by itself. There
should
be NO GRAPHICS capibility on that computer. Have a second computer
networked to the first that does any grasphics rendering that you may
need
to do. Problem solved.


Phooey! What’s the point of QNX? May as well go to some simpler
OS/Executive.

The point is that if you are using message based (directly or via resmgr
framework) you can easly move the GUI to another machine.

Simpler OS/Executive??? heu how much simpler could it be??



Evan

14

Bill Caroselli wrote:

If you have real-time applications and user interface application s running
on the same CPU there will always be instances where the lower priority user
applications will start an operations that MUST finish before the higher
priority real-time application can continue. Like a disk I/O.

Going to need a more specify example.

I can imagine a possibility of serialised disc access causing a bottleneck but I can’t see that interfering with a thread that doesn’t do any file handling and is higher priority than all others, as is the case with Nnamdi’s.

Might also need to clarify what is an acceptable maximum level of jitter for thread switch timing, ten microseconds seems unreasonable to me given that ISR’s will always have priority and can easily consume that sort of period on ioport accesses (Around 20 instructions is all it takes).


Evan

15

“Bill Caroselli” <qtps@earthlink.net> wrote in message
news:ddb03r$g1j$1@inn.qnx.com…

If you have real-time applications and user interface application s
running
on the same CPU there will always be instances where the lower priority
user
applications will start an operations that MUST finish before the higher
priority real-time application can continue. Like a disk I/O.

huh?

16

“Evan Hillas” <evanh@clear.net.nz> wrote in message
news:ddb6se$ko9$1@inn.qnx.com…

Bill Caroselli wrote:
If you have real-time applications and user interface application s
running
on the same CPU there will always be instances where the lower priority
user
applications will start an operations that MUST finish before the higher
priority real-time application can continue. Like a disk I/O.


Going to need a more specify example.

I can imagine a possibility of serialised disc access causing a bottleneck
but I can’t see that interfering with a thread that doesn’t do any file

handling and is higher priority than all others, as is the case with
Nnamdi’s.

In your example, it wouldn’t. If however the higher priority thread did any

disk I/O too, then it would be subject to waiting for the lower priority
thread’s I/O to finish before it could start.


Might also need to clarify what is an acceptable maximum level of jitter
for thread switch timing, ten microseconds seems unreasonable to me given

that ISR’s will always have priority and can easily consume that sort of
period on ioport accesses (Around 20 instructions is all it takes).

Evan