Priority inversion issues

We have a priority inversion problem. It’s roughly as follows:

We have a single-thread server process, started at priority 15,
which waits for work in MsgReceive. It gets MsgSend requests from both
real-time processes (at priority 14), and a Photon GUI process (at
priority 10). The non-real-time requests are very short; they just
return some status info.

When the server process is processing a non-real-time request,
its priority drops to 10, as is normal QNX practice. This delays
its response to incoming higher-priority requests, and we miss our
100ms timing constraint on the real-time requests when this happens.

I know that when a high-priority thread tries to lock a mutex
already locked by a lower priority thread the lower priority
thread gets a priority boost. But that doesn’t happen when a thread
handling a message has a higher priority thread trying to send to it,
but blocked at a MsgSend.

I can think of several ways to handle this, but what’s the
standard approach? Make the server multithreaded?

This all occurs on one QNX 6.21 x86 machine; it’s not a
networking issue at the moment.

John Nagle
Team Overbot

John Nagle wrote:

We have a priority inversion problem. It’s roughly as follows:

We have a single-thread server process, started at priority 15,
which waits for work in MsgReceive. It gets MsgSend requests from both
real-time processes (at priority 14), and a Photon GUI process (at
priority 10). The non-real-time requests are very short; they just
return some status info.
When the server process is processing a non-real-time request,
its priority drops to 10, as is normal QNX practice. This delays
its response to incoming higher-priority requests, and we miss our
100ms timing constraint on the real-time requests when this happens.

I know that when a high-priority thread tries to lock a mutex
already locked by a lower priority thread the lower priority
thread gets a priority boost. But that doesn’t happen when a thread
handling a message has a higher priority thread trying to send to it,
but blocked at a MsgSend.

I can think of several ways to handle this, but what’s the
standard approach? Make the server multithreaded?
This all occurs on one QNX 6.21 x86 machine; it’s not a
networking issue at the moment.

It is handled - the high priority thread blocked MSGSEND on the low prio
server should cause a prio boost to the servers threads to get the low
priority job completed.

Are you sure those low prio service requests are really that short (do
they log anything? display anything? require any kind of shared
resource?). Could anything else be pre-empting? io-net, serial,
devi-, devb- all do work at >14 IIRC.

Multi threaded helps, but mutexes for contented shared resources
produces the same result (ie. low prio thread being boosted to high prio
to force them out). If the real-time servicing is completely orthogonal
to the non real-time services, multiple threads help; then again, why
would you not have two processes instead if they are orthogonal.


Cheers,
Adam

QNX Software Systems
[ amallory@qnx.com ]

With a PC, I always felt limited by the software available.
On Unix, I am limited only by my knowledge.
–Peter J. Schoenster <pschon@baste.magibox.net>

Adam Mallory wrote:

John Nagle wrote:

We have a priority inversion problem. It’s roughly as follows:

We have a single-thread server process, started at priority 15,
which waits for work in MsgReceive. It gets MsgSend requests from both
real-time processes (at priority 14), and a Photon GUI process (at
priority 10). The non-real-time requests are very short; they just
return some status info. When the server process is processing a
non-real-time request,
its priority drops to 10, as is normal QNX practice. This delays
its response to incoming higher-priority requests, and we miss our
100ms timing constraint on the real-time requests when this happens.

I know that when a high-priority thread tries to lock a mutex
already locked by a lower priority thread the lower priority
thread gets a priority boost. But that doesn’t happen when a thread
handling a message has a higher priority thread trying to send to it,
but blocked at a MsgSend.

I can think of several ways to handle this, but what’s the
standard approach? Make the server multithreaded? This all occurs
on one QNX 6.21 x86 machine; it’s not a
networking issue at the moment.


It is handled - the high priority thread blocked MSGSEND on the low prio
server should cause a prio boost to the servers threads to get the low
priority job completed.

How? The kernel has no idea what thread is supposed to reply to
the MsgSend when there’s no thread waiting in MsgReceive Or is
there an assumption in the kernel that if a thread was activated
by a MsgReceive, it has some ongoing association with the connection
ID that was returned in the MsgReceive?

Are you sure those low prio service requests are really that short (do
they log anything? display anything? require any kind of shared
resource?). Could anything else be pre-empting? io-net, serial,
devi-, devb- all do work at >14 IIRC.

io-net is a possibility. I’ll check that priority.

Incidentally, this is part of the system that controls our robot
vehicle. It’s all QNX on ruggedized Pentium 4 machines,
and multiple processes are busily intercommunicating on a
moderately tight time schedule. The basic cycle time is 100ms,
so it’s not that tight a schedule. In normal operation,
a processing cycle only takes about 5ms of CPU time. Once in
a while it gets up to 6 or 7. But there’s other stuff going on,
which is supposed to be of lower priority than the main cycle.

If it ever takes more than 125ms, a hardware timer shuts
down the vehicle. Relays drop out, the throttle cuts to
idle, and the brakes go on. So delays are rather noticeable.

John Nagle
Team Overbot
http://www.overbot.com

John Nagle wrote:

How? The kernel has no idea what thread is supposed to reply to
the MsgSend when there’s no thread waiting in MsgReceive Or is
there an assumption in the kernel that if a thread was activated
by a MsgReceive, it has some ongoing association with the connection
ID that was returned in the MsgReceive?

It boosts all the threads which are lower prio than the high prio sender
IIRC.


Cheers,
Adam

QNX Software Systems
[ amallory@qnx.com ]

With a PC, I always felt limited by the software available.
On Unix, I am limited only by my knowledge.
–Peter J. Schoenster <pschon@baste.magibox.net>

Adam Mallory wrote:

John Nagle wrote:

How? The kernel has no idea what thread is supposed to reply to
the MsgSend when there’s no thread waiting in MsgReceive Or is
there an assumption in the kernel that if a thread was activated
by a MsgReceive, it has some ongoing association with the connection
ID that was returned in the MsgReceive?


It boosts all the threads which are lower prio than the high prio sender
IIRC.

Forgot to add that it does this to the threads whom last received on
that channel.


\

Cheers,
Adam

QNX Software Systems
[ amallory@qnx.com ]

With a PC, I always felt limited by the software available.
On Unix, I am limited only by my knowledge.
–Peter J. Schoenster <pschon@baste.magibox.net>