mqueue: how to arbitrate the access to the full queue?

QNX Newsgroups (Archives) qnx.qnx4
1

I want to try using the Mqueue for priority management of some messages
sent to a process. The messages will be of hex (non-text) nature. The
receiving process is bound to an IRQ-driven hardware. The sending
processes have different priorityes of their messages. One is sending the
higher priority messages only, the other is sending a normal (less
priority) ones only.

The lesser priority messages may appear with more speed than hardware may
consume them, so (I hope) the sending process will block if the queue is
full.

How do I make sure the higher priority message from the other process will
enter the queue?

Is it possible to reserve some space in the queue for higher priority
messages?

Or, I just must assign the sending processes with a slightly different
priorityes themselves, so as soon as the queue is able to accept a message

  • the pending higher priority one will enter the queue?

Tony.

PS
The messages’ sizes are multiple of 64 bytes. The hardware consumes a 64
bytes per IRQ. When the previously full queue is considered as ready from
a sending process’s perspective - when it is able to write whole, say,
5*64 bytes of a message, or when there is at least 64 bytes free?

2

Tony <mts.spb.suxx@mail.ru> wrote:

I want to try using the Mqueue for priority management of some messages
sent to a process. The messages will be of hex (non-text) nature. The
receiving process is bound to an IRQ-driven hardware. The sending
processes have different priorityes of their messages. One is sending the
higher priority messages only, the other is sending a normal (less
priority) ones only.

The lesser priority messages may appear with more speed than hardware may
consume them, so (I hope) the sending process will block if the queue is
full.

How do I make sure the higher priority message from the other process will
enter the queue?

Is it possible to reserve some space in the queue for higher priority
messages?

Or, I just must assign the sending processes with a slightly different
priorityes themselves, so as soon as the queue is able to accept a message

  • the pending higher priority one will enter the queue?

My first choice would be to use 2 queues, one for the high priority
process, and one for the low priority process.

The receiver could use mq_notify() to get a notification proxy from
either queue, but of course always drain the high-priority queue first.
The receiver could, even, use two different priorities for the two
different proxies for the different queues. Then, the receiver will
have a common Receive(0, …) blocking point, where it will look
at which proxy it got, and if it is the interrupt handler proxy, it
will process the interrupt work, or the queue from whichever queue
needs work.

Tony.

PS
The messages’ sizes are multiple of 64 bytes. The hardware consumes a 64
bytes per IRQ. When the previously full queue is considered as ready from
a sending process’s perspective - when it is able to write whole, say,
5*64 bytes of a message, or when there is at least 64 bytes free?

If using message queues, “able to send” will mean “able to add one
entry to the queue” – you get to decide what size your queue entries
are (and how long the queue is) in the attributes structure you use
when creating the queue. You could create it such that each entry
is 64 bytes, in which case “able to send on queue” would mean 64 bytes,
or you could create it such that each queue entry is 5*64 bytes.

-David

David Gibbs
QNX Training Services
dagibbs@qnx.com

3

On Tue, 03 May 2005 21:18:34 +0400, David Gibbs <dagibbs@qnx.com> wrote:

My first choice would be to use 2 queues, one for the high priority
process, and one for the low priority process.
Well…

This kills the very idea of mqueue for me. Why not just receive and react
to the messages directly then?!

…When the previously full queue is considered as ready from a sending
process’s perspective? > …you get to decide what size your queue
entries are (and how long the queue is) in the attributes structure you
use when creating the queue.
You could create it such that each entry is 64 bytes, in which case
“able to send on queue” would mean 64 bytes, or you could create it such
that each queue entry is 5*64 bytes.
The messages may be (1~32)*64 bytes long.

The queue is up to 64*64 bytes.

Will the trick of having the sending processes on different priority level
work if the queue has two max-sized low priority messages in it and the
higher priority process wants to put there a 2*64 message? What happens
when my hardware will fetch the first 64 bytes off the queue?

Tony.

4

Tony <mts.spb.suxx@mail.ru> wrote:

On Tue, 03 May 2005 21:18:34 +0400, David Gibbs <> dagibbs@qnx.com> > wrote:
My first choice would be to use 2 queues, one for the high priority
process, and one for the low priority process.
Well…
This kills the very idea of mqueue for me. Why not just receive and react
to the messages directly then?!

Sure, why not? (I’m actually going to suggest it…)

The main advantage of the message queues is the ability to have someone
else manage an “in-transit” message, while you’re off doing something
else.

Now, I’m not sure how using two queues kills the idea of queues for you.
It is one “logical” queue, composed of two “implementation queues”, one
high priority stream, and one low priority stream.

…When the previously full queue is considered as ready from a sending
process’s perspective? > …you get to decide what size your queue
entries are (and how long the queue is) in the attributes structure you
use when creating the queue.
You could create it such that each entry is 64 bytes, in which case
“able to send on queue” would mean 64 bytes, or you could create it such
that each queue entry is 5*64 bytes.
The messages may be (1~32)64 bytes long.
The queue is up to 6464 bytes.

That is a very small queue. The overhead of mqueue to manage 64 entries
of 64 bytes is pretty hefty.

Also, remember mqueue is NOT a stream of bytes. So, if you do create the
queue as 64 entries of 64 bytes, and you want to send a message of 32*64,
you will have to issue 32 mq_send() calls to transmit this, cause each
mq_send() will fill up to one queue entry.

But, is there a reason the queue can’t be longer than 64*64 bytes?

You could, actually, create a queue of 64 entries of 32*64 bytes (max
message size) this would allow you to always dump a complete message
as one mq_send().

Will the trick of having the sending processes on different priority level
work if the queue has two max-sized low priority messages in it and the
higher priority process wants to put there a 2*64 message? What happens
when my hardware will fetch the first 64 bytes off the queue?

If the queue is full – say it is as you say 64 entries of 64 bytes, all
filled up with low priority messages (each taking 32 entries). Then,
any send on the queue will either 1) fail with errno EWOULDBLOCK if
MQ_NONBLOCK attribute set or 2) block until a queue entry is freed.

Also, you say “when my hardware will fetch”. If you’re using mqueues,
this has to be the driver fetching, then handing off to the hardware,
not the hardware itself. [Does the 64*64 queue size come from some
hardware restriction? If so, it does not have to be mirrored in your
software mqueues, though it may have to be mirrored in some in-driver
structure.]

When the driver receives an entry from the queue, it will get the
first 64 bytes of the first low priority message. This will free
up the one queue entry, assuming both the high and low priority
senders are blocked trying to write to the queue, the mq_send()
(the first 64 bytes of the high-priority senders 2*64 message)
will be added to the queue, and assuming the mq_prio specified is
also higher than the mq_prio specified for the low-prio senders
messages, then this will be put at the head of the queue. Next
time the hardware driver receives from the queue, it will get the
first 64 bytes from the high-prio sender, and the high-prio sender
will be able to add another message. Next mq_receive() will get
the 2nd chunk of the high-prio message, and assuming the high-prio
is done, the next mq_receive() by the hardware driver will get the
2nd 64-byte chunk of the 1st low-priority message.

The mqueue server will not throw away low-prio messages to make room
for high-prio messages.

If you need that “throw-away”, and because the mqueue structure and
methodology doesn’t look to work well for your fixed 6464 byte
queue of multi-chunk messages, you may want to just maintain the
queue inside your driver, in memory the hardware can get at. This
will give far more efficient throughput (far fewer copies and
context switches per message, especially a 3264 byte one), and give
you complete control over whether or not you want to overwrite
messages that are already in queue.

-David

David Gibbs
QNX Training Services
dagibbs@qnx.com

5

On Wed, 04 May 2005 00:44:56 +0400, David Gibbs <dagibbs@qnx.com> wrote:

My first choice would be to use 2 queues, one for the high priority
process, and one for the low priority process.
This kills the very idea of mqueue for me. Why not just receive and
react to the messages directly then?!
Sure, why not? (I’m actually going to suggest it…)
This does not scale well if one wants something more intricate than just

two senders and two levels of priority for the messages.

The main advantage of the message queues is the ability to have someone
else manage an “in-transit” message, while you’re off doing something
else.
Exactly!

I want to isolate the receiving process from any details of how many are
the senders there, how many are the priority levels of the messages. Let
it read just one queue and keep the hardware working.

The queue is up to 64*64 bytes.
That is a very small queue. The overhead of mqueue to manage 64 entries
of 64 bytes is pretty hefty.
OK.

Does it track the actual data length in the entry? If the entry slot is
32*64 and I put there just a 64-bytes message - how much memory will be
copyed during the mq_send() and mq_receive()?
Does it needs (internally, “automagically”) "wipe"ing the remnants of a
longer previous message to put a shorter new one in?

…if you do create the queue as 64 entries of 64 bytes, and you want to
send a message of 32*64, you will have to issue 32 mq_send() calls to
transmit this, cause each mq_send() will fill up to one queue entry.
I see both “pro” and “con” sides about this in my project… Need more

thinking…

But, is there a reason the queue can’t be longer than 6464 bytes?
No reason actually, I’m still thinking of a way to use it properly.



You could, actually, create a queue of 64 entries of 3264 bytes (max
message size) this would allow you to always dump a complete message as
one mq_send().
I’m thinking about it now.



If the queue is full – say it is as you say 64 entries of 64 bytes, all
filled up with low priority messages (each taking 32 entries). Then,
any send on the queue will either 1) fail with errno EWOULDBLOCK if
MQ_NONBLOCK attribute set or 2) block until a queue entry is freed.
Seems, now I get the idea.

The state of the queue changes when I read the whole message, i.e. to free
a slot if it’s bigger than the message.
So, if the queue has 64 slots of 3264 bytes - the reading of the first 64
bytes of a pending 264 bytes message WILL NOT unblock the next pending
sender, right? No matter if a sender has just 64 bytes to put there, the
slot is not available yet.
Please comment.

When the driver receives an entry from the queue, it will get the first
64 bytes of the first low priority message. This will free up the one
queue entry, assuming both the high and low priority senders are blocked
trying to write to the queue, the mq_send() (the first 64 bytes of the
high-priority senders 2*64 message) will be added to the queue, and
assuming the mq_prio specified is also higher than the mq_prio specified
for the low-prio senders messages, then this will be put at the head of
the queue. Next time the hardware driver receives from the queue, it
will get the first 64 bytes from the high-prio sender, and the high-prio
sender will be able to add another message. Next mq_receive() will get
the 2nd chunk of the high-prio message, and assuming the high-prio is
done, the next mq_receive() by the hardware driver will get the 2nd
64-byte chunk of the 1st low-priority message.
That’s what I want, actually…



The mqueue server will not throw away low-prio messages to make room for
high-prio messages.
That’s perfectly OK for me.

Tony

6

Tony <mts.spb.suxx@mail.ru> wrote:

On Wed, 04 May 2005 00:44:56 +0400, David Gibbs <> dagibbs@qnx.com> > wrote:
My first choice would be to use 2 queues, one for the high priority
process, and one for the low priority process.
This kills the very idea of mqueue for me. Why not just receive and
react to the messages directly then?!
Sure, why not? (I’m actually going to suggest it…)
This does not scale well if one wants something more intricate than just
two senders and two levels of priority for the messages.

Well, have one queue per sender doens’t scale very well, but using
QNX Send/Receive/Reply does scale reasonably well for multiple senders
and priorities. And, if you don’t mind a client being blocked until
they reach the top of the priority list, it also isn’t that hard to
design/implement. But, if you want to dequeue and reply, that means
implementing a priority-list queue in the driver. Not too difficult,
quite scalable, but may be work you don’t need/want to do. (But, it
does give you the most custom and configurable choice; and probably
the most CPU and latency efficient method.)

The main advantage of the message queues is the ability to have someone
else manage an “in-transit” message, while you’re off doing something
else.
Exactly!
I want to isolate the receiving process from any details of how many are
the senders there, how many are the priority levels of the messages. Let
it read just one queue and keep the hardware working.

The queue is up to 6464 bytes.
That is a very small queue. The overhead of mqueue to manage 64 entries
of 64 bytes is pretty hefty.
OK.
Does it track the actual data length in the entry? If the entry slot is
3264 and I put there just a 64-bytes message - how much memory will be
copyed during the mq_send() and mq_receive()?

It does track the length. In that case, just 64 bytes will be copied
on both mq_send and mq_receive. Note, though, if the queue entry size
is 3264 bytes, and the sender did an mq_send() of 3264 bytes, and you
do an mq_receive() of 264 bytes, this is an error. You will not
be given the first 264 bytes of the message with the remaining 3064
bytes available to be mq_received later. mq_receive() should always
pass as it receive buffer a buffer big enough for the biggest possible
message, and if you are receiving a smaller one, it will only partially
fill your buffer. (If you have a queue size of 3264 bytes, with an
entry of 3264 bytes, and do an mq_receive() of, say, 264 bytes, I think
that QNX4’s Mqueue implementation will give you the first 2*64 bytes,
and discard the rest. I think that QNX6’s mqueue implementation will
give you an error on the mq_receive, give you no data, and not
de-queue the message. I think QNX6’s implementation is more correct
according to POSIX, and that you should not depend on QNX4’s
implementation. If you do the correct thing, which is to make sure
the mq_receive() buffer is always == the queue entry size, there will
be no problem.)

A good way to think about an mqueue is as a conveyor belt of fixed-size
buckets, say 1 litre. If you pour a 1/2 litre in, the bucket is half
full, and chugs forward 1/2 full. If you try to pour 2 litres in,
either you get an error, or the extra litre spills all over the floor,
and you have a full 1l bucket, and 1l of spilled water. At the
end of the line, when you empty a bucket, you dump the whole bucket;
if you have a one litre basin to empty it into, then you get it all.
If you have a bucket with only 1/2 litre in it, you only get 1/2
litre in the basin, cause you only empty one bucket at a time. If you
have a 1/2 litre basin and dump a 1 litre bucket into it, well, more
spilled water.

Does it needs (internally, “automagically”) "wipe"ing the remnants of a
longer previous message to put a shorter new one in?

Logically, yeah, cause it records the length (how full) each queue
entry actually is. In actual implementation, no. It doesn’t memset
data that will never be used. If you mq_send() 64 bytes to a 32*64
byte queue entry size queue that had been previously used for a full
message, it will over-write the first 64 bytes with new data, and record
that there is 64 bytes of data there. If you now mq_receive() from
that queue entry, you will be given the 64 bytes of data. The old data
is still there, but never looked at.

…if you do create the queue as 64 entries of 64 bytes, and you want to
send a message of 32*64, you will have to issue 32 mq_send() calls to
transmit this, cause each mq_send() will fill up to one queue entry.
I see both “pro” and “con” sides about this in my project… Need more
thinking…

But, is there a reason the queue can’t be longer than 64*64 bytes?
No reason actually, I’m still thinking of a way to use it properly.

You could, actually, create a queue of 64 entries of 32*64 bytes (max
message size) this would allow you to always dump a complete message as
one mq_send().
I’m thinking about it now.

If the queue is full – say it is as you say 64 entries of 64 bytes, all
filled up with low priority messages (each taking 32 entries). Then,
any send on the queue will either 1) fail with errno EWOULDBLOCK if
MQ_NONBLOCK attribute set or 2) block until a queue entry is freed.
Seems, now I get the idea.

The state of the queue changes when I read the whole message, i.e. to free
a slot if it’s bigger than the message.
So, if the queue has 64 slots of 3264 bytes - the reading of the first 64
bytes of a pending 264 bytes message WILL NOT unblock the next pending
sender, right?

Wrong. Here I have to be a bit careful, because I’m pretty sure the behaviour
changed from QNX4 to QNX6. In any case, it is an ERROR to request a receive
of less than the queue size – and you CAN NOT read a queue entry in
multiple pieces, it must be mq_receive()d in it’s entirety.

What I think will happen under QNX4: the queue entry will be removed from
the queue, and as much as will fit in your mq_receive() buffer will be
copied to your mq_receive() buffer, and then the queue entry will be
marked as available. At this point, the blocked sender will fill that queue
entry with it’s 64 byte entry and unblock.

(Under QNX6, your mq_receive() call will fail if the receive buffer size
is less than the queue size, independent of how full or empty any
particularly queue entry is.)

-David

David Gibbs
QNX Training Services
dagibbs@qnx.com

7

On Wed, 04 May 2005 21:45:56 +0400, David Gibbs <dagibbs@qnx.com> wrote:

So, if the queue has 64 slots of 3264 bytes - the reading of the first
64 bytes of a pending 264 bytes message WILL NOT unblock the next
pending sender, right?
Wrong.
(…snip…)
In any case, it is an ERROR to request a receive of less than the queue
size – and you CAN NOT read a queue entry in multiple pieces, it must
be mq_receive()d in it’s entirety.
Now I got it.

Thanks a lot!

Tony.