We are trying to analyse a system hang issue on a multicore machine.
We have a QNX application which is using Message passing, pulse, semaphore, mutex, message queues, signaling etc. Application is having two processes and one of the process is having almost
76 threads based on a worst case user input.The other process is single threaded and is running as a resource manager. All these threads in the application form a complex communication path. But we only ensure that message passing will not occur in two directions (upward-downward) to avoid a deadlock condition. We have used pulses in both upward-downward direction (within the first process). We have Photon based GUI update also something like a logging mechanism. Also message queue is also used between threads. Scenario is like 33 threads will be sending messages to a single message queue and one thread will be receiving messages from this and is doing msg_sendv() to a resource manager.
Now the application hangs on a quad-core machine on continuous run.
As per the following QNX post,
they say QNX message queues also can produce a deadlock scenario.
On analysing the application, we see that message queue used is getting full manytimes. We haven’t used NONBLOCK flag for the threads which send messages to message queue. So if the queue gets full, these threads will be blocked. So my doubt is:- is there any possibility that this queue getting full and the receiver of the queue is blocked on the resource manager and is not able to receive any message from the queue. This causes the 33 threads to block as they are waiting to send message to the queue and causes some other related threads also to block.
Is this queue full situation also cause a QNX system hang?
How can we avoid a QNX full condition? In our application, even 1000 messages can be generated in 1ms. We have put Queue size as 1500. So we experience a high rate of queue full condition. Can we set an “unlimited” value for maximum number of messages in Queue?