Hello All.
I have enocountered strange problem with multi-threaded resource managers.
I took part in development of large system based on multi-threaded resource managers.
Each resource manager registers processor for _IO_MSG message type and
handles client requests based on simple MsgSend() calls:
// initialize default message handlers and assign own processor for _IO_MSG message type
iofunc_func_init(_RESMGR_CONNECT_NFUNCS, &connect_funcs,
_RESMGR_IO_NFUNCS, &io_funcs);
io_funcs.msg = io_msg;
Yesterday, I was amazed, when I tested one critical resource manager. In spite of I am using
multi-threaded resource manager, only one message could be processed in time.
I have foud that resource manager don’t process any subsequent messages until previous io_msg() function
would be finished! All other threads stops in ‘CONDVAR’ state, until io_msg() function() finish.
I don’t used any synchronization calls in my own code explicitly!
I checked all available multi-threaded resource manager sources, I have. All resource managers have the similar
behaviour.
Finally, I took the sample from Neutrino help and got the same results.
It is easy to reproduce the problem:
In resource manager: Call sleep(10); after MsgReply() in io_msg() message handler.
In the client: Send messages in infinite loop to a resource manager.
Take a look at resource manager threads.
Some of threads would be in RECEIVE state (awaiting message)
One thread would be in NANOSLEEP state (performing sleep())
One thread would be in CONDVAR state. This thread received message but didn’t started it’s processing yet.
\
pidin | grep resmgr
33280030 1 ./resmgr 10o CONDVAR b034f040
33280030 2 ./resmgr 10o RECEIVE 1
33280030 3 ./resmgr 10o NANOSLEEP
33280030 4 ./resmgr 10o RECEIVE 1
33374244 1 ./resmgrclient 10o REPLY 33280030
33378341 1 ./resmgr 10o RECEIVE 1
33378341 2 ./resmgr 10o RECEIVE 1
33378341 3 ./resmgr 10o RECEIVE 1
If client program would be started several times, then resource manager will have several threads in CONDVAR state.
The number of such threads would be equal to number of client processes started. New clients connects slow to
resource manager because open() call is based on message processing too.
May be there are undocumented flags for resource manager API functions or (may be) thread pool functions.
I don’t know them.
Why multi-threaded resource manager behave such way? Does it right? Of course the problem could be avoided but
what pool thread does for me?
Any help would be appreciated.
With best regards, Andrew
Programmer, Volgasoft Ltd. Russia.
The sources of resource manager and client, which illustrate the problem follows (The resource manager is taken from
Neutrino help):
// file resmgr.cpp -----------------------------------------------------------------------------------------
// test multi-threaded resource manager -------------------------------------------------------------------
#include <errno.h>
#include <stdio.h>
#include <stddef.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
/*
- define THREAD_POOL_PARAM_T such that we can avoid a compiler
- warning when we use the dispatch_*() functions below
*/
#define THREAD_POOL_PARAM_T dispatch_context_t
#include <sys/iofunc.h>
#include <sys/dispatch.h>
static resmgr_connect_funcs_t connect_funcs;
static resmgr_io_funcs_t io_funcs;
static iofunc_attr_t attr;
int io_msg( resmgr_context_t *ctp, io_msg_t *iomsg, iofunc_ocb_t *ocb )
{
printf(“Messsage received\n”);
MsgReply(ctp->rcvid, 0, NULL, 0);
sleep(10);
return _RESMGR_NOREPLY;
}
main(int argc, char *argv)
{
/ declare variables we’ll be using */
thread_pool_attr_t pool_attr;
resmgr_attr_t resmgr_attr;
dispatch_t *dpp;
thread_pool_t *tpp;
dispatch_context_t *ctp;
int id;
/* initialize dispatch interface */
if((dpp = dispatch_create()) == NULL) {
fprintf(stderr, “%s: Unable to allocate dispatch handle.\n”,
argv[0]);
return EXIT_FAILURE;
}
/* initialize resource manager attributes */
memset(&resmgr_attr, 0, sizeof resmgr_attr);
resmgr_attr.nparts_max = 1;
resmgr_attr.msg_max_size = 2048;
/* initialize functions for handling messages */
iofunc_func_init(_RESMGR_CONNECT_NFUNCS, &connect_funcs,
_RESMGR_IO_NFUNCS, &io_funcs);
io_funcs.msg = io_msg;
/* initialize attribute structure used by the device */
iofunc_attr_init(&attr, S_IFNAM | 0666, 0, 0);
/* attach our device name /
id = resmgr_attach(dpp, / dispatch handle /
&resmgr_attr, / resource manager attrs /
“/dev/sample”, / device name /
_FTYPE_ANY, / open type /
0, / flags /
&connect_funcs, / connect routines /
&io_funcs, / I/O routines /
&attr); / handle */
if(id == -1) {
fprintf(stderr, “%s: Unable to attach name.\n”, argv[0]);
return EXIT_FAILURE;
}
/* initialize thread pool attributes */
memset(&pool_attr, 0, sizeof pool_attr);
pool_attr.handle = dpp;
pool_attr.context_alloc = dispatch_context_alloc;
pool_attr.block_func = dispatch_block;
pool_attr.handler_func = dispatch_handler;
pool_attr.context_free = dispatch_context_free;
pool_attr.lo_water = 2;
pool_attr.hi_water = 4;
pool_attr.increment = 1;
pool_attr.maximum = 50;
/* allocate a thread pool handle */
if((tpp = thread_pool_create(&pool_attr,
POOL_FLAG_EXIT_SELF)) == NULL) {
fprintf(stderr, “%s: Unable to initialize thread pool.\n”,
argv[0]);
return EXIT_FAILURE;
}
/* start the threads, will not return */
thread_pool_start(tpp);
}
Test client sources:
// file resmgrclient.cpp -----------------------------------------------------------------------------
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <string.h>
#include <sys/iofunc.h>
main(int argc, char **argv)
{
int nCoid = ::open("/dev/sample", O_WRONLY);
if ( nCoid == -1)
{
perror(“Can’t open resource:” );
return 0;
}
while (1)
{
_io_msg rMsg;
memset( &rMsg, 0, sizeof(rMsg));
rMsg.type = _IO_MSG;
printf(“Send message to resmgr.\n”);
MsgSend(nCoid, &rMsg, sizeof(rMsg), NULL, 0);
printf(“Reply received\n”);
}
}