multi-thread resource manager is blocked

QNX USERS QNX6 - OS
1

Hi all,

i am trying to implement a multi-threaded resource manager. i followed the example program in the Momentics help, but i didn’t get what i expected.

Here is my code of the resource manager initialization:

int ResMgrThread(void* args)
{
  /* declare variables we'll be using */
  thread_pool_attr_t   pool_attr;
  resmgr_attr_t        resmgr_attr;
  dispatch_t           *dpp;
  thread_pool_t        *tpp;
  int                  id;

  /* initialize dispatch interface */
  if((dpp = dispatch_create()) == NULL) 
  {
    return EXIT_FAILURE;
  }

  /* initialize resource manager attributes */
  memset(&resmgr_attr, 0, sizeof(resmgr_attr));
  resmgr_attr.nparts_max = 1;
  resmgr_attr.msg_max_size = 2048;

  /* initialize functions for handling messages */
  iofunc_func_init(_RESMGR_CONNECT_NFUNCS, &connect_funcs, 
                   _RESMGR_IO_NFUNCS, &io_funcs);
  connect_funcs.open = io_open;
  io_funcs.devctl = io_devctl;
  io_funcs.close_dup = io_close;

  /* initialize resource manager attributes */
  memset(&resmgr_attr, 0, sizeof(resmgr_attr));
  resmgr_attr.nparts_max = 1;
  resmgr_attr.msg_max_size = 2048;

  /* initialize device attributes */
  iofunc_attr_init(&(deviceattr.attr), S_IFNAM | 0666, 0, 0);
  deviceattr.attr.mount = &mountpoint;

  /* attach device name */
  id = resmgr_attach(
            dpp,            /* dispatch handle        */
            &resmgr_attr,   /* resource manager attrs */
            "/dev/my_resmgr", /* device name */
            _FTYPE_ANY,     /* open type              */
            0,              /* flags                  */
            &connect_funcs, /* connect routines       */
            &io_funcs,      /* I/O routines           */
            &deviceattr);   /* handle                 */

  if(id == -1) 
  {
    return EXIT_FAILURE;
  }

  /* initialize thread pool attributes */
  memset(&pool_attr, 0, sizeof pool_attr);
  pool_attr.handle        = dpp;
  pool_attr.context_alloc = dispatch_context_alloc;
  pool_attr.block_func    = dispatch_block;
  pool_attr.unblock_func  = dispatch_unblock;
  pool_attr.handler_func  = dispatch_handler;
  pool_attr.context_free  = dispatch_context_free;
  pool_attr.lo_water      =  8;
  pool_attr.hi_water      = 12;
  pool_attr.increment     = 1;
  pool_attr.maximum       = 16;

  /* allocate a thread pool handle */
  if((tpp = thread_pool_create(&pool_attr, 
                               POOL_FLAG_EXIT_SELF)) == NULL) 
  {
    return EXIT_FAILURE;
  }

  /* start the threads, will not return */
  thread_pool_start(tpp);
  
  return EOK;
}

And this is the devctl handler function of the resource manager:

int io_devctl( resmgr_context_t *ctp, io_devctl_t *msg, struct ocb *ocb)
{
  int i_ret, i_reqLen, i_respLen;
  void* pv_data;
  int cmd;

  /* number of bytes received */
  i_reqLen = (uint16_t) msg->i.nbytes;

  /* execute devctl handler default function */
  i_ret = iofunc_devctl_default(ctp, msg, &(ocb->hdr));
  if(i_ret != _RESMGR_DEFAULT)
  {
    return i_ret;
  }

  /* get pointer to the data passed by client library */
  pv_data = (void*) _DEVCTL_DATA(msg->i); 
  /* get command */
  cmd = (int)msg->i.dcmd;

  /* distinguish the given command */
  switch(cmd)
  {
  	/* command 1 */
    case CMD_1:
    {
      i_ret = proc_cmd1(pv_data, i_reqLen, &i_respLen);
      if(i_ret == EOK)
      {
      	/* copy client ID to Open Control Block */
        ocb->client_id = ((Cmd1_Data*) pv_data)->client_id;
        if(ocb->client_id == 0)
        {
          printf("Debug: Sleeping 20s for Client ID=0");
          sleep(20);
        }
      }
      else
      {
        PRINTD("Error Processing FEA_OPEN_SESSION_CMD");
      }
      break;
    }

    default:
    {
      return (ENOSYS);
    }
  }  

  /* return with an error */
  if(i_ret != EOK)
  {
    return i_ret;
  }
  
  return (_RESMGR_PTR(ctp, &msg->o, sizeof(msg->o) + i_respLen));
}

the function proc_cmd1() basically write a new unique client ID if it returns successfully.

so i did a small test by implementing 2 small test client program. Both clients send requests with the CMD_1 command using devctl to the resource manager.
The first client receives client ID=0, and the second client receives client ID=1.

In case of client 1, i expect that the handler will execute the 20 seconds sleep function, since client ID=0 is returned from the proc_cmd1() function. While the handler is sleeping due to client 1 request, i tried to run client 2.

I expected - since the resource manager is supposed to be multi-threading - that client 2 request is processed by another thread of the resource manager. But unfortunatelly it is not. Request from client 2 seems to be also blocked while resource manager is sleeping due to request client 1.

Am i doing something wrong here?

thanks for any answer.

2

I don’t have time this morning to check through your code carefully but this seems like a problem I ran into a while ago. I think what might be happening is that the device control block has a mutex that gets held before io_devctl is called, so although you get two threads, they are executed serially through io_devctl. This is default behavior to protect the innocent. The solution is to release and then reclaim the mutex around the sleep(). You will have to dig around to find out where the mutex is stored, but I assume it is in the device control block. There also might be related to a mutex in an iocb. Sorry for the fuzzy answer.

3

Operations on the same attribute are protected, checkout documentation on:

iofunc_attr_lock()
iofunc_attr_trylock()
iofunc_attr_unlock()

4

Hi,

thanks for the hints. i modified the devctl handler function:

int io_devctl( resmgr_context_t *ctp, io_devctl_t *msg, struct ocb *ocb)
{
  int i_ret, i_reqLen, i_respLen;
  void* pv_data;
  int cmd;

  /* number of bytes received */
  i_reqLen = (uint16_t) msg->i.nbytes;

  /* execute devctl handler default function */
  i_ret = iofunc_devctl_default(ctp, msg, &(ocb->hdr));
  if(i_ret != _RESMGR_DEFAULT)
  {
    return i_ret;
  }

[b]
  /* unlockling attributes */
  i_ret = iofunc_attr_unlock(&(deviceattr.attr));
  if(i_ret != EOK)
  {
    return i_ret;
  }
[/b]

  /* get pointer to the data passed by client library */
  pv_data = (void*) _DEVCTL_DATA(msg->i);
  /* get command */
  cmd = (int)msg->i.dcmd;
.........

then it works as i expected which the second thread is no longer blocked as the resource manager is blocked by the first thread.

But then the next questions:

  • who locks the attributes? is it the default handler function iofunc_devctl_default() that i calls in the beginning of my devctl ahndler function?

  • the manual of iofunc_attr_unlock() describes:
    “You must unlock the structure as many times as you locked it.”
    Does this mean that i should lock the attribute again after i unlock it?
    Will it affect my program if i don’t lock it back, after i do the unlock?

  • Is there any good resources(articles,manuals) about this? what i really want is that the resource manager can manage up to parallel 4 connections simultaneously. But it shall not block each others.

thanks,

5

If you not already found it - there is a good tutorial in the QNX docs:

qnx.com/developers/docs/6.3. … esmgr.html

-Peter

6

It’s the resmgr framework that locks the attributed. It makes sense. The attribute is suppose to represent the object being worked on. Imagine if two clients are doing operation at the same time, on an SMP machine. One is doing a write() and the other a ltrunk() at the same. Because these two functions modifies the attributes it could well result in a corrupted/invalid attribute.

Yes in should lock it back, if you don’t, unless you know exactly everything that is going to happen in that resmgr, when and how the attribute is going to be modified/accessed, etc, all sorts of weird thing is going to happen.

7

Hi,

thanks for the hints so far. i read already the “Writing a Resource Manager” article, altough i did’t get all of it.

but anyway i still have some questions to be asked:

  • are the examples in the article workingonly on the iofunc layer of the resource manager?

  • is there any example how to override the default handler functions in the lower layer (e.g. resmgr layer)?
    because i think in my case i would need to do that.

thanks for the answer.

8

Lbdgwgt,

this is my recommendation:

  • read the “Writing a Resource Manager” tutorial - it is excellent and you will learn enough for the first (and maybe also the second) step with resource managers
  • if you do not understand what you are reading - read it again
  • try out the examples

If you still have questions - I’m sure you will get answers here. But you need to understand the basics first.

-Peter

9

Lbdgwgt,

If you can still order it, I highly recommend Robert Krten’s book ‘The QNX Cookbook - Recipes for Programmers’. It has good sample programs of resource managers including one where he does a Ram based filesystem (ie a Ram disk).

Tim