Thread_pool + multiple threads + multiple devices +single RM

newbie_learner · April 19, 2013, 9:03pm

Hi All,
After reading many posts here and looking at thread pool examples, I am not fully successful in achieving what I wanted.
Basically I need to have single resource manager with 2 threads. Each thread registers its own pathname space and will have their own handler functions.
I was successful in doing this using normal creation of threads but not with thread pool.
Can anybody tell me whether I can achieve my requirements using thread pool?
So if I use thread pool, I am assuming the program which has main() is thread 1 and it registers its own pathname space. So I just need to create 1 more thread which is blocked on messages and will never return.

Thanks for any pointers before I give up on thread pool.
Sorry for the long email.

Here is my code for anybody interested.

#define NumDevices 2
char *names [NumDevices] =
{
“/dev/fpgaDev”,
“/dev/mdmDev”
};

int createThread2 ();

void* Thread2Handler (void*) ;

//Reg first device and create second thread using thread pool for handling msgs destined to dev2

int main(int argc, char *argv)
{
/ declare variables we’ll be using */
resmgr_attr_t resmgr_attr;
dispatch_t *dpp;
dispatch_context_t ctp;
int id;
/ declare variables we’ll be using */
thread_pool_attr_t pool_attr;
thread_pool_t *tpp;
dispatch_t *dpp;
resmgr_attr_t resmgr_attr;
int id;

/* initialize dispatch interface */
if((dpp = dispatch_create()) == NULL) {
    fprintf(stderr,
            "%s: Unable to allocate dispatch handle.\n",
            argv[0]);
    return EXIT_FAILURE;
}

/* initialize resource manager attributes */
memset(&resmgr_attr, 0, sizeof(resmgr_attr));
resmgr_attr.nparts_max = 1;
resmgr_attr.msg_max_size = 2048;

/* initialize functions for handling messages */
iofunc_func_init(_RESMGR_CONNECT_NFUNCS, &connect_funcs,
                 _RESMGR_IO_NFUNCS, &io_funcs);

connect_funcs.open = io_open;
io_funcs.read = io_read;
io_funcs.write = io_write;
io_funcs.devctl = io_devctl;
	
/* initialize attribute structure used by the device */
iofunc_attr_init(&attr, S_IFNAM | 0666, 0, 0);

/* attach our device name */
id = resmgr_attach(
        dpp,            /* dispatch handle        */
        &resmgr_attr,   /* resource manager attrs */
        names[0],  /* device name            */
        _FTYPE_ANY,     /* open type              */
        0,              /* flags                  */
        &connect_funcs, /* connect routines       */
        &io_funcs,      /* I/O routines           */
        &attr);    /* handle      */

if(id == -1) {
    fprintf(stderr, "%s: Unable to attach name.\n", argv[0]);
    return EXIT_FAILURE;
}

memset( &pool_attr, 0, sizeof pool_attr );
pool_attr.handle = dpp;
pool_attr.context_alloc = dispatch_context_alloc;
pool_attr.block_func = dispatch_block;
pool_attr.unblock_func = dispatch_unblock;
pool_attr.handler_func = dispatch_handler;
pool_attr.context_free = dispatch_context_free;
pool_attr.lo_water = 1;  [b]//I need only 1 more thread which will be blocked on messges and never return.[/b]
pool_attr.hi_water = 1;
pool_attr.increment = 1;
pool_attr.maximum = 1;

if((tpp = thread_pool_create( &pool_attr,
             POOL_FLAG_EXIT_SELF)) == NULL ) {
    fprintf(stderr,
            "%s: Unable to initialize thread pool.\n",
            argv[0]);
    return EXIT_FAILURE;
}

pthread_attr_t attrThread;

pthread_attr_init( &attrThread );
pthread_attr_setdetachstate( &attrThread, PTHREAD_CREATE_DETACHED );
/* Start the thread which will handle dev2 events. */
pthread_create ( NULL, &attrThread, Thread2Handler, NULL );

/* Never returns */
thread_pool_start( tpp );
return EXIT_SUCCESS;

}

//I am not sure whether this is the right way to handle messages destined for
//dev2 device which is handled by this thread.
void* Thread2Handler (void* ) {

/* declare variables we'll be using */
resmgr_attr_t        resmgr_attr;
dispatch_t           *dpp;
dispatch_context_t   *ctp;
int                  id;

/* initialize dispatch interface */
if((dpp = dispatch_create()) == NULL) {
    printf("\r\nUnable to allocate dispatch handle.\n");
    return 0;
}

/* initialize resource manager attributes */
memset(&resmgr_attr, 0, sizeof(resmgr_attr));
resmgr_attr.nparts_max = 1;
resmgr_attr.msg_max_size = 2048;

/* initialize functions for handling messages */
iofunc_func_init(_RESMGR_CONNECT_NFUNCS, &mdmConnect_funcs,
                 _RESMGR_IO_NFUNCS, &mdmio_funcs);

mdmConnect_funcs.open = mdmRm_io_open;
mdmio_funcs.read = mdmRm_io_read;   //different handler functions
mdmio_funcs.write = mdmRm_io_write;
mdmio_funcs.devctl = mdmRm_io_devctl; 

 /* initialize attribute structure used by the device */
iofunc_attr_init(&mdmAttr, S_IFNAM | 0666, 0, 0);

/* attach our device name */
id = resmgr_attach(
        dpp,            /* dispatch handle        */
        &resmgr_attr,   /* resource manager attrs */
        names[1],       /* [b]diff device name       [/b]     */
        _FTYPE_ANY,     /* open type              */
        0,              /* flags                  */
        &mdmConnect_funcs, /* connect routines       */
        &mdmio_funcs,      /* I/O routines           */
        &mdmAttr);    /* handle      */

if(id == -1) {
    printf("\r\nUnable to attach name.\n");
    return NULL;
}
  
/* allocate a context structure.The context structure contains a buffer where messages will be received. The size of
the buffer was set when we initialized the resource manager attribute structure */

//If I comment out the below section of the code, my resource manager will not work as expected when requests come from the clients
ctp = dispatch_context_alloc(dpp);

/* start the resource manager message loop */

while(1) {
    if((ctp = dispatch_block(ctp)) == NULL) {
        fprintf(stderr, "block error\n");
        return 0;
    }

    dispatch_handler(ctp);
}

return 0;

}

maschoen · April 20, 2013, 5:56pm

This attempt seems very confused and overly complicated. Two questions.

Why are you trying to do this with with two threads instead of two processes?
Assuming there is a good answer to 1), why are you creating two dispatch contexts with two sets of dispatch functions?

newbie_learner · April 22, 2013, 7:53pm

Sorry maschoen .

Let me ask my question in a different way. How do I implement a single resource manager process with 2 threads(in my case), each thread registering separate pathname space, and each thread with different handler functions using thread pool ? IS it possible at all with thread pool ?

Now coming to your question,
I said two different devices just to avoid confusion among readers.

Basically I’ll be dealing with a single FPGA device and the memory in the FPGA is separated into 2 different sections .Section A and section B. Each section will be accessed using different interfaces(say PCIe for section A and SPI interface for section B). We need the clients to access these memory parallely .
Hence I thought it would be a good idea to go with separate threads wherein each thread registers separate pathname space (treating it like a diff device), mmap() different address, and each read/write handler function interfaces with different device at the same time without being serialized.

Please let me know whether there is better way of achieving this ?

Thanks a lot for your time.

newbie_learner · April 23, 2013, 2:09am

Sorry maschoen .

I had posted my answer to your question this morning and I didn’t see that post in this forum.
Anyway I’ll write it again.

My question is whether it is possible to have a single process (resource manager) having 2 threads managing 2 separate device and each thread having its own handler functions using thread pool ? I am trying to understand that part. Would anybody give me pointers on how to implement separate handlers for each thread within a single resource manager using thread pool?

Coming to your question, I had said that I will be communicating with 2 diff. device to make it simple for readers to understand.

Basically I am going with a single process because I will be dealing with single FPGA device partioned into two sections based on their functionality.
And we have separate interfaces to access those sections of FPGA. PCIe interface deals with section A of FPGA and SPI interface deals with section B of FPGA. Since those sections are independent, we want the client applications to access this single device parallely.
Hence I thought it would be nice to go with separate threads. Each thread registers pathname space for each section , memory maps different address, and provides different handler functions so that diff. sections of FPGA registers can be
accessed at the same time.

Please let me know if there is any better way of achieving the same thing.

Thanks a lot for your time.

maschoen · April 23, 2013, 5:44pm

So the answer to my first question is YES, there is a good reason to put all of this in one resource manager.

But there is no reason that I can think of to have two dispatch managers.

So just use one dispatch context, and in the io callbacks have separate code based on which device has been opened.

Thread_pool + multiple threads + multiple devices +single RM

Thanks for any pointers before I give up on thread pool. Sorry for the long email.

Here is my code for anybody interested.

Thanks for any pointers before I give up on thread pool.
Sorry for the long email.