resource manager and blocking i/o

[Maksim_Yevmenkin1]

hello,

not exactly a ddk question, but close enough.

i’m trying to write a resource manager for a device (say /dev/foo) and
i’d like it to support both blocking and non-blocking i/o.

blocking i/o means that a read() from the /dev/foo should block unless
/dev/foo has data to return, or, /dev/foo was open()ed with o_nonblock.

consider the following meta code:

int
my_io_read(resmgr_context_t *ctp, io_read_t *msg, RESMGR_OCB_T *ocb)
{
int status, nonblock;

status = io_read_verify(ctp, msg, ocb, &nonblock);
if (status != EOK)
return (status);

if ((msg->i.xtype & _IO_XTYPE_MASK) != _IO_XTYPE_NONE)
return (ENOSYS);


…

if (!have_data_for_client) {
if (nonblock)
return (EWOULDBLOCK);

/* XXX block until we have data to return */
}

…
}

now, how could i block in my_io_read()?

obviously, my first idea to simply put pthread_conf_wait() to block
calling thread. however, after re-reading available resource manager
documentation, i realized that io_read (and other io_xxx callbacks)
apparently called with iofunc_attr_t locked. without knowing more about
resource manager internals, i assume that i’m allowed to call
iofunc_attr_unlock() in my_io_read(), right?

while searching for the answer, i found out that one can remember
sender’s id etc. from io_read msg/ocb and return _RESMGR_NOREPLY from
io_read. later, when data is available, one can send a reply using
previously stored information. is this the only solution to the problem?

thanks,
max

[Colin_Burgess1]

You need to store away the information of the client you wish to block, and then return _RESMGR_NOREPLY

Later, when you can satify the request, call resmgr_msg_again() and then do a MsgReplyv to the client with the information.

Maksim Yevmenkin wrote:

hello,

not exactly a ddk question, but close enough.

i’m trying to write a resource manager for a device (say /dev/foo) and
i’d like it to support both blocking and non-blocking i/o.

blocking i/o means that a read() from the /dev/foo should block unless
/dev/foo has data to return, or, /dev/foo was open()ed with o_nonblock.

consider the following meta code:

int
my_io_read(resmgr_context_t *ctp, io_read_t *msg, RESMGR_OCB_T *ocb)
{
int status, nonblock;

status = io_read_verify(ctp, msg, ocb, &nonblock);
if (status != EOK)
return (status);

if ((msg->i.xtype & _IO_XTYPE_MASK) != _IO_XTYPE_NONE)
return (ENOSYS);


…

if (!have_data_for_client) {
if (nonblock)
return (EWOULDBLOCK);

/* XXX block until we have data to return */
}

…
}

now, how could i block in my_io_read()?

obviously, my first idea to simply put pthread_conf_wait() to block
calling thread. however, after re-reading available resource manager
documentation, i realized that io_read (and other io_xxx callbacks)
apparently called with iofunc_attr_t locked. without knowing more about
resource manager internals, i assume that i’m allowed to call
iofunc_attr_unlock() in my_io_read(), right?

while searching for the answer, i found out that one can remember
sender’s id etc. from io_read msg/ocb and return _RESMGR_NOREPLY from
io_read. later, when data is available, one can send a reply using
previously stored information. is this the only solution to the problem?

thanks,
max

–
cburgess@qnx.com

[Maksim_Yevmenkin1]

Colin,

thanks for the quick reply. two questions:

You need to store away the information of the client you wish to block,
and then return _RESMGR_NOREPLY

1. do i need to store anything else besides rcvid in my io_read before
   returning _RESMGR_NOREPLY?

Later, when you can satify the request, call resmgr_msg_again() and then
do a MsgReplyv to the client with the information.

2. where can i read more about resmgr_msg_again() and more specifically
   where do i get a resmgr_context_t pointer? should it be stored (or deep
   copied) in my io_read?

i can only find resmgr_msg_again() in headers, but i can not find it in
the library reference.

not exactly a ddk question, but close enough.

i’m trying to write a resource manager for a device (say /dev/foo) and
i’d like it to support both blocking and non-blocking i/o.

blocking i/o means that a read() from the /dev/foo should block unless
/dev/foo has data to return, or, /dev/foo was open()ed with o_nonblock.

consider the following meta code:

int
my_io_read(resmgr_context_t *ctp, io_read_t *msg, RESMGR_OCB_T *ocb)
{
int status, nonblock;

status = io_read_verify(ctp, msg, ocb, &nonblock);
if (status != EOK)
return (status);

if ((msg->i.xtype & _IO_XTYPE_MASK) != _IO_XTYPE_NONE)
return (ENOSYS);


…

if (!have_data_for_client) {
if (nonblock)
return (EWOULDBLOCK);
/* XXX block until we have data to return */
}

…
}

now, how could i block in my_io_read()?

obviously, my first idea to simply put pthread_conf_wait() to block
calling thread. however, after re-reading available resource manager
documentation, i realized that io_read (and other io_xxx callbacks)
apparently called with iofunc_attr_t locked. without knowing more
about resource manager internals, i assume that i’m allowed to call
iofunc_attr_unlock() in my_io_read(), right?

while searching for the answer, i found out that one can remember
sender’s id etc. from io_read msg/ocb and return _RESMGR_NOREPLY from
io_read. later, when data is available, one can send a reply using
previously stored information. is this the only solution to the problem?

thanks,
max

[Colin_Burgess1]

Maksim Yevmenkin wrote:

Colin,

thanks for the quick reply. two questions:

You need to store away the information of the client you wish to
block, and then return _RESMGR_NOREPLY

1. do i need to store anything else besides rcvid in my io_read before
   returning _RESMGR_NOREPLY?

Not in terms of the resmgr support, no.

Later, when you can satify the request, call resmgr_msg_again() and
then do a MsgReplyv to the client with the information.

2. where can i read more about resmgr_msg_again() and more specifically
   where do i get a resmgr_context_t pointer? should it be stored (or deep
   copied) in my io_read?

resmgr_msg_again is not documented, no.

It does a MsgInfo into ctp->info
then it does a MsgRead of the rcvid into ctp->msg
and then it does a MsgCurrent() (another undocumented kernel call - it bumps your priority
to that of the client who’s waiting)

Take a look at services/slogger for an example of using it.

Cheers,

Colin

–
cburgess@qnx.com

[Maksim_Yevmenkin1]

Colin,

thanks for replying.

You need to store away the information of the client you wish to
block, and then return _RESMGR_NOREPLY

1. do i need to store anything else besides rcvid in my io_read before
   returning _RESMGR_NOREPLY?

Not in terms of the resmgr support, no.

i see.

Later, when you can satify the request, call resmgr_msg_again() and
then do a MsgReplyv to the client with the information.

2. where can i read more about resmgr_msg_again() and more
   specifically where do i get a resmgr_context_t pointer? should it be
   stored (or deep copied) in my io_read?

resmgr_msg_again is not documented, no. >

It does a MsgInfo into ctp->info
then it does a MsgRead of the rcvid into ctp->msg
and then it does a MsgCurrent() (another undocumented kernel call - it
bumps your priority
to that of the client who’s waiting)

Take a look at services/slogger for an example of using it.

all right, i took a quick look at sevices/slogger (io_write.c) to see
how one would use resmgr_msg_again().

after a very brief look, it appears that resmgr_msg_again() (in slogger)
is called from io_write() handler where a resmgr_context_t pointer is
available.

in my case i need to wake up blocked (in io_read) readers from another
thread, where i do not have resmgr_context_t pointer.

for example, my resource manager has a separate thread that polls
hardware and, if data is available, it puts the data into shared
structure. at this point i need to check if there is a blocked reader
and if there is, wake it up.

thanks,
max

[maschoen1]

I wanted to say that your first idea can work, and to me it is
preferable. You can wait in io_read(), typically with a cond_var,
waiting for more data to arrive. If only thread will open the
device at a time, you don’t have to worry about the attr being
locked. If you might have multiple readers, you will need to
unlock the attr structure with iofunc_attr_unlock() before you start
waiting, and relock it when awakened.

This all depends on having a multi-threaded resource manager using a
thread pool. This includes making sure that there are an adequate
number of threads for your application, as well as resources to
create those threads.
This model fits much better if you are porting to and from a
Unix/Linux like kernel, and alleviates your need to save and restore
state.

[Maksim_Yevmenkin1]

maschoen wrote:

I wanted to say that your first idea can work, and to me it is
preferable. You can wait in io_read(), typically with a cond_var,
waiting for more data to arrive. If only thread will open the
device at a time, you don’t have to worry about the attr being
locked. If you might have multiple readers, you will need to
unlock the attr structure with iofunc_attr_unlock() before you start
waiting, and relock it when awakened.

funny you should mention this. if fact, the semantic of my resource
manager is that it demands that only one process can access device at a
time.

in fact, i keep track of each open()/close() with my io_open() and
io_close(). i save process id from ctp->info.pid on first open() and
clear it on last close(). later, i check ctp->info.pid in my io_read()
and io_write() against saved one, and if it does not match i return EBUSY.

as it turns out i, can not return EBUSY from io_open(), otherwise i get
nasty errors “device is busy” when i do ‘ls /dev/foo’ while another
process has it open.

This all depends on having a multi-threaded resource manager using a
thread pool. This includes making sure that there are an adequate
number of threads for your application, as well as resources to
create those threads.
This model fits much better if you are porting to and from a
Unix/Linux like kernel, and alleviates your need to save and restore
state.

well, i really do not need multi-threaded resource manager. but if it is
the only way to make it work, i will have to do it this way.

btw, do i need to do any special tricks if blocked (in
io_read/pthread_cond_wait) process is killed?

thanks,
max

[John_A_Murphy1]

Colin Burgess wrote:

Maksim Yevmenkin wrote:
Colin,

thanks for the quick reply. two questions:

You need to store away the information of the client you wish to
block, and then return _RESMGR_NOREPLY

1. do i need to store anything else besides rcvid in my io_read before
   returning _RESMGR_NOREPLY?

Not in terms of the resmgr support, no.

Later, when you can satify the request, call resmgr_msg_again() and
then do a MsgReplyv to the client with the information.

2. where can i read more about resmgr_msg_again() and more
   specifically where do i get a resmgr_context_t pointer? should it be
   stored (or deep copied) in my io_read?

resmgr_msg_again is not documented, no. >

It does a MsgInfo into ctp->info
then it does a MsgRead of the rcvid into ctp->msg
and then it does a MsgCurrent() (another undocumented kernel call - it
bumps your priority
to that of the client who’s waiting)

Take a look at services/slogger for an example of using it.

Cheers,

Colin

Is there any more information available on resmgr_msg_again and

MsgCurrent? Perhaps a white paper on when and why they should be used,
or notes on the problems they solve? We make extensive use of resource
managers, and there seems to be something new going on here that we’d
like to be sure we understand.

Murf

[Colin_Burgess1]

Here’s the source to resmgr_msg_again()

int resmgr_msg_again(resmgr_context_t *ctp, int rcvid) {
if(MsgInfo(rcvid, &ctp->info) == -1) {
return -1;
}

if((ctp->info.msglen = MsgRead(ctp->rcvid = rcvid, ctp->msg, ctp->msg_max_size, 0)) == -1) {
return -1;
}

(void)MsgCurrent(rcvid);

_resmgr_handler(ctp);

return 0;
}

As you can see, it basically refreshes the resmgr_context_t to be the same as when you originally received the message.
In normal cases you wouldn’t need this done.

MsgCurrent() simply adjusts your server threads’ priority to that of the blocked client. I’ve written docs for it that will be in
a future docs update - I’ll post the copy when I dig it out.

Cheers,

Colin

John A. Murphy wrote:

Colin Burgess wrote:
Maksim Yevmenkin wrote:
Colin,

thanks for the quick reply. two questions:

You need to store away the information of the client you wish to
block, and then return _RESMGR_NOREPLY

1. do i need to store anything else besides rcvid in my io_read
   before returning _RESMGR_NOREPLY?

Not in terms of the resmgr support, no.

Later, when you can satify the request, call resmgr_msg_again() and
then do a MsgReplyv to the client with the information.

2. where can i read more about resmgr_msg_again() and more
   specifically where do i get a resmgr_context_t pointer? should it be
   stored (or deep copied) in my io_read?

resmgr_msg_again is not documented, no. >

It does a MsgInfo into ctp->info
then it does a MsgRead of the rcvid into ctp->msg
and then it does a MsgCurrent() (another undocumented kernel call - it
bumps your priority
to that of the client who’s waiting)

Take a look at services/slogger for an example of using it.

Cheers,

Colin

Is there any more information available on resmgr_msg_again and
MsgCurrent? Perhaps a white paper on when and why they should be used,
or notes on the problems they solve? We make extensive use of resource
managers, and there seems to be something new going on here that we’d
like to be sure we understand.

Murf

–
cburgess@qnx.com

[maschoen1]

Maksim Yevmenkinwrote:
funny you should mention this. if fact, the semantic of my resource
manager is that it demands that only one process can access device
at a
time.

in fact, i keep track of each open()/close() with my io_open() and
io_close(). i save process id from ctp->info.pid on first open()
and
clear it on last close(). later, i check ctp->info.pid in my
io_read()
and io_write() against saved one, and if it does not match i return
EBUSY.

as it turns out i, can not return EBUSY from io_open(), otherwise i
get
nasty errors “device is busy” when i do ‘ls /dev/foo’
while another
process has it open.

I think there is a better way to handle this. You can distinquish
between an open caused by ls from a user trying to get access. You
could then process the former, but EBUSY the latter.

[John_A_Murphy1]

Yeah, I’d looked at the source to resmgr_msg_again()(and MsgCurrent()),
but it wasn’t clear what problem was being solved by the addition of
this code. We’ll be anxiously awaiting the new docs…

Murf

Colin Burgess wrote:

Here’s the source to resmgr_msg_again()

int resmgr_msg_again(resmgr_context_t *ctp, int rcvid) {
if(MsgInfo(rcvid, &ctp->info) == -1) {
return -1;
}

if((ctp->info.msglen = MsgRead(ctp->rcvid = rcvid, ctp->msg,
ctp->msg_max_size, 0)) == -1) {
return -1;
}

(void)MsgCurrent(rcvid);

_resmgr_handler(ctp);

return 0;
}

As you can see, it basically refreshes the resmgr_context_t to be the
same as when you originally received the message.
In normal cases you wouldn’t need this done.

MsgCurrent() simply adjusts your server threads’ priority to that of the
blocked client. I’ve written docs for it that will be in
a future docs update - I’ll post the copy when I dig it out.

Cheers,

Colin

John A. Murphy wrote:
Colin Burgess wrote:
Maksim Yevmenkin wrote:
Colin,

thanks for the quick reply. two questions:

You need to store away the information of the client you wish to
block, and then return _RESMGR_NOREPLY

1. do i need to store anything else besides rcvid in my io_read
   before returning _RESMGR_NOREPLY?

Not in terms of the resmgr support, no.

Later, when you can satify the request, call resmgr_msg_again() and
then do a MsgReplyv to the client with the information.

2. where can i read more about resmgr_msg_again() and more
   specifically where do i get a resmgr_context_t pointer? should it be
   stored (or deep copied) in my io_read?

resmgr_msg_again is not documented, no. >

It does a MsgInfo into ctp->info
then it does a MsgRead of the rcvid into ctp->msg
and then it does a MsgCurrent() (another undocumented kernel call -
it bumps your priority
to that of the client who’s waiting)

Take a look at services/slogger for an example of using it.

Cheers,

Colin

Is there any more information available on resmgr_msg_again and
MsgCurrent? Perhaps a white paper on when and why they should be
used, or notes on the problems they solve? We make extensive use of
resource managers, and there seems to be something new going on here
that we’d like to be sure we understand.

Murf

[Colin_Burgess1]

Here’s the rough copy I provided to our documentation guys…



John A. Murphy wrote:

Yeah, I’d looked at the source to resmgr_msg_again()(and MsgCurrent()),
but it wasn’t clear what problem was being solved by the addition of
this code. We’ll be anxiously awaiting the new docs…

Murf

Colin Burgess wrote:
Here’s the source to resmgr_msg_again()

int resmgr_msg_again(resmgr_context_t *ctp, int rcvid) {
if(MsgInfo(rcvid, &ctp->info) == -1) {
return -1;
}

if((ctp->info.msglen = MsgRead(ctp->rcvid = rcvid, ctp->msg,
ctp->msg_max_size, 0)) == -1) {
return -1;
}

(void)MsgCurrent(rcvid);

_resmgr_handler(ctp);

return 0;
}

As you can see, it basically refreshes the resmgr_context_t to be the
same as when you originally received the message.
In normal cases you wouldn’t need this done.

MsgCurrent() simply adjusts your server threads’ priority to that of
the blocked client. I’ve written docs for it that will be in
a future docs update - I’ll post the copy when I dig it out.

Cheers,

Colin

John A. Murphy wrote:
Colin Burgess wrote:
Maksim Yevmenkin wrote:
Colin,

thanks for the quick reply. two questions:

You need to store away the information of the client you wish to
block, and then return _RESMGR_NOREPLY

1. do i need to store anything else besides rcvid in my io_read
   before returning _RESMGR_NOREPLY?

Not in terms of the resmgr support, no.

Later, when you can satify the request, call resmgr_msg_again()
and then do a MsgReplyv to the client with the information.

2. where can i read more about resmgr_msg_again() and more
   specifically where do i get a resmgr_context_t pointer? should it
   be stored (or deep copied) in my io_read?

resmgr_msg_again is not documented, no. >

It does a MsgInfo into ctp->info
then it does a MsgRead of the rcvid into ctp->msg
and then it does a MsgCurrent() (another undocumented kernel call -
it bumps your priority
to that of the client who’s waiting)

Take a look at services/slogger for an example of using it.

Cheers,

Colin

Is there any more information available on resmgr_msg_again and
MsgCurrent? Perhaps a white paper on when and why they should be
used, or notes on the problems they solve? We make extensive use of
resource managers, and there seems to be something new going on here
that we’d like to be sure we understand.

Murf

–
cburgess@qnx.com

[Maksim_Yevmenkin1]

maschoen wrote:

Maksim Yevmenkinwrote:
funny you should mention this. if fact, the semantic of my resource
manager is that it demands that only one process can access device
at a
time.

in fact, i keep track of each open()/close() with my io_open() and
io_close(). i save process id from ctp->info.pid on first open()
and
clear it on last close(). later, i check ctp->info.pid in my
io_read()
and io_write() against saved one, and if it does not match i return
EBUSY.
as it turns out i, can not return EBUSY from io_open(), otherwise i
get
nasty errors “device is busy” when i do ‘ls /dev/foo’
while another
process has it open.


I think there is a better way to handle this. You can distinquish
between an open caused by ls from a user trying to get access. You
could then process the former, but EBUSY the latter.

and how do i do that?

thanks,
max