Blocking clients in io_write() and using resmgr_msgread()

QNX Newsgroups (Archives) comp.os.qnx
1

I’m working on a resmgr that needs to be able to block callers of io_write
when the underlying hardware is busy. There will be some memory buffer space
so most of the time clients will not block. In the case of a burst of write
activity, however, the buffer may become full and clients will need to block
until space is available. (i.e., I can’t drop data and don’t want to fail
the write() call).

What I’d like to do is maintain a queue of blocked client rcvids and return
from io_write without a MsgReply, with the intent of offloading the client’s
data later when buffer space becomes available.

So finally…my question:

Is is safe to queue the ctp and use it later to call resmgr_msgread() after
io_write returns _RESMGR_NOREPLY? If not, what is an acceptable way to
resmgr_msgread outside of an io_func? Secondly, am I likely to break combine
messages with this technique?

Thanks!

–Adam

2

Hi Adam,

I’m by far not an expert in writing QNX device drivers and am in fact trying
to learn some of these things as well, I came across an article in the QNX
literature, see:
http://www.qnx.com/developer/docs/momentics_nc_docs/neutrino/prog/resmgr.htm
l#ReturnComesFromResMgr , specifically the section “Leaving the client
blocked, replying later” which actually describes blocking and unblocking.
The literature here does imply you can handle the ctp and save and process
the reply ID anyway you want and it seems the driver is the one that decides
in fact to block or unblock. I’m not totally clear on your intentions but
it sounds like your trying to make this driver be as efficient as possible,
whether you need to really buffer any of the data depends on the rest of
your system’s architecture :slight_smile: Maybe you could give the user of the driver
the option of opening with O_NONBLOCK and never block, always implying a
double buffer and worker thread to dispatch it. If they didn’t ask to be
blocked, just allow the hardware to flow-control the caller of the driver
service; if it takes a while to return when blocking so be it, as long as
other parts of the system can handle the back log. Anyways, I think QNX
lets you do what your asking no matter which way you go.

Hopefully I understood correctly and didn’t miss the point. I’m sure there
are others out there that can offer more insight, just my 1/2 cents worth
:wink:

Regards,
John Roul

“Adam Kropelin” <adk0212@hotmail.com> wrote in message
news:an1tlp$30m$1@inn.qnx.com…

I’m working on a resmgr that needs to be able to block callers of io_write
when the underlying hardware is busy. There will be some memory buffer
space
so most of the time clients will not block. In the case of a burst of
write
activity, however, the buffer may become full and clients will need to
block
until space is available. (i.e., I can’t drop data and don’t want to fail
the write() call).

What I’d like to do is maintain a queue of blocked client rcvids and
return
from io_write without a MsgReply, with the intent of offloading the
client’s
data later when buffer space becomes available.

So finally…my question:

Is is safe to queue the ctp and use it later to call resmgr_msgread()
after
io_write returns _RESMGR_NOREPLY? If not, what is an acceptable way to
resmgr_msgread outside of an io_func? Secondly, am I likely to break
combine
messages with this technique?

Thanks!

–Adam

3

Hi, John…

Thanks for your reply. You’ve given me some ideas to think about.

I’ve been unable to access QNX’s developer site for the last several days.
I’m not sure if the problem is on their end or mine. After IE spins for a
few minutes I get what appears to be an error from our local proxy server.

I looked in my local copy of the docs and read the article I think you were
referencing. Unfortunately, the section on blocking the client is a little
too simplistic :wink: Blocking reads are easy to handle: save the recvid away
and do the MsgReply yourself at some later time. The client stays blocked
until you reply. Easy.

But writes, on the other hand… The reason I would be blocking the client
in that case is that I don’t have room in my internal buffer to hold however
much data he’s trying to write. (If I have no internal buffer at all, the
same problem exists, except that I’m waiting for the hardware to become
available.) I want to block him until I have room for the data (wither in
the hw or my buffer). Saving the recvid alone is not enough for that case.
You need the whole ctp in order to call resmgr_msgread() when you’re finally
ready to retrieve his data. Without the ctp, I don’t know what to do.

As for O_NONBLOCK, opening with that flag generally implies that you’ll fail
the call (EAGAIN) if it would block, not that you would enable some
double-buffering scheme. It would be possible to change the semantics and do
as you suggest but I’m not sure that’s the POSIXy way to do it.

Letting the hardware flow-control the system makes a lot of sense, but even
then I have the same problem. I need to block the writer. I can’t tie up a
server thread sitting in io_write() because I’ll potentially deadlock the
driver. (Think what happens when write() gets called from several clients,
each of which blocks a server thread, and soon there are no threads left
even for read()s.) So I have to return from io_write() while leaving the
client blocked and having not read his data yet. Same problem.

Thanks again for your help. Any further ideas?

–Adam

“John Roul” <fck8828@yahoo.com> wrote in message
news:an4rvi$4gh$1@inn.qnx.com…

Hi Adam,

I’m by far not an expert in writing QNX device drivers and am in fact
trying
to learn some of these things as well, I came across an article in the QNX
literature, see:

http://www.qnx.com/developer/docs/momentics_nc_docs/neutrino/prog/resmgr.htm
l#ReturnComesFromResMgr , specifically the section “Leaving the client
blocked, replying later” which actually describes blocking and unblocking.
The literature here does imply you can handle the ctp and save and process
the reply ID anyway you want and it seems the driver is the one that
decides
in fact to block or unblock. I’m not totally clear on your intentions but
it sounds like your trying to make this driver be as efficient as
possible,
whether you need to really buffer any of the data depends on the rest of
your system’s architecture > :slight_smile: > Maybe you could give the user of the
driver
the option of opening with O_NONBLOCK and never block, always implying a
double buffer and worker thread to dispatch it. If they didn’t ask to be
blocked, just allow the hardware to flow-control the caller of the driver
service; if it takes a while to return when blocking so be it, as long as
other parts of the system can handle the back log. Anyways, I think QNX
lets you do what your asking no matter which way you go.

Hopefully I understood correctly and didn’t miss the point. I’m sure
there
are others out there that can offer more insight, just my 1/2 cents worth
:wink:

Regards,
John Roul

4

Hi Adam,

Have you considered using a thread pool in your resource manager? There is
an example @
file:///C:/QNXsdk/target/qnx6/usr/help/product/neutrino/prog/resmgr.html#HAN
DLING_UNBLOCKING . Specifically the section on “Multi-threaded Resource
Managers”.

Coincidently, I think it was sorta obvious, I meant to say in my last reply:
“If they didn’t ask to NOT be blocked…”

Suddenly I too am having problems reading the QNX on-line version :slight_smile:

Regards,
John

“Adam Kropelin” <adk0212@hotmail.com> wrote in message
news:an9grh$b9g$1@inn.qnx.com…

Hi, John…

Thanks for your reply. You’ve given me some ideas to think about.

I’ve been unable to access QNX’s developer site for the last several days.
I’m not sure if the problem is on their end or mine. After IE spins for a
few minutes I get what appears to be an error from our local proxy server.

I looked in my local copy of the docs and read the article I think you
were
referencing. Unfortunately, the section on blocking the client is a little
too simplistic > :wink: > Blocking reads are easy to handle: save the recvid away
and do the MsgReply yourself at some later time. The client stays blocked
until you reply. Easy.

But writes, on the other hand… The reason I would be blocking the client
in that case is that I don’t have room in my internal buffer to hold
however
much data he’s trying to write. (If I have no internal buffer at all, the
same problem exists, except that I’m waiting for the hardware to become
available.) I want to block him until I have room for the data (wither in
the hw or my buffer). Saving the recvid alone is not enough for that case.
You need the whole ctp in order to call resmgr_msgread() when you’re
finally
ready to retrieve his data. Without the ctp, I don’t know what to do.

As for O_NONBLOCK, opening with that flag generally implies that you’ll
fail
the call (EAGAIN) if it would block, not that you would enable some
double-buffering scheme. It would be possible to change the semantics and
do
as you suggest but I’m not sure that’s the POSIXy way to do it.

Letting the hardware flow-control the system makes a lot of sense, but
even
then I have the same problem. I need to block the writer. I can’t tie up a
server thread sitting in io_write() because I’ll potentially deadlock the
driver. (Think what happens when write() gets called from several clients,
each of which blocks a server thread, and soon there are no threads left
even for read()s.) So I have to return from io_write() while leaving the
client blocked and having not read his data yet. Same problem.

Thanks again for your help. Any further ideas?

–Adam

“John Roul” <> fck8828@yahoo.com> > wrote in message
news:an4rvi$4gh$> 1@inn.qnx.com> …
Hi Adam,

I’m by far not an expert in writing QNX device drivers and am in fact
trying
to learn some of these things as well, I came across an article in the
QNX
literature, see:


http://www.qnx.com/developer/docs/momentics_nc_docs/neutrino/prog/resmgr.htm
l#ReturnComesFromResMgr , specifically the section “Leaving the client
blocked, replying later” which actually describes blocking and
unblocking.
The literature here does imply you can handle the ctp and save and
process
the reply ID anyway you want and it seems the driver is the one that
decides
in fact to block or unblock. I’m not totally clear on your intentions
but
it sounds like your trying to make this driver be as efficient as
possible,
whether you need to really buffer any of the data depends on the rest of
your system’s architecture > :slight_smile: > Maybe you could give the user of the
driver
the option of opening with O_NONBLOCK and never block, always implying a
double buffer and worker thread to dispatch it. If they didn’t ask to
be
blocked, just allow the hardware to flow-control the caller of the
driver
service; if it takes a while to return when blocking so be it, as long
as
other parts of the system can handle the back log. Anyways, I think QNX
lets you do what your asking no matter which way you go.

Hopefully I understood correctly and didn’t miss the point. I’m sure
there
are others out there that can offer more insight, just my 1/2 cents
worth
:wink:

Regards,
John Roul

5

Adam Kropelin <adk0212@hotmail.com> wrote:

Kevin,

Thanks for your reply! I had originally thought I could do exactly as you
say but what stopped me was a sentence in the Krten “Getting Started” book,
Chapter 5, pg 343:

“In most cases we could have just used MsgRead(), but in the case where this
message is part of a “combine message,” resmgr_msgread() performs the
appropriate “magic” for us.”

I was worried about missing out on the “magic”…I don’t want to break
combine messages in the event that the C lib decides to use one. But in the
implementation you dug up I see no evidence of “magic”. > :wink:

Perhaps I can get away with MsgRead() directly…that would be a fine
solution, indeed.

[Mr Krten: Hope you don’t mind that I cc’ed you… I’m curious as to whether
the magic you mentioned is non-existent in the implementation Kevin showed
or just invisible to us mortals… > :wink: > Would you consider it safe to just
blindly use MsgRead()?]

Replied via email; summary: ctp->offset is the “magic” refered to in the
book.

Cheers,
-RK

–Adam

“Kevin Stallard” <> kevin@robots.flyingrobots.fly> .> wrote in message
news:an9mc2$ffm$> 1@inn.qnx.com> …
Because io_writes are a MsgSend() from a client I belive all you really
need
is the rcvid in order to finish reading data from the client. MsgRead()
is
what you would use with the recvid.

I just verified this by checking the source code for resmgr_msgread() (on
cvs.qnx.com) …it is a wrapper for MsgRead(). The only thing you would
have to do is also track the number of bytes you’ve already read from the
client performing the write().

int resmgr_msgread(resmgr_context_t *ctp, void *msg, int nbytes, int
offset)
{
return MsgRead(ctp->rcvid, msg, nbytes, ctp->offset + offset);
}


Hope this helps

Kevin


“Adam Kropelin” <> adk0212@hotmail.com> > wrote in message
news:an9grh$b9g$> 1@inn.qnx.com> …
Hi, John…

Thanks for your reply. You’ve given me some ideas to think about.

I’ve been unable to access QNX’s developer site for the last several
days.
I’m not sure if the problem is on their end or mine. After IE spins for
a
few minutes I get what appears to be an error from our local proxy
server.

I looked in my local copy of the docs and read the article I think you
were
referencing. Unfortunately, the section on blocking the client is a
little
too simplistic > :wink: > Blocking reads are easy to handle: save the recvid
away
and do the MsgReply yourself at some later time. The client stays
blocked
until you reply. Easy.

But writes, on the other hand… The reason I would be blocking the
client
in that case is that I don’t have room in my internal buffer to hold
however
much data he’s trying to write. (If I have no internal buffer at all,
the
same problem exists, except that I’m waiting for the hardware to become
available.) I want to block him until I have room for the data (wither
in
the hw or my buffer). Saving the recvid alone is not enough for that
case.
You need the whole ctp in order to call resmgr_msgread() when you’re
finally
ready to retrieve his data. Without the ctp, I don’t know what to do.

As for O_NONBLOCK, opening with that flag generally implies that you’ll
fail
the call (EAGAIN) if it would block, not that you would enable some
double-buffering scheme. It would be possible to change the semantics
and
do
as you suggest but I’m not sure that’s the POSIXy way to do it.

Letting the hardware flow-control the system makes a lot of sense, but
even
then I have the same problem. I need to block the writer. I can’t tie up
a
server thread sitting in io_write() because I’ll potentially deadlock
the
driver. (Think what happens when write() gets called from several
clients,
each of which blocks a server thread, and soon there are no threads left
even for read()s.) So I have to return from io_write() while leaving the
client blocked and having not read his data yet. Same problem.

Thanks again for your help. Any further ideas?

–Adam

“John Roul” <> fck8828@yahoo.com> > wrote in message
news:an4rvi$4gh$> 1@inn.qnx.com> …
Hi Adam,

I’m by far not an expert in writing QNX device drivers and am in fact
trying
to learn some of these things as well, I came across an article in the
QNX
literature, see:



http://www.qnx.com/developer/docs/momentics_nc_docs/neutrino/prog/resmgr.htm
l#ReturnComesFromResMgr , specifically the section “Leaving the client
blocked, replying later” which actually describes blocking and
unblocking.
The literature here does imply you can handle the ctp and save and
process
the reply ID anyway you want and it seems the driver is the one that
decides
in fact to block or unblock. I’m not totally clear on your intentions
but
it sounds like your trying to make this driver be as efficient as
possible,
whether you need to really buffer any of the data depends on the rest
of
your system’s architecture > :slight_smile: > Maybe you could give the user of the
driver
the option of opening with O_NONBLOCK and never block, always implying
a
double buffer and worker thread to dispatch it. If they didn’t ask to
be
blocked, just allow the hardware to flow-control the caller of the
driver
service; if it takes a while to return when blocking so be it, as long
as
other parts of the system can handle the back log. Anyways, I think
QNX
lets you do what your asking no matter which way you go.

Hopefully I understood correctly and didn’t miss the point. I’m sure
there
are others out there that can offer more insight, just my 1/2 cents
worth
:wink:

Regards,
John Roul


\

–
Robert Krten, PARSE Software Devices +1 613 599 8316.
Realtime Systems Architecture, Books, Video-based and Instructor-led
Training and Consulting at www.parse.com.
Email my initials at parse dot com.

6

Kevin,

Thanks for your reply! I had originally thought I could do exactly as you
say but what stopped me was a sentence in the Krten “Getting Started” book,
Chapter 5, pg 343:

“In most cases we could have just used MsgRead(), but in the case where this
message is part of a “combine message,” resmgr_msgread() performs the
appropriate “magic” for us.”

I was worried about missing out on the “magic”…I don’t want to break
combine messages in the event that the C lib decides to use one. But in the
implementation you dug up I see no evidence of “magic”. :wink:

Perhaps I can get away with MsgRead() directly…that would be a fine
solution, indeed.

[Mr Krten: Hope you don’t mind that I cc’ed you… I’m curious as to whether
the magic you mentioned is non-existent in the implementation Kevin showed
or just invisible to us mortals… :wink: Would you consider it safe to just
blindly use MsgRead()?]

–Adam

“Kevin Stallard” <kevin@robots.flyingrobots.fly.> wrote in message
news:an9mc2$ffm$1@inn.qnx.com…

Because io_writes are a MsgSend() from a client I belive all you really
need
is the rcvid in order to finish reading data from the client. MsgRead()
is
what you would use with the recvid.

I just verified this by checking the source code for resmgr_msgread() (on
cvs.qnx.com) …it is a wrapper for MsgRead(). The only thing you would
have to do is also track the number of bytes you’ve already read from the
client performing the write().

int resmgr_msgread(resmgr_context_t *ctp, void *msg, int nbytes, int
offset)
{
return MsgRead(ctp->rcvid, msg, nbytes, ctp->offset + offset);
}


Hope this helps

Kevin


“Adam Kropelin” <> adk0212@hotmail.com> > wrote in message
news:an9grh$b9g$> 1@inn.qnx.com> …
Hi, John…

Thanks for your reply. You’ve given me some ideas to think about.

I’ve been unable to access QNX’s developer site for the last several
days.
I’m not sure if the problem is on their end or mine. After IE spins for
a
few minutes I get what appears to be an error from our local proxy
server.

I looked in my local copy of the docs and read the article I think you
were
referencing. Unfortunately, the section on blocking the client is a
little
too simplistic > :wink: > Blocking reads are easy to handle: save the recvid
away
and do the MsgReply yourself at some later time. The client stays
blocked
until you reply. Easy.

But writes, on the other hand… The reason I would be blocking the
client
in that case is that I don’t have room in my internal buffer to hold
however
much data he’s trying to write. (If I have no internal buffer at all,
the
same problem exists, except that I’m waiting for the hardware to become
available.) I want to block him until I have room for the data (wither
in
the hw or my buffer). Saving the recvid alone is not enough for that
case.
You need the whole ctp in order to call resmgr_msgread() when you’re
finally
ready to retrieve his data. Without the ctp, I don’t know what to do.

As for O_NONBLOCK, opening with that flag generally implies that you’ll
fail
the call (EAGAIN) if it would block, not that you would enable some
double-buffering scheme. It would be possible to change the semantics
and
do
as you suggest but I’m not sure that’s the POSIXy way to do it.

Letting the hardware flow-control the system makes a lot of sense, but
even
then I have the same problem. I need to block the writer. I can’t tie up
a
server thread sitting in io_write() because I’ll potentially deadlock
the
driver. (Think what happens when write() gets called from several
clients,
each of which blocks a server thread, and soon there are no threads left
even for read()s.) So I have to return from io_write() while leaving the
client blocked and having not read his data yet. Same problem.

Thanks again for your help. Any further ideas?

–Adam

“John Roul” <> fck8828@yahoo.com> > wrote in message
news:an4rvi$4gh$> 1@inn.qnx.com> …
Hi Adam,

I’m by far not an expert in writing QNX device drivers and am in fact
trying
to learn some of these things as well, I came across an article in the
QNX
literature, see:



http://www.qnx.com/developer/docs/momentics_nc_docs/neutrino/prog/resmgr.htm
l#ReturnComesFromResMgr , specifically the section “Leaving the client
blocked, replying later” which actually describes blocking and
unblocking.
The literature here does imply you can handle the ctp and save and
process
the reply ID anyway you want and it seems the driver is the one that
decides
in fact to block or unblock. I’m not totally clear on your intentions
but
it sounds like your trying to make this driver be as efficient as
possible,
whether you need to really buffer any of the data depends on the rest
of
your system’s architecture > :slight_smile: > Maybe you could give the user of the
driver
the option of opening with O_NONBLOCK and never block, always implying
a
double buffer and worker thread to dispatch it. If they didn’t ask to
be
blocked, just allow the hardware to flow-control the caller of the
driver
service; if it takes a while to return when blocking so be it, as long
as
other parts of the system can handle the back log. Anyways, I think
QNX
lets you do what your asking no matter which way you go.

Hopefully I understood correctly and didn’t miss the point. I’m sure
there
are others out there that can offer more insight, just my 1/2 cents
worth
:wink:

Regards,
John Roul

\

7

Because io_writes are a MsgSend() from a client I belive all you really need
is the rcvid in order to finish reading data from the client. MsgRead() is
what you would use with the recvid.

I just verified this by checking the source code for resmgr_msgread() (on
cvs.qnx.com) …it is a wrapper for MsgRead(). The only thing you would
have to do is also track the number of bytes you’ve already read from the
client performing the write().

int resmgr_msgread(resmgr_context_t *ctp, void *msg, int nbytes, int offset)
{
return MsgRead(ctp->rcvid, msg, nbytes, ctp->offset + offset);
}


Hope this helps

Kevin


“Adam Kropelin” <adk0212@hotmail.com> wrote in message
news:an9grh$b9g$1@inn.qnx.com…

Hi, John…

Thanks for your reply. You’ve given me some ideas to think about.

I’ve been unable to access QNX’s developer site for the last several days.
I’m not sure if the problem is on their end or mine. After IE spins for a
few minutes I get what appears to be an error from our local proxy server.

I looked in my local copy of the docs and read the article I think you
were
referencing. Unfortunately, the section on blocking the client is a little
too simplistic > :wink: > Blocking reads are easy to handle: save the recvid away
and do the MsgReply yourself at some later time. The client stays blocked
until you reply. Easy.

But writes, on the other hand… The reason I would be blocking the client
in that case is that I don’t have room in my internal buffer to hold
however
much data he’s trying to write. (If I have no internal buffer at all, the
same problem exists, except that I’m waiting for the hardware to become
available.) I want to block him until I have room for the data (wither in
the hw or my buffer). Saving the recvid alone is not enough for that case.
You need the whole ctp in order to call resmgr_msgread() when you’re
finally
ready to retrieve his data. Without the ctp, I don’t know what to do.

As for O_NONBLOCK, opening with that flag generally implies that you’ll
fail
the call (EAGAIN) if it would block, not that you would enable some
double-buffering scheme. It would be possible to change the semantics and
do
as you suggest but I’m not sure that’s the POSIXy way to do it.

Letting the hardware flow-control the system makes a lot of sense, but
even
then I have the same problem. I need to block the writer. I can’t tie up a
server thread sitting in io_write() because I’ll potentially deadlock the
driver. (Think what happens when write() gets called from several clients,
each of which blocks a server thread, and soon there are no threads left
even for read()s.) So I have to return from io_write() while leaving the
client blocked and having not read his data yet. Same problem.

Thanks again for your help. Any further ideas?

–Adam

“John Roul” <> fck8828@yahoo.com> > wrote in message
news:an4rvi$4gh$> 1@inn.qnx.com> …
Hi Adam,

I’m by far not an expert in writing QNX device drivers and am in fact
trying
to learn some of these things as well, I came across an article in the
QNX
literature, see:


http://www.qnx.com/developer/docs/momentics_nc_docs/neutrino/prog/resmgr.htm
l#ReturnComesFromResMgr , specifically the section “Leaving the client
blocked, replying later” which actually describes blocking and
unblocking.
The literature here does imply you can handle the ctp and save and
process
the reply ID anyway you want and it seems the driver is the one that
decides
in fact to block or unblock. I’m not totally clear on your intentions
but
it sounds like your trying to make this driver be as efficient as
possible,
whether you need to really buffer any of the data depends on the rest of
your system’s architecture > :slight_smile: > Maybe you could give the user of the
driver
the option of opening with O_NONBLOCK and never block, always implying a
double buffer and worker thread to dispatch it. If they didn’t ask to
be
blocked, just allow the hardware to flow-control the caller of the
driver
service; if it takes a while to return when blocking so be it, as long
as
other parts of the system can handle the back log. Anyways, I think QNX
lets you do what your asking no matter which way you go.

Hopefully I understood correctly and didn’t miss the point. I’m sure
there
are others out there that can offer more insight, just my 1/2 cents
worth
:wink:

Regards,
John Roul