UDP Socket Not Buffering

I have an application running under QNX 4.25 on a Teknor PCI-936 Single
Board Computer with a 233 MHz Pentium (430 HX chipset) using the onboard
Intel 82557 chipset for communications (Net.ether82557 Ver 4.25G
driver). I set up a datagram socket using socket(…SOCK_DGRAM…),
setsockopt(…SO_REUSEADDR…), bind() to bind to a specific port and any
address, and ioctl(…FIONBIO…) to set the socket to non-blocking. I
used getsockopt(…SO_RCVBUF…) to check the size of the receive buffer
and it returned 41600. I then loop forever waiting at a select() for
the socket to indicate that data is present, using a call to recvfrom()
to read a fixed size 897 byte packet from the socket each time, and then
processing the packet and returning back to the top of my loop. Another
system is sending these fixed size packets every approx. 20 msec to the
socket I am reading from.

The problem I am encountering is that only one packet is ever getting
buffered at the socket I am reading from. For example, if I sleep for
one second after reading a packet and then try reading as much data as
possible from the socket there should
be 5 packets received and buffered in that time, but there only ever is
one (the last one received). Can anyone explain why a socket with
sufficient receive buffer space would not be buffering data?


Tim Meakin, P.Eng.
SED Systems

Try setsockopt(… SO_RCVBUFF…) with 32000, it worked for my application. I
guess the internal buffer may use a short integer as buffer size variale???

“Tim Meakin” <meakin@sedsystems.ca> wrote in message
news:3B8D49E9.4011184B@sedsystems.ca

I have an application running under QNX 4.25 on a Teknor PCI-936 Single
Board Computer with a 233 MHz Pentium (430 HX chipset) using the onboard
Intel 82557 chipset for communications (Net.ether82557 Ver 4.25G
driver). I set up a datagram socket using socket(…SOCK_DGRAM…),
setsockopt(…SO_REUSEADDR…), bind() to bind to a specific port and any
address, and ioctl(…FIONBIO…) to set the socket to non-blocking. I
used getsockopt(…SO_RCVBUF…) to check the size of the receive buffer
and it returned 41600. I then loop forever waiting at a select() for
the socket to indicate that data is present, using a call to recvfrom()
to read a fixed size 897 byte packet from the socket each time, and then
processing the packet and returning back to the top of my loop. Another
system is sending these fixed size packets every approx. 20 msec to the
socket I am reading from.

The problem I am encountering is that only one packet is ever getting
buffered at the socket I am reading from. For example, if I sleep for
one second after reading a packet and then try reading as much data as
possible from the socket there should
be 5 packets received and buffered in that time, but there only ever is
one (the last one received). Can anyone explain why a socket with
sufficient receive buffer space would not be buffering data?


Tim Meakin, P.Eng.
SED Systems

jim_nie wrote:

Try setsockopt(… SO_RCVBUFF…) with 32000, it worked for my application. I
guess the internal buffer may use a short integer as buffer size variale???

Thanks for the tip Jim, unfortunately it makes no difference in our
application what we set the SO_RCVBUF size to. We have also tried
updating our socket library to the latest Tcpip update, tried a
different ethernet card and using the Net.ether1000 driver, and tried
playing with the various other parameters settable using the setsockopt
function (e.g., SO_RCVLOWAT, SO_RCVTIMEO, etc.) but with no success.

We still can’t get our UDP socket to buffer more than one message. This
is getting VERY FRUSTRATING as our application must be able to handle
the receipt of bursty data and therefore requires some amount of
buffering to work.

I’m on the only QNX project we have in-house but I’ve discussed this
problem with technical experts using UDP sockets on other projects at
our workplace. They say they have run into a similar problem using
Linux whereby using setsockopt to set the size of the receive buffer for
a UDP socket made no difference unless you first ensured that a couple
of values (i.e., rmem_max and wmem_max) used to set OS memory buffer
sizes were set high enough in /proc/sys/net/core. However, the default
buffer sizes were always at least 8K so some buffering was always
present.

Does anyone know if QNX has some equivalent kernel constants somewhere
that might effect the size of memory allocated to socket buffers?

I have seen some documentation that talks about needing the right “mbuf”
size for UDP sockets. Does anyone know where this size gets set or can
be tweaked in QNX?

\

Tim Meakin, P.Eng.
SED Systems

Tim Meakin <meakin@sedsystems.ca> wrote:

Try setsockopt(… SO_RCVBUFF…) with 32000, it worked for my application. I
guess the internal buffer may use a short integer as buffer size variale???

Thanks for the tip Jim, unfortunately it makes no difference in our
application what we set the SO_RCVBUF size to. We have also tried
updating our socket library to the latest Tcpip update, tried a
different ethernet card and using the Net.ether1000 driver, and tried
playing with the various other parameters settable using the setsockopt
function (e.g., SO_RCVLOWAT, SO_RCVTIMEO, etc.) but with no success.

We still can’t get our UDP socket to buffer more than one message. This
is getting VERY FRUSTRATING as our application must be able to handle
the receipt of bursty data and therefore requires some amount of
buffering to work.

Instead of doing the final recieve in your client program, take a look
at you network stats. Do a “netstat -an” to see the queued amount of
packets for the UDP port you’ve bind()'d. Another place to look is
under “netstat -s” and look at both the IP section and the UDP section
to see if you’re getting the messages in the first place (since this is
UDP).

-Adam

Operating System for Tech Supp wrote:

Instead of doing the final recieve in your client program, take a look
at you network stats. Do a “netstat -an” to see the queued amount of
packets for the UDP port you’ve bind()'d. Another place to look is
under “netstat -s” and look at both the IP section and the UDP section
to see if you’re getting the messages in the first place (since this is
UDP).

-Adam

Thanks for the pointer. We were using “netstat -s” to verify that we
were getting the expected number of “datagrams received” and this was
indicating that the ones we were missing were “dropped due to full
socket buffers”.

The “netstat -an” was useful for observing the size of the receive queue
on our UDP socket. When our socket was left at its default receive
buffer size (getsockopt indicates that this is 41600) then we were
seeing Recv-Q rise very quickly to a maximum of 32868 and then we
started losing packets.

Just to make sure, I retried using setsockopt to set the SO_RCVBUF size
to something larger than the default value (this hadn’t worked before
with the old Sockets library) and it worked this time. Apparently
something has changed with the new version of the Tcpip library we
installed that allows the receive buffer size to be set larger. We
calculated that under bursty conditions we might be receiving up to 85
KB of data in a very short time frame (432 msecs). This was probably
why the 40 KB receive buffer was overflowing. I tried setting SO_RCVBUF
to 128 KB and we now seem to keep up.

I have noticed that the receive buffer size cannot be set higher than
(240 * 1024) using the setsockopt call. Can you tell me what is magic
about this upper limit and can it be increased even further?

One other thing that still presents a problem is that we can never read
more than one datagram packet from our socket at any one time. We tried
using recvfrom(), recv(), and read() and nothing seems to make a
difference. No matter how much data netstat tells us is buffered at the
socket, we have to do individual reads to get each packet. Why can’t a
single read from a UDP socket get all the data that is buffered there?


Tim Meakin, P.Eng.
SED Systems

Tim Meakin <meakin@sedsystems.ca> wrote:

[snip]

One other thing that still presents a problem is that we can never read
more than one datagram packet from our socket at any one time. We tried
using recvfrom(), recv(), and read() and nothing seems to make a
difference. No matter how much data netstat tells us is buffered at the
socket, we have to do individual reads to get each packet. Why can’t a
single read from a UDP socket get all the data that is buffered there?

That’s the defination of “UDP”, each read can only read
one “packet”, all others buffered in socket receive buffer.
Not like tcp, udp is not a stream.

-xtang