I have a question regarding the scheduler. I have a process with about 350
threads. Of these threads only about 15 are actively communicating reading
data from a server via IPC and relaying that message to a client on a TCP
socket. The others are waiting for messages from another process via IPC to
send this data to the same client via TCP socket. The memory utilization is
near 75% when viewed by spin. It appears that procnto (the kernel?) is
using 45% of the CPU time.
Can anyone tell me if the scheduling overhead is proportional to the number
of threads?
Is there anything that can be done to decrease the impact of having this
many threads in the system?
Thanks in advance.
Tim
“Tim Bochenek” <tim.bochenek@bepco.com> wrote in
news:a6o5km$lc0$1@inn.qnx.com:
I have a question regarding the scheduler. I have a process with about
350 threads. Of these threads only about 15 are actively communicating
reading data from a server via IPC and relaying that message to a
client on a TCP socket. The others are waiting for messages from
another process via IPC to send this data to the same client via TCP
socket. The memory utilization is near 75% when viewed by spin. It
appears that procnto (the kernel?) is using 45% of the CPU time.
Can anyone tell me if the scheduling overhead is proportional to the
number of threads?
Is there anything that can be done to decrease the impact of having
this many threads in the system?
Are you using a threadpool or just creating 350 threads?
\
Cheers,
Adam
With a PC, I always felt limited by the software available.
On Unix, I am limited only by my knowledge.
–Peter J. Schoenster <pschon@baste.magibox.net>
I am using a thread pool.
“Adam Mallory” <amallory@qnx.com> wrote in message
news:Xns91D1799B84A16amalloryqnxcom@209.226.137.4…
“Tim Bochenek” <> tim.bochenek@bepco.com> > wrote in
news:a6o5km$lc0$> 1@inn.qnx.com> :
I have a question regarding the scheduler. I have a process with about
350 threads. Of these threads only about 15 are actively communicating
reading data from a server via IPC and relaying that message to a
client on a TCP socket. The others are waiting for messages from
another process via IPC to send this data to the same client via TCP
socket. The memory utilization is near 75% when viewed by spin. It
appears that procnto (the kernel?) is using 45% of the CPU time.
Can anyone tell me if the scheduling overhead is proportional to the
number of threads?
Is there anything that can be done to decrease the impact of having
this many threads in the system?
Are you using a threadpool or just creating 350 threads?
\
Cheers,
Adam
QNX Software Systems Ltd.
[ > amallory@qnx.com > ]
With a PC, I always felt limited by the software available.
On Unix, I am limited only by my knowledge.
–Peter J. Schoenster <> pschon@baste.magibox.net
“Tim Bochenek” <tim.bochenek@bepco.com> wrote in
news:a6qmb4$hi8$1@inn.qnx.com:
I am using a thread pool.
What are the high/low levels you’re using? A thread pool has some overhead
in terms of creating/killing threads to maintain the pool levels.
\
Cheers,
Adam
With a PC, I always felt limited by the software available.
On Unix, I am limited only by my knowledge.
–Peter J. Schoenster <pschon@baste.magibox.net>
The thread pool has a low water value of 10, high water value of 20,
increment of 5, and a maximum or 500. The thread pool is effectively static
after the server has started handling requests. A client connects via
TCP/IP and the connection is maintained. There is a separate thread for
each connection. The connections do not come and go unless there is an
error in the communication. In this case the client (or server) would close
the connection and the client would retry. There are about 15 threads
serving data to the client at 10 times per second while the other 330 are
blocked on IPC channels waiting for messages to come in and be forwarded to
the client. I am pretty sure that the problem lies in the scheduling of all
these threads. If I create a process with 15 threads, I get a usage of
about 15%, and the usage increases a little more than linearly as the number
or idle threads are added. These are threads that are just blocked. Is the
number of threads going to affect the scheduling that dramatically since
they are all scheduled globally? Does the scheduler punish the system that
much for having so many threads?
Thanks,
Tim
“Adam Mallory” <amallory@qnx.com> wrote in message
news:Xns91D262910F36Bamalloryqnxcom@209.226.137.4…
“Tim Bochenek” <> tim.bochenek@bepco.com> > wrote in
news:a6qmb4$hi8$> 1@inn.qnx.com> :
I am using a thread pool.
What are the high/low levels you’re using? A thread pool has some
overhead
in terms of creating/killing threads to maintain the pool levels.
\
Cheers,
Adam
QNX Software Systems Ltd.
[ > amallory@qnx.com > ]
With a PC, I always felt limited by the software available.
On Unix, I am limited only by my knowledge.
–Peter J. Schoenster <> pschon@baste.magibox.net
“Tim Bochenek” <tim.bochenek@bepco.com> wrote in
news:a74uic$q00$1@inn.qnx.com:
The thread pool has a low water value of 10, high water value of 20,
increment of 5, and a maximum or 500. The thread pool is effectively
static after the server has started handling requests. A client
connects via TCP/IP and the connection is maintained. There is a
separate thread for each connection. The connections do not come and
go unless there is an error in the communication. In this case the
client (or server) would close the connection and the client would
retry. There are about 15 threads serving data to the client at 10
times per second while the other 330 are blocked on IPC channels
waiting for messages to come in and be forwarded to the client. I am
pretty sure that the problem lies in the scheduling of all these
threads. If I create a process with 15 threads, I get a usage of about
15%, and the usage increases a little more than linearly as the number
or idle threads are added. These are threads that are just blocked.
Is the number of threads going to affect the scheduling that
dramatically since they are all scheduled globally? Does the scheduler
punish the system that much for having so many threads?
Perhaps I’m mis-understanding, you have a thread pool w/ a low of 10, and
high of 20 and a max of 500? You could be paying the price that once there
are threads waiting over the high mark, we start killing them off, and then
you get another request and then new threads are created. This type of
oscillation could be causing the “cpu usage” where creation and destruction
are occuring over and over.
Perhaps bringing the highwater or max in closer to each other (I doubt you
want 400+ threads waiting around, each thread does take up resources in
terms of stack, thread spec. info etc)
\
Cheers,
Adam
With a PC, I always felt limited by the software available.
On Unix, I am limited only by my knowledge.
–Peter J. Schoenster <pschon@baste.magibox.net>
I am pretty sure that the threads never exit the Handle phase. Once a
thread is created it never dies. I agree that the overhead of managing the
thread pool could be an issue. This is why I don’t need many threads
waiting but have a need for many threads. The client will make connections
that stay persistent for the lifetime of the process. There are not many
threads in the blocking phase, but there are many in the Handling phase.
Thanks,
Tim
“Adam Mallory” <amallory@qnx.com> wrote in message
news:Xns91D68A6517B61amalloryqnxcom@209.226.137.4…
“Tim Bochenek” <> tim.bochenek@bepco.com> > wrote in
news:a74uic$q00$> 1@inn.qnx.com> :
The thread pool has a low water value of 10, high water value of 20,
increment of 5, and a maximum or 500. The thread pool is effectively
static after the server has started handling requests. A client
connects via TCP/IP and the connection is maintained. There is a
separate thread for each connection. The connections do not come and
go unless there is an error in the communication. In this case the
client (or server) would close the connection and the client would
retry. There are about 15 threads serving data to the client at 10
times per second while the other 330 are blocked on IPC channels
waiting for messages to come in and be forwarded to the client. I am
pretty sure that the problem lies in the scheduling of all these
threads. If I create a process with 15 threads, I get a usage of about
15%, and the usage increases a little more than linearly as the number
or idle threads are added. These are threads that are just blocked.
Is the number of threads going to affect the scheduling that
dramatically since they are all scheduled globally? Does the scheduler
punish the system that much for having so many threads?
Perhaps I’m mis-understanding, you have a thread pool w/ a low of 10, and
high of 20 and a max of 500? You could be paying the price that once
there
are threads waiting over the high mark, we start killing them off, and
then
you get another request and then new threads are created. This type of
oscillation could be causing the “cpu usage” where creation and
destruction
are occuring over and over.
Perhaps bringing the highwater or max in closer to each other (I doubt you
want 400+ threads waiting around, each thread does take up resources in
terms of stack, thread spec. info etc)
\
Cheers,
Adam
QNX Software Systems Ltd.
[ > amallory@qnx.com > ]
With a PC, I always felt limited by the software available.
On Unix, I am limited only by my knowledge.
–Peter J. Schoenster <> pschon@baste.magibox.net
“Tim Bochenek” <tim.bochenek@bepco.com> wrote in
news:a783i6$60n$1@inn.qnx.com:
I am pretty sure that the threads never exit the Handle phase. Once a
thread is created it never dies. I agree that the overhead of managing
the thread pool could be an issue. This is why I don’t need many
threads waiting but have a need for many threads. The client will make
connections that stay persistent for the lifetime of the process.
There are not many threads in the blocking phase, but there are many in
the Handling phase.
Can you post the output of spin before you run you process, after you run
your process and about 1 hour after (to get the normalized state).
Also, can post the specs on the system you’re running on (ie. CPU/RAM etc
etc). Why do you feel there is a scheduling impact? Are you seeing
slowdown or threads not being scheduled correctly?
\
Cheers,
Adam
With a PC, I always felt limited by the software available.
On Unix, I am limited only by my knowledge.
–Peter J. Schoenster <pschon@baste.magibox.net>