I kind of stumbled into discovering some rather strange behavior
regarding Fsys and its cash size. It seems there is an upper limit to
how big a cash Fsys can handle effectively. Once that threshold has
been crossed, Fsys through-put will drop drastically, apparently by a
factor of 20 or more.
Here’s the background…
I’ve been doing some development and testing to try to alleviate peak
load levels on our production systems when large database updates and/or
refreshes are being run. My cohorts and I have suspected for quite a
while that we were becoming disk bound and from watching sac output when
DB updates were in progress, it appeared that we were also becoming CPU
bound. So the plan was to figure out a way to selectively off load
some of our larger database intensive processing to an auxiliary node.
To that end, we built a test node and thinking that it will be needing
all the resources it could possibly use, we installed 2G of RAM in it.
While testing database updates on this new test node, I observed that a
batched update would start running at a rate of ~2000 trans/sec, and
then at some point would slow to ~100 trans/sec with the CPU pegged at
the priority level of the DB engine process. Once that “slow” state
was reached, it seemed there was no predicting when “normal” through-put
would be restored. For smaller batches of updates (10,000 records or
so), an entire batch might run at normal levels, or there might be
several slow downs lasting anywhere from 10 seconds to several minutes.
For large batch updates (500,000 records or more), at some point the
through-put slowdown would just lock in for hours at a time. Then after
just sitting there idle (like over night) through-put would return to
Suspecting that there might possibly be something wrong with the
hardware on the new test node, I tried the same set of tests on several
of our other test nodes. Although I could get the slowed through-put to
happen, it would never last for more than 20 seconds. How strange,
thought I… The only real difference between the new test node and the
older ones was that the older nodes had only 1G of RAM… This got me
thinking about Fsys & its cash allocation, which was by default 8% of
system memory i.e. 160M for the new test node and 80M for the older ones.
Also, studying sac output, I noticed that when things were running
normally, the CPU was only occasionally pegged, and there would be a
regular blip at priority 22 (Fsys doing its private business) and also
regular hardware interrupt activity. When things slowed down, the CPU
would be pegged at the priority of the DB engine (priority 9 in this
case), with nearly no activity at any other priority level.
So… to make a long story shorter, I started playing around with Fsys
cash allocation, and 64M seems to be the magic number. Allocating any
more than that will cause the slow down symptoms to start appearing.
The more you allocate, the sooner, more frequent and longer the
slowdowns will be. Allocating less than 64M seems to work just fine,
BUT the lower the cash size the more frequently Fsys will be running at
priority 22 and agregate through-put will drop slightly (a few seconds
per 100,000 records).
I think I may have found a significant and rather easy (for a change
solution for helping to alleviate our production system woes… They
also have 2G of RAM in them, and therefore the default 160M Fsys cash.
The same tests on the new test node w/ the Fsys cash set at 64M, now
very consistently run at a rate of over 2000 trans/sec, the CPU is never
pegged and I have not observed any slowdown symptoms.
Obviously, somebody at QNX should look into this further, but for now,
my only remaining concern is…
Is there any foreseeable down side to setting the Fsys cash to 64M?
For example, that shouldn’t affect heap allocation (the -H diskN option)
for large file systems, should it?