io-blk library dynamic memory usage/memory leak

QNX USERS QNX6 - OS
1

Hi,

I am running my system for long time continuously and monitoring the memory usage of all components running in system. I see the increase in memory usage of devb-eide and other drivers that are using io-blk.so library for block I/O. I think the memory is allocated for vnodes created dynamically whenever i am doing for disk operations. However i want to know whether the increase in memory usage will stop at some point of time and how can i determine when any further increase should stop.?
I also want to know what are the factors that can cause the increase in memory of driver at run time.?

Thanks

2

Omesh,

By default io-blk increases it’s memory use until it reaches 15% of the system RAM. This can be adjusted up/down by a parameter given to io-blk when devb-eide (and other drivers) are started.

You can also pre-allocate all the RAM at start time so that it doesn’t grow dynamically if you prefer to see the max usage.

Take a look at the help doc’s on io-blk.so for more details.

Tim

3

Hello Tim,
Thanks for the reply.
I have set the options of alloc=upfront and cache=64K.Once i set these options i should not see any growth at run time and memory should not grow dynamically beyond 64KB ,but this is not the case happening on my system. Is there any other parameter that i should set in order to completely stop the growth at runtime.?

Thanks,

4

I’m just guessing here, but 64K is a tiny amount of memory (these days) for use as the disk cache. I wouldn’t be surprised if that number is being ignored, or pushed up to some more reasonable minimum.

5

If i am setting the cache=512KB ,i am able to see the growth at run time even though i have set the alloc=upfront in options.what is the reason for this behavior. ?

Thanks,

6

I think you will have to ask on the QNX support boards. The source was never released (as far as I can tell). It might well be an oversight or a bug.

7

I think 512K is still probably WAY under the minimum.

A typical system these days has 1 Gig of Ram. 512K is .0005% of the RAM. Even if you only have 512 Megs of Ram, 512K is .001% or 1/10 of 1%.

I suspect 1% is probably the default minimum percentage so on a 1 Gig system than would be 10 Megs.

Tim

8

Thanks Tim,

My system has 256MB of RAM and 256MB of Flash Drive with four partitions. Even though i am setting cache of 10MB upfront with fixed vonde count,I still see the heap usage of devb-eide driver increasing at run time.
(cmd line options:alloc=upfront,auto=partition,cache=10240k,vnode=200:200).

I am executing following sample program on different partitions to reproduce the increase in heap memory . But i am not able to figure out what could cause this memory increase and what would be the limit when it should stop increasing further.

	for(i=0; i<=1000; i++)       
	{             
		sprintf(filename, "%s%d.txt", "file",i+1);           
		if( ( fp = fopen(filename, "a+") ) == NULL )
		{
			printf("Error: File \"%s\ %d\" cannot be opened\n", filename,errno);
		}
		else
		{
			for(k=0;k<1000;k++)
			{
				fprintf( fp, "%s", "Hello There");
			}
            fclose(fp);
		}
	}

(Basically i am trying to create n number of files and write some data into the file. )
If i am setting the alloc=upfront option i should not see any increase in the memory at run time. ?

Thanks,

9

The option are just for the cache, devb/io-blk may require more memory for other purposes. You are also limiting vnode to 200, but are opening 1000 files. Note sure how it`s handled.

I would also be the memory consumption would stop increasing otherwise any QNX6 system out there would be in serious trouble.

10

Hello mario,

I tried with different values of vnodes and observed the memory usage of the driver after initialization increases as i increase the vnode count(May be more Memory is required for more number of vnodes and additional memory is allocated in initialization itself since i have provided alloc=upfront option ).
I also tried to set the value to default(max) and ran the same test case and i could still see the memory usage increasing at runtime

I am looking for that either i can figure out how to prevent any increase at run time and allocate all the memory during initialization or find out when the increase in memory would stop and for that i should know where is memory is required at run time ?

Thanks,

11

At think at this point you have exhausted all the help this site can offer ;-) You will need to get in touch with QNX

12

I agree with Mario you may have to contact QNX.

Many of our systems stay up for months at a time without re-booting. I have never run out of RAM or seen devb-eide or other drivers use an enormous amount of it. On the other hand none of my systems ever attempts to create 1000 files IN ONE DIRECTORY either :open_mouth:

I do recall a long time ago there was a problem (can’t remember if it was QNX 4 or 6 or both) where putting a large number of files in one directory (a 1000 is a large number) caused some issues. I can’t remember all the details now but maybe Mario or Mitchell will remember.

Tim

13

Well certainly QNX 2 had a problem with a large number of files in a directory. I remember talking with Bill Flowers the improvements he put into the QNX 4 file system to deal with this. I also recall that there was some standard open source used at the time for posting that required huge numbers of files in a directory, which did work quite well with it.