/proc/<pid>/as vs. pidin mem sizes

QNX Newsgroups (Archives) qdn.public.qnxrtp.os
1

Or, alternate title, where has all the memory gone?

I’m writing the /proc filesystem chapter, and have a question.

I’ve picked a random process, “pipe” (pid 4105) as my guinnea-pig.

ls -l /proc/4105/as shows the size as 1,236,992 bytes.

pidin mem shows:

pid tid name prio STATE code data stack
4105 1 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
4105 2 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
4105 4 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
4105 5 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
ldqnx.so.2 @b0300000 312K 16K

Doing the math, I get 16kx4 + 148kx4 + 4kx4 + 312k + 16k = 1000k (1,048,576 bytes)
Where are the other almost 200k?

I presume that the stack is grown on demand, so I’m correct to use “4kx4” rather
than “132kx4” as the stack size, right? (Otherwise, the number would be
16kx4 + 148kx4 + 132kx4 + 312k + 16k = 1512k (1,585,447 bytes), or about 250k too
much…)

Any explanations?

Thanks in advance,
-RK

–
[If replying via email, you’ll need to click on the URL that’s emailed to you
afterwards to forward the email to me – spam filters and all that]
Robert Krten, PDP minicomputer collector http://www.parse.com/~pdp8/

2

Robert Krten wrote:

Or, alternate title, where has all the memory gone?

I’m writing the /proc filesystem chapter, and have a question.

I’ve picked a random process, “pipe” (pid 4105) as my guinnea-pig.

ls -l /proc/4105/as shows the size as 1,236,992 bytes.

pidin mem shows:

pid tid name prio STATE code data stack
4105 1 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
4105 2 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
4105 4 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
4105 5 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
ldqnx.so.2 @b0300000 312K 16K

Doing the math, I get 16kx4 + 148kx4 + 4kx4 + 312k + 16k = 1000k (1,048,576 bytes)
Where are the other almost 200k?

I presume that the stack is grown on demand, so I’m correct to use “4kx4” rather
than “132kx4” as the stack size, right? (Otherwise, the number would be
16kx4 + 148kx4 + 132kx4 + 312k + 16k = 1512k (1,585,447 bytes), or about 250k too
much…)

Any explanations?

The code is shared between threads (so no 16Kx4), so is the data (so no 148Kx4).
That means 132Kx4 + 16K + 312K + 16K = 872K. I don’t know why ls -l
/proc/4105/as is different, I’m not even sure that the filesize of “as” should
represent the size of the address space (admittedly it seems intuitive that it
would).

Personally, I feel a little better about a pipe process that is only 250K or so
bigger than the entire QNX2 operating system running 20 or so processes, than I
do about it being almost twice the size :wink:

3

Robert Krten <rk@parse.com> wrote:

Or, alternate title, where has all the memory gone?

I’m writing the /proc filesystem chapter, and have a question.

I’ve picked a random process, “pipe” (pid 4105) as my guinnea-pig.

ls -l /proc/4105/as shows the size as 1,236,992 bytes.

pidin mem shows:

pid tid name prio STATE code data stack
4105 1 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
4105 2 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
4105 4 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
4105 5 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
ldqnx.so.2 @b0300000 312K 16K

Doing the math, I get 16kx4 + 148kx4 + 4kx4 + 312k + 16k = 1000k (1,048,576 bytes)
Where are the other almost 200k?

I presume that the stack is grown on demand, so I’m correct to use “4kx4” rather
than “132kx4” as the stack size, right? (Otherwise, the number would be
16kx4 + 148kx4 + 132kx4 + 312k + 16k = 1512k (1,585,447 bytes), or about 250k too
much…)

Any explanations?

Anyone? I’d hate to put “but noone seems to know why” in the book… :slight_smile:

Cheers,
-RK

–
[If replying via email, you’ll need to click on the URL that’s emailed to you
afterwards to forward the email to me – spam filters and all that]
Robert Krten, PDP minicomputer collector http://www.parse.com/~pdp8/

4

Rennie Allen <rgallen@attbi.com> wrote:

Robert Krten wrote:
Or, alternate title, where has all the memory gone?

I’m writing the /proc filesystem chapter, and have a question.

I’ve picked a random process, “pipe” (pid 4105) as my guinnea-pig.

ls -l /proc/4105/as shows the size as 1,236,992 bytes.

pidin mem shows:

pid tid name prio STATE code data stack
4105 1 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
4105 2 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
4105 4 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
4105 5 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
ldqnx.so.2 @b0300000 312K 16K

Doing the math, I get 16kx4 + 148kx4 + 4kx4 + 312k + 16k = 1000k (1,048,576 bytes)
Where are the other almost 200k?

I presume that the stack is grown on demand, so I’m correct to use “4kx4” rather
than “132kx4” as the stack size, right? (Otherwise, the number would be
16kx4 + 148kx4 + 132kx4 + 312k + 16k = 1512k (1,585,447 bytes), or about 250k too
much…)

Any explanations?

The code is shared between threads (so no 16Kx4), so is the data (so no 148Kx4).

Correct; thanks :slight_smile:

That means 132Kx4 + 16K + 312K + 16K = 872K. I don’t know why ls -l
/proc/4105/as is different, I’m not even sure that the filesize of “as” should
represent the size of the address space (admittedly it seems intuitive that it
would).

Well, one of them must be wrong. Like the Dire Straits song goes, “there are
two Jesus’s on the corner, one of them must be wrong” :slight_smile:


Personally, I feel a little better about a pipe process that is only 250K or so
bigger than the entire QNX2 operating system running 20 or so processes, than I
do about it being almost twice the size > :wink:

Cheers,
-RK

–
[If replying via email, you’ll need to click on the URL that’s emailed to you
afterwards to forward the email to me – spam filters and all that]
Robert Krten, PDP minicomputer collector http://www.parse.com/~pdp8/

5

Hi Rennie,

strip…

Rennie Allen a écrit:

The code is shared between threads (so no 16Kx4), so is the data (so
no 148Kx4). That means 132Kx4 + 16K + 312K + 16K = 872K.

I’m agree about the code shared between threads but I think that every
threads has its set of data no?

Alain.

6

“Robert Krten” <rk@parse.com> wrote in message
news:biigat$ceh$2@inn.qnx.com…
[…]

That means 132Kx4 + 16K + 312K + 16K = 872K. I don’t know why ls -l
/proc/4105/as is different, I’m not even sure that the filesize of “as”
should
represent the size of the address space (admittedly it seems intuitive
that it
would).

Well, one of them must be wrong. Like the Dire Straits song goes, “there
are
two Jesus’s on the corner, one of them must be wrong” > :slight_smile:

Unless one of them is just great and another is truly, truly wonderful. Then
they can both be right, like catholics & protestants.

Personally, I feel a little better about a pipe process that is only
250K or so
bigger than the entire QNX2 operating system running 20 or so processes,
than I
do about it being almost twice the size > :wink:

Measuring size of the process that way is not very helpful. Take a look at
what spin reports for size of the ‘pipe’. You need to read and interpret all
the memory mappings for a given process using certain devctls() on
/proc/pid/as. There’s some black magic involved.

As of why ls -l shows rather different size of ‘as’, I am not sure. But
intuitively, that could mean that address space reserved for a process can
be a lot bigger than the actual amount of memory consumed. This can be due
to many reasons, some of which could be quite logical and others could be
purely implementation-imposed.

– igor

7

The code for pidin is available. Did you take a look and see how it comes
up with the numbers? It might provide some insight.

cheers,

Kris

“Robert Krten” <rk@parse.com> wrote in message
news:biig0n$ceh$1@inn.qnx.com…

Robert Krten <> rk@parse.com> > wrote:
Or, alternate title, where has all the memory gone?

I’m writing the /proc filesystem chapter, and have a question.

I’ve picked a random process, “pipe” (pid 4105) as my guinnea-pig.

ls -l /proc/4105/as shows the size as 1,236,992 bytes.

pidin mem shows:

pid tid name prio STATE code data
stack
4105 1 sbin/pipe 10o RECEIVE 16K 148K
4096(132K)
4105 2 sbin/pipe 10o RECEIVE 16K 148K
4096(132K)
4105 4 sbin/pipe 10o RECEIVE 16K 148K
4096(132K)
4105 5 sbin/pipe 10o RECEIVE 16K 148K
4096(132K)
ldqnx.so.2 @b0300000 312K 16K

Doing the math, I get 16kx4 + 148kx4 + 4kx4 + 312k + 16k = 1000k
(1,048,576 bytes)
Where are the other almost 200k?

I presume that the stack is grown on demand, so I’m correct to use
“4kx4” rather
than “132kx4” as the stack size, right? (Otherwise, the number would be
16kx4 + 148kx4 + 132kx4 + 312k + 16k = 1512k (1,585,447 bytes), or about
250k too
much…)

Any explanations?

Anyone? I’d hate to put “but noone seems to know why” in the book… > :slight_smile:

Cheers,
-RK

–
[If replying via email, you’ll need to click on the URL that’s emailed to
you
afterwards to forward the email to me – spam filters and all that]
Robert Krten, PDP minicomputer collector > http://www.parse.com/~pdp8/

8

“Alain Bonnefoy” <alain.bonnefoy@icbt.com> wrote in message
news:3F4D94B0.9050004@icbt.com…

Hi Rennie,

strip…

Rennie Allen a écrit:

The code is shared between threads (so no 16Kx4), so is the data (so
no 148Kx4). That means 132Kx4 + 16K + 312K + 16K = 872K.

I’m agree about the code shared between threads but I think that every
threads has its set of data no?

No, only a different stack.

That’s one different between thread and fork, with fork code is share but
not data.

Alain.

9

Kris Warkentin <kewarken@qnx.com> wrote:

The code for pidin is available. Did you take a look and see how it comes
up with the numbers? It might provide some insight.

It sums up the memory blocks returned by the devctl() to the /proc
file. I guess the real question, in that case, would be, how is the
/proc//as size determined? Igor may have hit the nail on the
head when he said that there is more allocated than is used; that
seems reasonable…

Cheers,
-RK

cheers,

Kris

“Robert Krten” <> rk@parse.com> > wrote in message
news:biig0n$ceh$> 1@inn.qnx.com> …
Robert Krten <> rk@parse.com> > wrote:
Or, alternate title, where has all the memory gone?

I’m writing the /proc filesystem chapter, and have a question.

I’ve picked a random process, “pipe” (pid 4105) as my guinnea-pig.

ls -l /proc/4105/as shows the size as 1,236,992 bytes.

pidin mem shows:

pid tid name prio STATE code data
stack
4105 1 sbin/pipe 10o RECEIVE 16K 148K
4096(132K)
4105 2 sbin/pipe 10o RECEIVE 16K 148K
4096(132K)
4105 4 sbin/pipe 10o RECEIVE 16K 148K
4096(132K)
4105 5 sbin/pipe 10o RECEIVE 16K 148K
4096(132K)
ldqnx.so.2 @b0300000 312K 16K

Doing the math, I get 16kx4 + 148kx4 + 4kx4 + 312k + 16k = 1000k
(1,048,576 bytes)
Where are the other almost 200k?

I presume that the stack is grown on demand, so I’m correct to use
“4kx4” rather
than “132kx4” as the stack size, right? (Otherwise, the number would be
16kx4 + 148kx4 + 132kx4 + 312k + 16k = 1512k (1,585,447 bytes), or about
250k too
much…)

Any explanations?

Anyone? I’d hate to put “but noone seems to know why” in the book… > :slight_smile:

Cheers,
-RK

–
[If replying via email, you’ll need to click on the URL that’s emailed to
you
afterwards to forward the email to me – spam filters and all that]
Robert Krten, PDP minicomputer collector > http://www.parse.com/~pdp8/

–
[If replying via email, you’ll need to click on the URL that’s emailed to you
afterwards to forward the email to me – spam filters and all that]
Robert Krten, PDP minicomputer collector http://www.parse.com/~pdp8/

10

Robert Krten <rk@parse.com> wrote:

It sums up the memory blocks returned by the devctl() to the /proc
file. I guess the real question, in that case, would be, how is the
/proc//as size determined? Igor may have hit the nail on the
head when he said that there is more allocated than is used; that
seems reasonable…

Most definitely. Beside the fact of MAP_LAZY mappings, consider
this:

first mmap returns 4K as the address to use
second mmap return 1M as the address to use

the /proc//as file will have to be 1M (plus the size of the second
mmap) in size, but only a couple of pages of physical memory will
actually be needed/used. This sort of thing happens all the
time because of shared objects.

–
Brian Stecher (bstecher@qnx.com) QNX Software Systems, Ltd.
phone: +1 (613) 591-0931 (voice) 175 Terence Matthews Cr.
+1 (613) 591-3579 (fax) Kanata, Ontario, Canada K2M 1W8

11

Of course!
cheers,
Alain.

Mario Charest a écrit:

“Alain Bonnefoy” <> alain.bonnefoy@icbt.com> > wrote in message
news:> 3F4D94B0.9050004@icbt.com> …


Hi Rennie,

strip…

Rennie Allen a écrit:



The code is shared between threads (so no 16Kx4), so is the data (so
no 148Kx4). That means 132Kx4 + 16K + 312K + 16K = 872K.


I’m agree about the code shared between threads but I think that every
threads has its set of data no?



No, only a different stack.

That’s one different between thread and fork, with fork code is share but
not data.



Alain.



\

12

bstecher@qnx.com wrote:

Robert Krten <> rk@parse.com> > wrote:
It sums up the memory blocks returned by the devctl() to the /proc
file. I guess the real question, in that case, would be, how is the
/proc//as size determined? Igor may have hit the nail on the
head when he said that there is more allocated than is used; that
seems reasonable…

Most definitely. Beside the fact of MAP_LAZY mappings, consider
this:

first mmap returns 4K as the address to use
second mmap return 1M as the address to use

the /proc//as file will have to be 1M (plus the size of the second
mmap) in size, but only a couple of pages of physical memory will
actually be needed/used. This sort of thing happens all the
time because of shared objects.

Hold on a second – you’re saying that the /proc//as file is
NOT a sparse file? (If I’ve understood your example, there are
two distinct mmap’d regions, say pretty small, but offset by 1M
from each other…) What does this say about the syspage data area
at the top, surely that’s a disjoint section of address space?

Thanks,
-RK

–
Brian Stecher (> bstecher@qnx.com> ) QNX Software Systems, Ltd.
phone: +1 (613) 591-0931 (voice) 175 Terence Matthews Cr.
+1 (613) 591-3579 (fax) Kanata, Ontario, Canada K2M 1W8

–
[If replying via email, you’ll need to click on the URL that’s emailed to you
afterwards to forward the email to me – spam filters and all that]
Robert Krten, PDP minicomputer collector http://www.parse.com/~pdp8/

13

Robert Krten <rk@parse.com> wrote:

Hold on a second – you’re saying that the /proc//as file is
NOT a sparse file? (If I’ve understood your example, there are
two distinct mmap’d regions, say pretty small, but offset by 1M
from each other…) What does this say about the syspage data area
at the top, surely that’s a disjoint section of address space?

It doesn’t mean anything to talk about “as” being a sparse file or
not. The “as” file doesn’t take up space - it merely reflects the
virtual address space that’s been allocated to a process via mmap calls.

\

Brian Stecher (bstecher@qnx.com) QNX Software Systems, Ltd.
phone: +1 (613) 591-0931 (voice) 175 Terence Matthews Cr.
+1 (613) 591-3579 (fax) Kanata, Ontario, Canada K2M 1W8

14

Igor Kovalenko <kovalenko@attbi.com> wrote:

“Robert Krten” <> rk@parse.com> > wrote in message
news:biigat$ceh$> 2@inn.qnx.com> …
[…]
That means 132Kx4 + 16K + 312K + 16K = 872K. I don’t know why ls -l
/proc/4105/as is different, I’m not even sure that the filesize of “as”
should
represent the size of the address space (admittedly it seems intuitive
that it
would).

Well, one of them must be wrong. Like the Dire Straits song goes, “there
are
two Jesus’s on the corner, one of them must be wrong” > :slight_smile:


Unless one of them is just great and another is truly, truly wonderful. Then
they can both be right, like catholics & protestants.

:slight_smile:

Personally, I feel a little better about a pipe process that is only
250K or so
bigger than the entire QNX2 operating system running 20 or so processes,
than I
do about it being almost twice the size > :wink:

Measuring size of the process that way is not very helpful. Take a look at
what spin reports for size of the ‘pipe’. You need to read and interpret all
the memory mappings for a given process using certain devctls() on
/proc/pid/as. There’s some black magic involved.

As of why ls -l shows rather different size of ‘as’, I am not sure. But
intuitively, that could mean that address space reserved for a process can
be a lot bigger than the actual amount of memory consumed. This can be due
to many reasons, some of which could be quite logical and others could be
purely implementation-imposed.

Since I’m going through this in great and horrible detail for the book, here
are the current readings…

pidin -p4105 mem

pid tid name prio STATE code data stack
4105 1 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
4105 2 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
4105 4 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
4105 5 sbin/pipe 10o RECEIVE 16K 148K 4096(132K)
ldqnx.so.2 @b0300000 312K 16K

procfs (output editted)

Info from DCMD_PROC_PAGEDATA
Buff# --vaddr— —size— —flags-- —dev---- —ino----
[ 0] 0x07F22000 0x00001000 0x01001083 0x00000002 0x00000001
[ 1] 0x07F23000 0x0001F000 0x01001783 0x00000002 0x00000001
[ 2] 0x07F42000 0x00001000 0x01401783 0x00000002 0x00000001

[ 3] 0x07F43000 0x00001000 0x01001083 0x00000002 0x00000001
[ 4] 0x07F44000 0x0001F000 0x01001783 0x00000002 0x00000001
[ 5] 0x07F63000 0x00001000 0x01401783 0x00000002 0x00000001

[ 6] 0x07F64000 0x00001000 0x01001083 0x00000002 0x00000001
[ 7] 0x07F65000 0x0001F000 0x01001783 0x00000002 0x00000001
[ 8] 0x07F84000 0x00001000 0x01401783 0x00000002 0x00000001

[ 9] 0x07FA6000 0x00001000 0x01001083 0x00000002 0x00000001
[ 10] 0x07FA7000 0x0001F000 0x01001783 0x00000002 0x00000001
[ 11] 0x07FC6000 0x00001000 0x01401783 0x00000002 0x00000001

[ 12] 0x07FC7000 0x00001000 0x01001083 0x00000002 0x00000001
[ 13] 0x07FC8000 0x0007E000 0x01001383 0x00000002 0x00000001
[ 14] 0x08046000 0x00002000 0x01401383 0x00000002 0x00000001

[ 15] 0x08048000 0x00004000 0x00400571 0x00000001 0x00000009
[ 16] 0x0804C000 0x00001000 0x01400372 0x00000001 0x00000009
[ 17] 0x0804D000 0x00024000 0x01400303 0x00000002 0x00000001

[ 18] 0xB0300000 0x0004E000 0x00410561 0x00000004 0xB0300000
[ 19] 0xB034E000 0x00004000 0x01400772 0x00000004 0xB0300000

Ok, buffers 0, 3, 6, and 9 are the 4k stack in use by each thread.
Buffers 1, 4, 7, and 10 are the allocated (but not used) stack for each thread.
Buffers 2, 5, 8, and 11 are the 4k guard page for each stack.

Buffers 12, 13, and 14 are MAP_STACK… Prolly for the dead thread (thread 3).
(Bit 0x0400 might be PROT_EXEC, turned on for the first 4 stacks and off for the fifth…???)

Buffers 18 and 19 are the shared library (code and data).

What can we make of buffers 15, 16, and 17?
Buffer 15 is EXEC/READable, prolly the code (0x4000 matches pidin mem output)
Buffer 16 is 4k, r/w, mapped private… ?
Buffer 17 is r/w mapped shared AND private ???

Summing up all the memory shown by DCMD_PROC_PAGEDATA yields:
0x1000 x 10 + 0x1f000 x 4 + 0x7e000 + 0x2000 + 0x4000 + 0x24000 + 0x4e000 + 0x4000 = 0x180000 (1.5MB)
So, it’s bigger than the “as” file.

Any insights?

Cheers,
-RK

–
[If replying via email, you’ll need to click on the URL that’s emailed to you
afterwards to forward the email to me – spam filters and all that]
Robert Krten, PDP minicomputer collector http://www.parse.com/~pdp8/

15

Hi Robert

Just to confirm, you are still writing this new book?

You were suggesting a week or two ago that there wasn’t enough demand.

I hope you do write it. It looks like it will be useful. And it will
promote the QNX community.

16

Bill Caroselli <qtps@earthlink.net> wrote:

Hi Robert

Just to confirm, you are still writing this new book?

Yes.

You were suggesting a week or two ago that there wasn’t enough demand.

Marketing support, rather than demand was/is the main issue.

I hope you do write it. It looks like it will be useful. And it will
promote the QNX community.

http://www.parse.com/products/books/book_v2/index.html

362 pages and counting.

:slight_smile:

–
[If replying via email, you’ll need to click on the URL that’s emailed to you
afterwards to forward the email to me – spam filters and all that]
Robert Krten, PDP minicomputer collector http://www.parse.com/~pdp8/

17

Hi RK…



Robert Krten wrote:

http://www.parse.com/products/books/book_v2/index.html

362 pages and counting.

Great chapter! It seems that your book will become a classic very early
on. If you have more chapters, perhaps you can generate a second
volume? Hope so.

Regards…

Miguel.


:slight_smile:

18

Miguel Simon <simon@ou.edu> wrote:

Hi RK…



Robert Krten wrote:

http://www.parse.com/products/books/book_v2/index.html

362 pages and counting.



Great chapter! It seems that your book will become a classic very early
on. If you have more chapters, perhaps you can generate a second
volume? Hope so.

I’m starting to run out of juice :slight_smile: It’s at 372 today.
It should be just over 400 by completion.

Cheers,
-RK

\

[If replying via email, you’ll need to click on the URL that’s emailed to you
afterwards to forward the email to me – spam filters and all that]
Robert Krten, PDP minicomputer collector http://www.parse.com/~pdp8/