There are 663 files in various sub-directories on a normal
QNX4 filesystem under /home/wrm/somedir. If I run either
of the commands
find /home/wrm/somedir -type f
or
ls -Rl /home/wrm/somedir
I get a long list of files reported within a second
And if I do
find /home/wrm/somedir -type f | xargs ls -l
It takes about 2 seconds.
However, if I type
find /home/wrm/somedir -type f -exec /bin/ls -l {} ;
It takes 40 seconds to list all the files.
I imagine that /bin/ls is being read from disk and executed
663 times in the last example, whereas in the other examples
it executes many times less. But the docs for devb-eide say
that its cache defaults to 15% of system ram (we have 256MB)
so ls should be chached. We use the default system
startup/OS-image; devb-eide is reported in “sin args” as:
devb-eide blk auto=partition dos exec=all cam quiet
Can anyone explain this to me? Is there a way to speed
it up without changing hardware?
Note: I don’t really want to execute this command, it
just illustrates some problems with script execution speed
we encountered reading values from resources managed by our
own resource manager. As the same behaviour is noticed with
system filesystems and commands, we can rule out our RM code.
Thanks in advance
William Morris
William Morris <william@bangel.demon.co.uk> wrote:
find /home/wrm/somedir -type f -exec /bin/ls -l {} ;
You’re ignoring many issues here, such as process creation
(each file will create a new ls process), runtime linking
(ls against libc.so) and pathname management (ls will open()
its argument file). These are expensive operations, which
don’t need to be done in the more optimised find/ls variants,
and none would be impacted by any amount of disk cache
(actually the filesystem doesn’t go out of its way to cache
executables, on the grounds that procnto should be caching
mmap’d text regions). I doubt the filesystem itself is at
all a bottleneck here (especially as in the other cases when
you just let it get on with it it apparently runs well) …
John Garvey wrote:
William Morris <> william@bangel.demon.co.uk> > wrote:
find /home/wrm/somedir -type f -exec /bin/ls -l {} ;
You’re ignoring many issues here, such as process creation
(each file will create a new ls process), runtime linking
(ls against libc.so) and pathname management (ls will open()
its argument file). These are expensive operations, which
don’t need to be done in the more optimised find/ls variants,
I can see that extra runtime linking will occur, but I don’t
see how the other variants avoid opening each file.
Is 60ms for this activity (400MHZ P5) comparable with other OS/RTOS?
Thanks for the reply
William
William Morris <william@bangel.demon.co.uk> wrote:
There are 663 files in various sub-directories on a normal
QNX4 filesystem under /home/wrm/somedir. If I run either
of the commands
find /home/wrm/somedir -type f
or
ls -Rl /home/wrm/somedir
I get a long list of files reported within a second
And if I do
find /home/wrm/somedir -type f | xargs ls -l
It takes about 2 seconds.
What you have done here, is traversed the filesystem once, and
loaded 3 programs – meaning 3 process creations.
However, if I type
find /home/wrm/somedir -type f -exec /bin/ls -l {} ;
It takes 40 seconds to list all the files.
What you have done here is traversed the filesystem once, and done
664 process creations.
Process creation is not a cheap operation – and the issue
is not whether or not ls is cached, it still requires the
allocation and initialization of memory, finding and linking
the shared object(s) used by ls, duping of stdin/stdout/stderr
and any other inheritted fds, etc. Then, you also
have the 663 process terminations – notification of
parents, freeing of resources, flushing and closing of all
their fds, etc as well.
This process creation/termination overhead and the associated
expense is why a program like xargs was created in the first
place.
-David
QNX Training Services
http://www.qnx.com/support/training/
Please followup in this newsgroup if you have further questions.
David Gibbs wrote:
What you have done here is traversed the filesystem once, and done
664 process creations.
Process creation is not a cheap operation – and the issue
is not whether or not ls is cached, it still requires the
allocation and initialization of memory, finding and linking
the shared object(s) used by ls, duping of stdin/stdout/stderr
and any other inheritted fds, etc. Then, you also
have the 663 process terminations – notification of
parents, freeing of resources, flushing and closing of all
their fds, etc as well.
This process creation/termination overhead and the associated
expense is why a program like xargs was created in the first
place.
Seeing it all listed, 60ms no longer seems so excessive.
Many thanks
William
William Morris <william@bangel.demon.co.uk> wrote:
John Garvey wrote:
William Morris <> william@bangel.demon.co.uk> > wrote:
find /home/wrm/somedir -type f -exec /bin/ls -l {} ;
I can see that extra runtime linking will occur, but I don’t
see how the other variants avoid opening each file.
find/ls set the _IO_XFLAG_DIR_EXTRA_HINT bit for their readdir(),
which makes the filesystem place the stat details in with the
filenames (saving find/ls from doing an explicit stat() - which
is an open()/fstat()/close() cycle - themselves).