IDE Profiling

Robert Muil <r.muil@crcmining.com.au> wrote:

I copied the source, profiled it, and got completely different results,
though. (And, I got the results I expected, too.)

I don’t understand that. Are you using 6.2.1 IDE? Why would I get completely
different results?

I’m using a 6.3.0 IDE. I don’t have a 6.2.1 installation around.

I do not think you can get this information. I do not think the data
needed to answer this question is generated. (That is, I don’t think
gprof could tell you this either.)

I do not understand how the profiler can be used, therefore, to give useful
information. This seems to me to be so fundamental as to erode the purpose
of the profiler. It is akin to measuring the pressure on soldiers’ feet to
determine the movement an army.

For an application that is 99% library code, 1% user code it doesn’t help
much. For an application that is 10% library code, and 90% user code,
the tool can help a lot more.

-David

David Gibbs
QNX Training Services
dagibbs@qnx.com

David Gibbs wrote:

I have added -gdwarf-2 to CCFLAGS and LDFLAGS in Makefile.

CFLAGS or CCFLAGS?

(I thought CFLAGS was for C code, CCFLAGS for C++ code, but I could
be wrong.)

David is correct, CFLAGS for C code, CCFLAGS for C++ code.


Chris Herborth (cherborth@qnx.com)
Never send a monster to do the work of an evil scientist.

Robert Muil wrote:

I appreciate that people like yourself do, and QNX’s development is
admirable. But please, less marketing hype. Aim the documentation at
technicians and engineers, not other marketing departments.

The IDE User’s Guide has had quite a number of changes for SP1, but
we’re always interested in specific feedback about the docs…


Chris Herborth (cherborth@qnx.com)
Never send a monster to do the work of an evil scientist.

David Gibbs wrote:
[… need to run as root to get times in profiling data …]

Yeah, it attaches an interrupt handler to the timer to do the sampling.
No interrupt handler, no samples.

The IDE docs don’t mention this, so I’ll file a PR.


Chris Herborth (cherborth@qnx.com)
Never send a monster to do the work of an evil scientist.

Chris Herborth <cherborth@qnx.com> wrote:

David Gibbs wrote:
[… need to run as root to get times in profiling data …]
Yeah, it attaches an interrupt handler to the timer to do the sampling.
No interrupt handler, no samples.

The IDE docs don’t mention this, so I’ll file a PR.

Not sure the IDE docs need to mention it – the IDE launches everything
through qconn, which runs as root.

This only matters if you are trying to do profiling by running from the
commandline.

-David

David Gibbs
QNX Training Services
dagibbs@qnx.com

David Gibbs wrote:

This only matters if you are trying to do profiling by running from the
commandline.

And since we support loading profiler output generated by running from
the command-line, people should know how to do it… :wink:


Chris Herborth (cherborth@qnx.com)
Never send a monster to do the work of an evil scientist.

Wasn’t intended to be a compilable example – I’ll include at the end
the actual sample code I used, and the gprof output.

:slight_smile:

I didn’t try to compile what you posted. I added relevant code. Run as root,
I get similar results to you. I see that your point is valid - and very
surprising. gprof is blatantly fabricating information.

Word up to Microsoft for their profiler.

I think it is added to the core with Eclipse 3.0. It will definitely
be post 6.3.0.

So support and development for 6.2.1 is over?

I’m using a 6.3.0 IDE. I don’t have a 6.2.1 installation around.

So would I be, if it wasn’t completely incompatible with 6.2.1 targets.

I do not understand how the profiler can be used, therefore, to give
useful
information. This seems to me to be so fundamental as to erode the
purpose
of the profiler. It is akin to measuring the pressure on soldiers’ feet
to
determine the movement an army.

For an application that is 99% library code, 1% user code it doesn’t help
much. For an application that is 10% library code, and 90% user code,
the tool can help a lot more.

Not if the bottleneck is the 1% of user code that calls 90% of the library
code.

Agreed, and I have played with samples that just spin etc, but I was trying
to work out how to look at cumulative times.

Robert.

“Colin Burgess” <cburgess@qnx.com> wrote in message
news:cnvg2p$23e$1@inn.qnx.com

I don’t want to get into the functionality of the profiler too much, but
the test program in your source is going to spend 99.9% of it’s time in
libc functions usleep (which will end up being a TimerTimeout ) and printf
(all sorts of libc functions).

Since libc isn’t profiled, this will cause varying and unclear results.

Robert Muil wrote:
David,

I hope this screenshot is viewable. It should show how complicated a
process it is to profile even a stupidly simple program.

To determine how much time is spent in do_loop (), and where that is
spent, I first must find out what resolve_rels(), lookup(), hash(),
ConnectAttach(), static_strcmp(), __SysCpupageGet(), _dladdr() functions
are. Most are not even mentioned in the documentation (although this is
hard to be certain of because there is no index).

I must then work out where they are being called from.

I must then work out how much time they are using, presumably by guessing
percentage CPU from the %Time Usage bar or and calculating time in
functions as an percentage of the overall program run time (which would
need to be determined, I suppose, with a separate tool).

I hope that I am wrong about this. If so, please tell me how I can use
the information shown in the Profiler perspective to tell how much time
has been used by the do_loop() function, and where that time was spent).

Also note the coloured bars in the c editor. What is this supposed to
suggest, beyond that QNX likes the sound of its own name and has coloured
in the margin to highlight it?

Robert.


“David Gibbs” <> dagibbs@qnx.com> > wrote in message
news:cnl3nr$5n4$> 1@inn.qnx.com> …

Robert Muil <> r.muil@crcmining.com.au> > wrote:

David,

I am unable to determine how long a program spends, cumulatively, in a
function.

Total time directly in a function would be given by the total time in
Sampling information view.

Or, do you mean for a function, and all sub-functions, cumulated to that
function?


It does not help me to know that 5% of my time was in ldiv() or .plt()
(whatever they are). I want to know that 100% was in main(), then 90% of
that was in do_loop(), etc.

Sounds like that’s what you want. That can’t be done. Well, in theory
it can be done, but the data collection to provide that would be
enormous.
What the profiler does is collect 2 types of information – it annotates
any code compiled with profiling to get function call counts (basically
call pairs), and it samples the execution of the program from the timer
interrupt, storing the current IP, and active thread at that point.
Then,
the time useage is “estimated” based on the sampling – but it doesn’t
know what the call path to get to the function is. To get the cumulative
sub-function useage your asking for, at every sample point a full stack
backtrace would have to be collected and stored – the overhead to
collect
that information, and store that information, would be quite impressive,
and heavily impact whatever you were trying to profile. Also, the tool
chain (GCC) doesn’t supply tools to do that.

Why would full backtrace be neede?
Consider the following:

int func1() { /* use lots of CPU */}
int funca() { func1(); }
int funcb() { func1(); }
int main() { while (1) { funca(); funcb(); } }

Now, is CPU time spent in func1() attributed to funca() or funcb()?
Without
a stack backtrace, you can’t know.


Also, it does not seem to correctly read the symbol table. I have a very
simple program to test, which just loops and does a few printfs. When I
profile it, the calling information only displays call information for
the
source I compiled. For example, it does not tell me that do_loop()
called
printf().

Nope, it doesn’t. printf() is in our library – so it’s not
instrumented.
If I understand it properly, the call information is put in the prefix
of the called function – so any of your functions that get called should
have call count information – but if you call functions from our
library,
from source files not compiled -p, from your libraries not compiled -p,
etc, you won’t get call count information for those.


The sampling information is all over the place. If I reduce the
iterations
in do_loop(), the sampling information doesn’t even mention printf(). It
never mentions usleep(). With a higher number of iterations (like
99999), I
get a couple of little coloured bars in the text editor, but they don’t
correspond to where the program would have run. I only see 1 green and 1
blue bar - no breakdown at all. With 999 iterations, I don’t get any
coloured bars.

I ran it, I got coloured bars with 999 iterations. But, I ran it on
a pretty slow CPU target. (A VMWare session, in fact.) I got a little
bit of CPU attributed to the loop in do_loop(), but looking at the
Sampling information, most of the CPU was attributed to some unknown
function – I’m going to have to check with a developper as to what
is going on for that. I’m pretty sure that would be the printf().
(And, you won’t see the time in a function attributed to the function
call line in the editor.)

Your usleep() is 1000 usec, which is 1 ms – with a sampling rate of
1ms, I wouldn’t expect much attributed to usleep(). It might be another
of the “unknown” functions I’m seeing.


I have attached a screenshot of the editor after a profile. This is
about as
much useful information as I can get out of it.

I tried to view the screenshot, but got a garbled image. > :frowning:


There is no way to look at the profiling data statistically. No logs, or
textual data, such as the actual numbers that the %Time Usage bars are
drawn
from. For CPU usage, all I can get are little coloured bars, and then
only
if I am perseverent and lucky.

Are you just looking at the editor for the sampling data, or are you
actually using the QNX Application Profiler perspective, which includes
the Sampling Information and Call Information views which both provide
statistics summaries?

When you launched with profiling, did you click the “Switch to this
Tool’s Perspective on Launch” choice to open it automatically?


This is not a tool worthy of the advertising hype that populates the
help
documentation. If the documents spent as much time telling me how to
use
their brilliant, time-saving, modular, single, consistent, integrated
environment as they spent placing adjectives in front of their products,
I
may have less to complain about. Unfortunately it seems the deficiencies
are
in the product also - not only the documentation. This is especially
true of
the profiling and update/installation perspectives.

The update/installation perspective is core Eclipse, and should be
documented in the Eclipse docs, through the Help->Help Contents
menu, in particular I think this is talked about in the Workbench
Users Guide → Tasks → Updating Features with the update manager.

The IDE docs seem to be task-oriented, rather than reference oriented.


Just try to save a source file to My Documents quickly - or open an
external
c file in an existing editor.

That is an Eclipse paradigm, it works within a subset of the directory
structure, the workspace, and expects everything to be there.
(I think this is not uncommon with IDEs – they import/export stuff,
but only really deal with stuff that is in their more limitted
view/working
area.)

I found it frustrating, too, though.

Hope some of this helps.

-David

David Gibbs
QNX Training Services
dagibbs@qnx.com



\

cburgess@qnx.com

I have added -gdwarf-2 to CCFLAGS and LDFLAGS in Makefile.

CFLAGS or CCFLAGS?

(I thought CFLAGS was for C code, CCFLAGS for C++ code, but I could
be wrong.)

CFLAGS is completely ignored.

Is there supposed to be a GUI option to modify these settings, or have I
done it correctly?

Is it a QNX C project, or a Standard Make project?

A QNX C project.

If a QNX C project, you probably need to do it through the GUI, you’re
not supposed to edit the Makefile. (In fact, I don’t know what editing
the Makefile does.)

There is no way to modify the command line in the 6.2.1 IDE.

Does the binary parser option in project settings have anything to do
with
this?
For a QNX project, it should be set properly automatically.

It is not set by default: the 2 radio buttons are both blank.

For 6.2.1, I don’t remember for sure, but for a Standard Make project,
it should probably be set to QNX Binary parser. I don’t think it will
make a difference, but I’m not sure.

There is no ‘QNX binary parser’. It is either ‘Elf Parser’ or ‘PE Windows
Parser’. The distinction eludes me.

Robert Muil wrote:

Is it a QNX C project, or a Standard Make project?

A QNX C project.

Ah… don’t attempt to edit CFLAGS/CCFLAGS/LDFLAGS/etc. in the Makefile
or common.mk files. You want to edit the project properties
(right-click the project, choose Properties). Click QNX C/C++ Project
in the left-hand list, then on the Options pane check the “Build with
Profiling” option. Apply, then rebuild your application.

If you need or want to, you can also specify other compiler options,
defines, etc. on the Compiler and Linker panes.

If a QNX C project, you probably need to do it through the GUI, you’re
not supposed to edit the Makefile. (In fact, I don’t know what editing
the Makefile does.)

There is no way to modify the command line in the 6.2.1 IDE.

Instructions above are for the 6.3.0 IDE, sorry.

In your common.mk you could add:

CCOPTS+=-p
LDOPTS+=-p

Yeah, I know I said not to edit common.mk above, but if you don’t have
another way to change this, common.mk is actually safe (you specify your
‘use’ message file in there, too).

For 6.2.1, I don’t remember for sure, but for a Standard Make project,
it should probably be set to QNX Binary parser. I don’t think it will
make a difference, but I’m not sure.

There is no ‘QNX binary parser’. It is either ‘Elf Parser’ or ‘PE Windows
Parser’. The distinction eludes me.

Use the ELF parser. Binaries in QNX are in ELF format; the PE parser is
there to support Windows development from Eclipse (ie, it’s part of the
generic CDT, not an add-on from Momentics)… PE is the binary format
for Windows.


Chris Herborth (cherborth@qnx.com)
Never send a monster to do the work of an evil scientist.

Robert Muil <r.muil@crcmining.com.au> wrote:

Does the binary parser option in project settings have anything to do
with
this?
For a QNX project, it should be set properly automatically.

It is not set by default: the 2 radio buttons are both blank.

Sorry, what I meant is that for a QNX C project, the tools know
to use the QNX analysis tool chain.

For a standard make project, it isn’t a QNX project, so you have
to tell the IDE how to tear apart executables, etc – so setting
the binary parser is needed there. In some cases, for 6.2.1
a tool may not have worked, or worked properly, for a Standard
Make project, because the IDE does not how to figure out the
toolset, I think with 6.3.0 this was “fixed” by giving a
QNX Binary parser choice for Standard Make project. (That is,
I think that choice was added in the 6.2.1B → 6.3.0 changes.)

For 6.2.1, I don’t remember for sure, but for a Standard Make project,
it should probably be set to QNX Binary parser. I don’t think it will
make a difference, but I’m not sure.

There is no ‘QNX binary parser’. It is either ‘Elf Parser’ or ‘PE Windows
Parser’. The distinction eludes me.

For a QNX C project, you shouldn’t need to touch this setting.

-David

David Gibbs
QNX Training Services
dagibbs@qnx.com

Robert Muil <r.muil@crcmining.com.au> wrote:

Wasn’t intended to be a compilable example – I’ll include at the end
the actual sample code I used, and the gprof output.

:slight_smile:

I didn’t try to compile what you posted. I added relevant code. Run as root,
I get similar results to you. I see that your point is valid - and very
surprising. gprof is blatantly fabricating information.

Yup.

Word up to Microsoft for their profiler.

Ah, their profiler does handle the CPU useage attribution better?

I think it is added to the core with Eclipse 3.0. It will definitely
be post 6.3.0.

So support and development for 6.2.1 is over?

Still supported, but new feature development will be in current
release head.

-David

David Gibbs
QNX Training Services
dagibbs@qnx.com

Ah… don’t attempt to edit CFLAGS/CCFLAGS/LDFLAGS/etc. in the Makefile or
common.mk files. You want to edit the project properties (right-click the
project, choose Properties). Click QNX C/C++ Project in the left-hand
list, then on the Options pane check the “Build with Profiling” option.
Apply, then rebuild your application.

If you need or want to, you can also specify other compiler options,
defines, etc. on the Compiler and Linker panes.

If a QNX C project, you probably need to do it through the GUI, you’re
not supposed to edit the Makefile. (In fact, I don’t know what editing
the Makefile does.)

There is no way to modify the command line in the 6.2.1 IDE.

Instructions above are for the 6.3.0 IDE, sorry.

I really wish I could use the new IDE. Is it possible to make it backward
compatible with 6.2.1 targets? Like specifying different qconn etc?

In your common.mk you could add:

CCOPTS+=-p
LDOPTS+=-p

I am not trying to add profiling - the IDE options do that fine. I am trying
to change the debug format to dwarf2 with the -gdwarf-2 compile flag.

I have added CCOPTS+=-gdwarf-2 to every conceivable place in common.mk and
it does not affect the compile options. CFLAGS is also ignored.

If I put CCFLAGS+=-gdwarf-2 and LDFLAGS+=-gdwarf-2 in the common.mk, it will
add the options as expected to the command line i see in the C-Build view.

However, this does NOT solve the off-by-one line problem.

Use the ELF parser. Binaries in QNX are in ELF format; the PE parser is
there to support Windows development from Eclipse (ie, it’s part of the
generic CDT, not an add-on from Momentics)… PE is the binary format for
Windows.

Aha. Thanks - that makes sense. Perhaps a mention in the docs would be worth
someone’s time?

Hi Cris…

I wonder, when you say ‘SP1’, are you referring to a QNX release? If
so, is this available any where? When I go to the ‘myQNX’ place, I see
no updates any where. Thanks.

Regards…

Miguel.

Chris Herborth wrote:

Robert Muil wrote:

I appreciate that people like yourself do, and QNX’s development is
admirable. But please, less marketing hype. Aim the documentation at
technicians and engineers, not other marketing departments.


The IDE User’s Guide has had quite a number of changes for SP1, but
we’re always interested in specific feedback about the docs…

Miguel Simon wrote:

Hi Cris…

I wonder, when you say ‘SP1’, are you referring to a QNX release? If
so, is this available any where? When I go to the ‘myQNX’ place, I see
no updates any where. Thanks.

“Patch A” would be the correct term, unless QSS have changed their
naming scheme. And, no it’s not released yet.

Miguel Simon <simon@ou.edu> wrote:

Hi Cris…

I wonder, when you say ‘SP1’, are you referring to a QNX release? If
so, is this available any where? When I go to the ‘myQNX’ place, I see
no updates any where. Thanks.

Service Pack 1 (SP1) for 6.3.0 is in pre-release testing right now.

-David

David Gibbs
QNX Training Services
dagibbs@qnx.com

Robert Muil wrote:

I really wish I could use the new IDE. Is it possible to make it backward
compatible with 6.2.1 targets? Like specifying different qconn etc?

If you install 6.3.0 on a system with 6.2.1 already installed, you have
the option of using 6.2.1 or 6.3.0 tools. As for things that aren’t
under the aegis of the IDE (like qconn), if you didn’t upgrade your
target machine, it’d still be running the 6.2.1 versions.

In your common.mk you could add:

CCOPTS+=-p
LDOPTS+=-p

I am not trying to add profiling - the IDE options do that fine. I am trying
to change the debug format to dwarf2 with the -gdwarf-2 compile flag.

Ah, sorry!

I have added CCOPTS+=-gdwarf-2 to every conceivable place in common.mk and
it does not affect the compile options. CFLAGS is also ignored.

If I put CCFLAGS+=-gdwarf-2 and LDFLAGS+=-gdwarf-2 in the common.mk, it will
add the options as expected to the command line i see in the C-Build view.

I think the QNX Makefile hierarchy is going to use qcc, so you probably
want to add:

CCOPTS+=-Wc,-gdwarf-2
LDOPTS+=-Wl,-gdwarf-2

(although I’m not entirely sure it’s required for LDOPTS… better safe
than sorry).

However, this does NOT solve the off-by-one line problem.

No, but I think 6.3.0SP1 improves this.

Use the ELF parser. Binaries in QNX are in ELF format; the PE parser is
there to support Windows development from Eclipse (ie, it’s part of the
generic CDT, not an add-on from Momentics)… PE is the binary format for
Windows.

Aha. Thanks - that makes sense. Perhaps a mention in the docs would be worth
someone’s time?

Good idea, I’ll add it to The List™ for the IDE User’s Guide.


Chris Herborth (cherborth@qnx.com)
Never send a monster to do the work of an evil scientist.

Miguel Simon wrote:

I wonder, when you say ‘SP1’, are you referring to a QNX release? If
so, is this available any where? When I go to the ‘myQNX’ place, I see
no updates any where. Thanks.

6.3.0 Service Pack 1 is currently in the “coming soon!” state.


Chris Herborth (cherborth@qnx.com)
Never send a monster to do the work of an evil scientist.

Hi Chris…

Thanks. That is what I thought. I think that this release or patch has
been coming for a long time now. What is the name of the release, ‘SP1’
or ‘patch A’? Just as others have mentioned, is QSSL changing to the
‘SP’ versus ‘patch’ naming convention? Thanks.

Also, do you know (or can you tell) what are some of the things that are
included in this next release? For example, the Java VM, would it be a
new version close to at least Java 1.4.2_0x? How about the IDE for the
PE edition, would it be closer to Eclipse 3.0? Thanks.


Regards…

Miguel.



Chris Herborth wrote:

Miguel Simon wrote:

I wonder, when you say ‘SP1’, are you referring to a QNX release? If
so, is this available any where? When I go to the ‘myQNX’ place, I
see no updates any where. Thanks.


6.3.0 Service Pack 1 is currently in the “coming soon!” state.