Proposal to add FDO profile quality related diagnostics

[Indu Bhagat]

Proposal to add diagnostics to know which functions were not run in the
training run in FDO.

Motivation :
When using FDO, it is often desirable to know the quality of function profile.
This means knowing which functions have STALE, NO/MISSING or ZERO profiles. GCC
has diagnostics for STALE (Wcoverage-mismatch) and NO/MISSING profiles
(Wmissing-profile).

Stats regarding number of bbs, branches, number of edges profiled etc. are
available in the dump file.

It will be useful to add a new diagnostic, -Wprofile-quality for the case when
there are ZERO profiles, i.e, those functions which were not exercised in the
training run.
(I submitted a patch https://gcc.gnu.org/ml/gcc-patches/2018-11/msg00508.html
after which there will be some reliable information in the -fdump-ipa-profile
dump file for the user to grep, but adding a adding a diagnostic gives user a
direct way.)

For example, after the above-mentioned patch precise-ipa-dump-optinfo.patch.ver1 :
After the posted patch precise-ipa-dump-optinfo.patch.ver1, the way to know
which functions were not run in the training run in O1, O2, O3 etc is to grep
for text in the -fdump-ipa-profile dump file:

"(estimated locally, globally 0)"

Proposal : Provide a more direct way than above. There are two options :

OPTION 1 : WARNING ONLY APPROACH
--------------------------------
1. Add a new warning -Wprofile-quality=[none|file|all]. Warning is disabled by
default with -fprofile-use.

-Wprofile-quality
-Wprofile-quality=file
emits a warning message for each compilation unit summarising the number of
functions which were not profiled in the training run.
"warning: XX out of YY functions not exercised in training run [-Wprofile-quality=]"

-Wprofile-quality=all
emits a warning message for each function which was not run in the training
run, in addition to the above
"warning: function XXXX not exercised in training run [-Wprofile-quality=]"

Sample Output :
"warning: XX out of YY functions not exercised in training run [-Wprofile-quality=]"
"warning: function AAA not exercised in training run [-Wprofile-quality=]"
"warning: function BBB not exercised in training run [-Wprofile-quality=]"

OPTION 2 : WARNING + OPT-INFO APPROACH
--------------------------------------
1. Add a new warning -Wprofile-quality. Warning is disabled by default.

-Wprofile-quality
emits a warning message for each compilation unit summarizing the number of
functions which were profiled in the training run.
"warning: XX out of YY functions not exercised in training run [-Wprofile-quality=]"

2. Next, using the dump_printf_loc API, add messages of MSG_NOTE dump_flag for
each function that was not run in the training run. This information is then
available to the user via flag -fopt-info-ipa.

Which option is preferable ?
OPTION 1 has the advantage that it is direct and information will not get diluted as
opt-info framework evolves.

Thanks

[Andi Kleen]

Post by Indu Bhagat
Proposal to add diagnostics to know which functions were not run in the
training run in FDO.

Don't you think the warning will be very noisy? I assume most programs
have a lot of cold error handling functions etc. that are never executed
in a normal execution.

Like how does it look like for a gcc build for example?

I suspect it would need some heuristics to cut it down at least, like if
the function ends with exit (perhaps propagated through the call tree)
it's not flagged. Or if there is warning for a function don't warn
about callees that are not called from somewhere else.

-Andi

[Martin Liška]

Post by Andi Kleen

Post by Indu Bhagat
Proposal to add diagnostics to know which functions were not run in the
training run in FDO.

Don't you think the warning will be very noisy? I assume most programs
have a lot of cold error handling functions etc. that are never executed
in a normal execution.
Like how does it look like for a gcc build for example?

I guess like this:
https://users.suse.com/~mliska/lcov/

Indu, have you thought about using gcov/lcov rather than inventing a new warning
(or opt info)?

Can you please define exactly your use case?

Martin

Post by Andi Kleen
I suspect it would need some heuristics to cut it down at least, like if
the function ends with exit (perhaps propagated through the call tree)
it's not flagged. Or if there is warning for a function don't warn
about callees that are not called from somewhere else.
-Andi

[Indu Bhagat]

Usecase : The usecase is to get a high-level assessment of the quality of
function profiles available (-Wcoverage-mismatch and -Wmissing-profile are
useful in the same spirit). For large codebases, it is useful to make the
profile quality information more handy for ease of development. Doing a full
gcov run is more cumbersome than keeping the warning flag ENABLED in the
codebase at compile/development time to catch these.

For these reasons, I do think that atleast adding the summary-level diagnostic
is useful in general -
"warning: XX out of YY functions not exercised in training run [-Wprofile-quality]"
This lets the user make the decision of whether and how to drill down.

Now for drill down, I agree gcov/lcov is a good choice. But a full gcov run is
sometimes a deterrent for large codebases.
For this reason, my proposal has a second-level option in the form of
opt-info msgs/-Wprofile-quality=all (depending on whether we choose OPTION 1 or
OPTION 2).

Regarding the function level detail being too noisy : I sort of agree with that
comment. But I am of the opinion that I would rather leave it to the user to
infer the profile quality as per the application characteristics.

Thanks
Indu

Post by Martin Liška

Post by Andi Kleen

Post by Indu Bhagat
Proposal to add diagnostics to know which functions were not run in the
training run in FDO.

Don't you think the warning will be very noisy? I assume most programs
have a lot of cold error handling functions etc. that are never executed
in a normal execution.
Like how does it look like for a gcc build for example?

https://users.suse.com/~mliska/lcov/
Indu, have you thought about using gcov/lcov rather than inventing a new warning
(or opt info)?
Can you please define exactly your use case?
Martin

Post by Andi Kleen
I suspect it would need some heuristics to cut it down at least, like if
the function ends with exit (perhaps propagated through the call tree)
it's not flagged. Or if there is warning for a function don't warn
about callees that are not called from somewhere else.
-Andi

[Andi Kleen]

Post by Indu Bhagat
Regarding the function level detail being too noisy : I sort of agree with that
comment. But I am of the opinion that I would rather leave it to the user to
infer the profile quality as per the application characteristics.

Makes sense I guess. But I would keep the drill down as opt-info.
opt-info is better designed for really high volume data like this.

-Andi