[PyTorch debug] Improve precision debug tools performance #1909
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
pggPL
force-pushed
the
nvinspect_performance
branch
from
dda223b to
7f56375
Compare
pggPL
force-pushed
the
nvinspect_performance
branch
from
e2f237d to
b5024af
Compare
for more information, see https://pre-commit.ci
pggPL
force-pushed
the
nvinspect_performance
branch
from
58cf805 to
9893831
Compare
for more information, see https://pre-commit.ci
pggPL
force-pushed
the
nvinspect_performance
branch
from
dfa89ee to
a0ae480
Compare
for more information, see https://pre-commit.ci
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
for more information, see https://pre-commit.ci
pggPL
force-pushed
the
nvinspect_performance
branch
from
9522170 to
1343547
Compare
for more information, see https://pre-commit.ci
pggPL
force-pushed
the
nvinspect_performance
branch
from
968336f to
7c1a1f7
Compare
|
PR ready for review, waiting for NVIDIA/nvidia-dlfw-inspect#7 to be merged to update version and run tests. |
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
pggPL
force-pushed
the
nvinspect_performance
branch
from
ba3d72e to
7322fc2
Compare
for more information, see https://pre-commit.ci
|
/te-ci pytorch L1 |
|
/te-ci pytorch |
1 similar comment
|
/te-ci pytorch |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR aims to speed up layers which are not affected by any feature in particular iteration. They should be exactly as fast as layers without initializing debug tools.
I needed to fix 3 things:
inspect_tensor_enabledand few similar calls for each layer, iteration and tensor. I changed calls likeinspect_tensor_enabled- they may return tuple (bool, int), where int indicated number of iteration the feature will be enabled next time. If each tensor for one layer returns (bool, n) we run non-debug layer for next n iterations,debug_api.step()is called after every iteration. Inside it, we callSTATS_BUFFER.log()which performs synchonization and some cpu ops, even if no stats is logged. I disable this logic if no stat was logged.COMM/GEMMoverlap was disabled for the whole time, now it is disabled when layer is affected by at least one feature.If we want to only log some stats every n iterations, then this PR should make it work as fast as non-debug workflow when n -> infinity.
Type of change
Checklist: