Make AUCPR work with multiple query groups #4436
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #4431.
I read the paper referenced by the AUCPR implementation, and as far as I can tell, the
EvalAucPR
seems to be consistent with the paper.The issue was that
EvalAucPR
was assuming the existence of only a single query group. More precisely, it throws an exception whensum_auc >= 1.0
, where it should have checkedsum_auc >= ngroup
instead. Since all tests for AUCPR was performing binary classification (where there is only one group), the bug went undetected.Fix. Have a proper check on the AUC value, in presence of multiple query groups. Also add a test with learning-to-rank example.