Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parameterize benchmarks, add throughput measurements #418

Merged
merged 2 commits into from
Jul 13, 2022

Conversation

dralley
Copy link
Collaborator

@dralley dralley commented Jul 12, 2022

Added a throughput measurement to the macrobenchmark runs

fully_parse_document/sample_rss.xml                                                                            
                        time:   [338.77 us 343.69 us 349.75 us]
                        thrpt:  [539.25 MiB/s 548.75 MiB/s 556.72 MiB/s]       <-------------
                 change:
                        time:   [-28.438% -22.166% -15.255%] (p = 0.00 < 0.05)
                        thrpt:  [+18.001% +28.479% +39.740%]
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high severe

@dralley dralley marked this pull request as ready for review July 12, 2022 21:29
@dralley dralley changed the title Parameterize macrobenchmarks, add throughput estimations Parameterize macrobenchmarks, add throughput measurements Jul 12, 2022
@codecov-commenter
Copy link

codecov-commenter commented Jul 12, 2022

Codecov Report

Merging #418 (9cadfbe) into master (d87eff8) will increase coverage by 0.17%.
The diff coverage is 0.00%.

❗ Current head 9cadfbe differs from pull request most recent head de51d9b. Consider uploading reports for the commit de51d9b to get more accurate results

@@            Coverage Diff             @@
##           master     #418      +/-   ##
==========================================
+ Coverage   49.39%   49.57%   +0.17%     
==========================================
  Files          22       22              
  Lines       13987    13935      -52     
==========================================
- Hits         6909     6908       -1     
+ Misses       7078     7027      -51     
Flag Coverage Δ
unittests 49.57% <0.00%> (+0.17%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
benches/macrobenches.rs 0.00% <0.00%> (ø)
src/lib.rs 12.26% <0.00%> (-0.07%) ⬇️
src/de/mod.rs 77.38% <0.00%> (+0.09%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d87eff8...de51d9b. Read the comment docs.

@dralley dralley force-pushed the benches branch 4 times, most recently from bf950ea to 3a974d3 Compare July 13, 2022 01:02
@dralley dralley changed the title Parameterize macrobenchmarks, add throughput measurements Parameterize benchmarks, add throughput measurements Jul 13, 2022
@dralley
Copy link
Collaborator Author

dralley commented Jul 13, 2022

This does make the low-level comparative benchmarks take quite a bit longer, because obviously there's 12x as many test inputs now.

That might not be worth it, we could just select a few that give a nice representative sample. Or, the user can just filter out the ones they don't want with the standard command-line filtering.

];

for (id, data) in inputs.iter() {
group.throughput(Throughput::Bytes(data.len() as u64));
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Criterion makes this so easy 👌

@dralley
Copy link
Collaborator Author

dralley commented Jul 13, 2022

xml-rs is so slow in comparison that unfortunately it makes the comparative charts almost useless. quick-xml is a tiny dot that doesn't always render :)

@Mingun Mingun merged commit a6588c2 into tafia:master Jul 13, 2022
@dralley dralley deleted the benches branch July 13, 2022 04:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants