Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Changes at 4/8/2022 8:59:46 PM #67916

Closed
performanceautofiler bot opened this issue Apr 12, 2022 · 9 comments
Closed

[Perf] Changes at 4/8/2022 8:59:46 PM #67916

performanceautofiler bot opened this issue Apr 12, 2022 · 9 comments
Assignees
@steveharter
Labels
arch-x64 area-System.Reflection runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark

Comments

@performanceautofiler
Copy link

Run Information

Architecture x64
OS alpine 3.12
Baseline dee2bb50f12e5d6ef16eee7ed944b0fb7ef27053
Compare 285b8259af30e255980b1ebfb2ddd60ca291a83a
Diff Diff

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Min - Duration of single invocation 344.72 ns 394.62 ns 1.14 0.15 False
Select - Duration of single invocation 516.14 ns 543.85 ns 1.05 0.06 False
WhereLast_LastElementMatches - Duration of single invocation 189.00 ns 211.84 ns 1.12 0.16 False
ToArray - Duration of single invocation 538.21 ns 587.94 ns 1.09 0.15 False
Max - Duration of single invocation 326.12 ns 345.08 ns 1.06 0.18 False
WhereSingleOrDefault_LastElementMatches - Duration of single invocation 188.09 ns 208.33 ns 1.11 0.17 False
WhereSingle_LastElementMatches - Duration of single invocation 241.01 ns 275.88 ns 1.14 0.09 False
WhereSingle_LastElementMatches - Duration of single invocation 187.60 ns 209.41 ns 1.12 0.16 False
Concat_Once - Duration of single invocation 1.39 μs 1.50 μs 1.08 0.10 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Payloads

Baseline
Compare

Histogram

System.Linq.Tests.Perf_Enumerable.Min(input: IEnumerable)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 394.61806294713875 > 362.05982889539916.
IsChangePoint: Marked as a change because one of 2/2/2022 4:36:37 PM, 2/17/2022 3:48:53 PM, 3/31/2022 1:45:25 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -99.13838721792497 (T) = (0 -388.6858653503989) / Math.Sqrt((0.06605368871080244 / (41)) + (2.8981968010071477 / (15))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (15) - 2, .025) and -0.1268632234122872 = (344.92727890560485 - 388.6858653503989) / 344.92727890560485 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Select(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 543.8486089348033 > 541.8632477060415.
IsChangePoint: Marked as a change because one of 2/2/2022 4:36:37 PM, 3/21/2022 1:28:14 PM, 3/30/2022 6:37:35 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -91.33155035845984 (T) = (0 -544.4831203660419) / Math.Sqrt((1.3619256284845913 / (42)) + (0.9522971978068583 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.05479538769554284 = (516.1978585776695 - 544.4831203660419) / 516.1978585776695 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereLast_LastElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 211.8397032049015 > 198.47407088024477.
IsChangePoint: Marked as a change because one of 2/12/2022 1:03:18 PM, 3/4/2022 1:10:06 AM, 3/7/2022 6:10:39 PM, 3/11/2022 8:42:43 PM, 3/21/2022 1:28:14 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -92.3480253125493 (T) = (0 -212.04095446475998) / Math.Sqrt((0.30497366979072915 / (42)) + (0.805795091292106 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.12050925633637162 = (189.2362363502924 - 212.04095446475998) / 189.2362363502924 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.ToArray(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 587.9408925094091 > 564.7677348619952.
IsChangePoint: Marked as a change because one of 2/10/2022 10:02:40 PM, 2/17/2022 3:48:53 PM, 3/21/2022 1:28:14 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -51.82444215521497 (T) = (0 -588.2956558538465) / Math.Sqrt((9.63356249153452 / (42)) + (10.169338297592477 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.09159742758017234 = (538.930965748029 - 588.2956558538465) / 538.930965748029 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Max(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 345.07754686518666 > 342.0248002740456.
IsChangePoint: Marked as a change because one of 3/11/2022 8:42:43 PM, 3/31/2022 1:45:25 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -139.34082000925326 (T) = (0 -345.197599340174) / Math.Sqrt((0.5755955163133378 / (42)) + (0.10429400322464001 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.06158955182995014 = (325.1704943262942 - 345.197599340174) / 325.1704943262942 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereSingleOrDefault_LastElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 208.32933835851554 > 197.3437324453004.
IsChangePoint: Marked as a change because one of 2/19/2022 11:51:48 PM, 2/26/2022 10:27:55 AM, 3/21/2022 1:28:14 PM, 3/30/2022 6:37:35 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -101.23452382617847 (T) = (0 -209.62666586137888) / Math.Sqrt((0.7794940838382208 / (42)) + (0.45340920531385925 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.11940420136927643 = (187.26628469408956 - 209.62666586137888) / 187.26628469408956 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereSingle_LastElementMatches(input: List)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 275.8796801489393 > 252.88159557032284.
IsChangePoint: Marked as a change because one of 3/15/2022 12:53:39 AM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -53.3125925874796 (T) = (0 -283.40697332734703) / Math.Sqrt((1.2439723975903147 / (42)) + (9.212186628477783 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.17776261706336152 = (240.63165974311124 - 283.40697332734703) / 240.63165974311124 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereSingle_LastElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 209.4082840855427 > 197.10058986883456.
IsChangePoint: Marked as a change because one of 2/1/2022 1:24:52 PM, 3/21/2022 1:28:14 PM, 3/30/2022 6:37:35 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -117.08418979343334 (T) = (0 -209.53530180583914) / Math.Sqrt((0.1663522930094863 / (42)) + (0.4657993240282166 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.11676841756592894 = (187.62645729410514 - 209.53530180583914) / 187.62645729410514 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Concat_Once(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4960792177155142 > 1.454677141392647.
IsChangePoint: Marked as a change because one of 2/10/2022 10:02:40 PM, 3/21/2022 1:28:14 PM, 4/8/2022 2:40:20 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -18.7239373644994 (T) = (0 -1537.2144220094967) / Math.Sqrt((1224.5197593867324 / (41)) + (385.2264574045426 / (16))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (16) - 2, .025) and -0.09824972335700226 = (1399.6947955613575 - 1537.2144220094967) / 1399.6947955613575 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS alpine 3.12
Baseline dee2bb50f12e5d6ef16eee7ed944b0fb7ef27053
Compare 285b8259af30e255980b1ebfb2ddd60ca291a83a
Diff Diff

Regressions in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LastIndexOfAnyValues - Duration of single invocation 9.68 ns 14.16 ns 1.46 0.54 False
BinarySearch - Duration of single invocation 9.03 ns 10.39 ns 1.15 0.14 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Span&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Byte>.LastIndexOfAnyValues(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.159845620731966 > 11.656600717395374.
IsChangePoint: Marked as a change because one of 3/16/2022 3:05:50 AM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -19.357725869138704 (T) = (0 -14.196068057031201) / Math.Sqrt((1.1733959313355158 / (42)) + (0.016550021977281348 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.3027240891349547 = (10.897217741983868 - 14.196068057031201) / 10.897217741983868 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.BinarySearch(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.393370663094947 > 9.473804526883557.
IsChangePoint: Marked as a change because one of 1/31/2022 7:28:13 PM, 2/1/2022 2:18:14 AM, 3/17/2022 6:54:53 PM, 3/29/2022 3:04:55 PM, 4/7/2022 3:14:09 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -19.952848366603714 (T) = (0 -10.240215896289447) / Math.Sqrt((0.00018233033663987982 / (35)) + (0.08278754013758374 / (22))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (22) - 2, .025) and -0.13586019021992402 = (9.015384097849884 - 10.240215896289447) / 9.015384097849884 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.12
Baseline dee2bb50f12e5d6ef16eee7ed944b0fb7ef27053
Compare 285b8259af30e255980b1ebfb2ddd60ca291a83a
Diff Diff

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitArrayNot - Duration of single invocation 7.85 ns 10.85 ns 1.38 0.23 False
BitArrayOr - Duration of single invocation 8.83 ns 11.73 ns 1.33 0.61 False
BitArrayAnd - Duration of single invocation 8.74 ns 11.74 ns 1.34 0.59 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Tests.Perf_BitArray*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_BitArray.BitArrayNot(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.845571805248149 > 7.709763432514391.
IsChangePoint: Marked as a change because one of 2/2/2022 7:54:56 PM, 2/15/2022 2:06:03 PM, 2/23/2022 4:28:51 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -13.674706395032752 (T) = (0 -10.67244949209505) / Math.Sqrt((0.02248767927592186 / (42)) + (0.8856408150250037 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.45507466613645986 = (7.3346404418082045 - 10.67244949209505) / 7.3346404418082045 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayOr(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.728589380002509 > 9.498874987226355.
IsChangePoint: Marked as a change because one of 2/12/2022 6:41:57 AM, 2/22/2022 5:41:06 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -5.717326926706129 (T) = (0 -12.748922818187394) / Math.Sqrt((0.23028461872526312 / (42)) + (6.360176386981104 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.41622845710003387 = (9.00202418209627 - 12.748922818187394) / 9.00202418209627 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayAnd(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.741024619579484 > 9.77697683189218.
IsChangePoint: Marked as a change because one of 2/15/2022 2:06:03 PM, 2/21/2022 10:29:02 AM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -11.387721713527105 (T) = (0 -11.539000714157897) / Math.Sqrt((0.18397043435650473 / (42)) + (0.6424268018298204 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.27295633754901966 = (9.06472623905967 - 11.539000714157897) / 9.06472623905967 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.12
Baseline dee2bb50f12e5d6ef16eee7ed944b0fb7ef27053
Compare 285b8259af30e255980b1ebfb2ddd60ca291a83a
Diff Diff

Regressions in System.Memory.Span<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Fill - Duration of single invocation 21.22 ns 28.77 ns 1.36 0.65 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Span&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Int32>.Fill(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.774092619192036 > 23.95949360303189.
IsChangePoint: Marked as a change because one of 2/22/2022 2:43:53 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -11.485891512345141 (T) = (0 -28.970127171976586) / Math.Sqrt((7.816805707043072 / (42)) + (3.0514221177095733 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.3288196960209214 = (21.801398081866232 - 28.970127171976586) / 21.801398081866232 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added alpine 3.12 untriaged New issue has not been triaged by the area owner labels Apr 12, 2022
@EgorBo EgorBo transferred this issue from dotnet/perf-autofiling-issues Apr 12, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@EgorBo
Copy link
Member

EgorBo commented Apr 12, 2022

Most likely caused by #66357 cc @steveharter

@EgorBo EgorBo added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels Apr 12, 2022
@EgorBo
Copy link
Member

EgorBo commented Apr 12, 2022

ubuntu-x64: #66357

@ghost
Copy link

ghost commented Apr 13, 2022

Tagging subscribers to this area: @dotnet/area-system-reflection
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS alpine 3.12
Baseline dee2bb50f12e5d6ef16eee7ed944b0fb7ef27053
Compare 285b8259af30e255980b1ebfb2ddd60ca291a83a
Diff Diff

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Min - Duration of single invocation 344.72 ns 394.62 ns 1.14 0.15 False
Select - Duration of single invocation 516.14 ns 543.85 ns 1.05 0.06 False
WhereLast_LastElementMatches - Duration of single invocation 189.00 ns 211.84 ns 1.12 0.16 False
ToArray - Duration of single invocation 538.21 ns 587.94 ns 1.09 0.15 False
Max - Duration of single invocation 326.12 ns 345.08 ns 1.06 0.18 False
WhereSingleOrDefault_LastElementMatches - Duration of single invocation 188.09 ns 208.33 ns 1.11 0.17 False
WhereSingle_LastElementMatches - Duration of single invocation 241.01 ns 275.88 ns 1.14 0.09 False
WhereSingle_LastElementMatches - Duration of single invocation 187.60 ns 209.41 ns 1.12 0.16 False
Concat_Once - Duration of single invocation 1.39 μs 1.50 μs 1.08 0.10 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Payloads

Baseline
Compare

Histogram

System.Linq.Tests.Perf_Enumerable.Min(input: IEnumerable)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 394.61806294713875 > 362.05982889539916.
IsChangePoint: Marked as a change because one of 2/2/2022 4:36:37 PM, 2/17/2022 3:48:53 PM, 3/31/2022 1:45:25 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -99.13838721792497 (T) = (0 -388.6858653503989) / Math.Sqrt((0.06605368871080244 / (41)) + (2.8981968010071477 / (15))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (15) - 2, .025) and -0.1268632234122872 = (344.92727890560485 - 388.6858653503989) / 344.92727890560485 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Select(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 543.8486089348033 > 541.8632477060415.
IsChangePoint: Marked as a change because one of 2/2/2022 4:36:37 PM, 3/21/2022 1:28:14 PM, 3/30/2022 6:37:35 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -91.33155035845984 (T) = (0 -544.4831203660419) / Math.Sqrt((1.3619256284845913 / (42)) + (0.9522971978068583 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.05479538769554284 = (516.1978585776695 - 544.4831203660419) / 516.1978585776695 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereLast_LastElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 211.8397032049015 > 198.47407088024477.
IsChangePoint: Marked as a change because one of 2/12/2022 1:03:18 PM, 3/4/2022 1:10:06 AM, 3/7/2022 6:10:39 PM, 3/11/2022 8:42:43 PM, 3/21/2022 1:28:14 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -92.3480253125493 (T) = (0 -212.04095446475998) / Math.Sqrt((0.30497366979072915 / (42)) + (0.805795091292106 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.12050925633637162 = (189.2362363502924 - 212.04095446475998) / 189.2362363502924 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.ToArray(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 587.9408925094091 > 564.7677348619952.
IsChangePoint: Marked as a change because one of 2/10/2022 10:02:40 PM, 2/17/2022 3:48:53 PM, 3/21/2022 1:28:14 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -51.82444215521497 (T) = (0 -588.2956558538465) / Math.Sqrt((9.63356249153452 / (42)) + (10.169338297592477 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.09159742758017234 = (538.930965748029 - 588.2956558538465) / 538.930965748029 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Max(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 345.07754686518666 > 342.0248002740456.
IsChangePoint: Marked as a change because one of 3/11/2022 8:42:43 PM, 3/31/2022 1:45:25 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -139.34082000925326 (T) = (0 -345.197599340174) / Math.Sqrt((0.5755955163133378 / (42)) + (0.10429400322464001 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.06158955182995014 = (325.1704943262942 - 345.197599340174) / 325.1704943262942 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereSingleOrDefault_LastElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 208.32933835851554 > 197.3437324453004.
IsChangePoint: Marked as a change because one of 2/19/2022 11:51:48 PM, 2/26/2022 10:27:55 AM, 3/21/2022 1:28:14 PM, 3/30/2022 6:37:35 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -101.23452382617847 (T) = (0 -209.62666586137888) / Math.Sqrt((0.7794940838382208 / (42)) + (0.45340920531385925 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.11940420136927643 = (187.26628469408956 - 209.62666586137888) / 187.26628469408956 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereSingle_LastElementMatches(input: List)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 275.8796801489393 > 252.88159557032284.
IsChangePoint: Marked as a change because one of 3/15/2022 12:53:39 AM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -53.3125925874796 (T) = (0 -283.40697332734703) / Math.Sqrt((1.2439723975903147 / (42)) + (9.212186628477783 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.17776261706336152 = (240.63165974311124 - 283.40697332734703) / 240.63165974311124 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereSingle_LastElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 209.4082840855427 > 197.10058986883456.
IsChangePoint: Marked as a change because one of 2/1/2022 1:24:52 PM, 3/21/2022 1:28:14 PM, 3/30/2022 6:37:35 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -117.08418979343334 (T) = (0 -209.53530180583914) / Math.Sqrt((0.1663522930094863 / (42)) + (0.4657993240282166 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.11676841756592894 = (187.62645729410514 - 209.53530180583914) / 187.62645729410514 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Concat_Once(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4960792177155142 > 1.454677141392647.
IsChangePoint: Marked as a change because one of 2/10/2022 10:02:40 PM, 3/21/2022 1:28:14 PM, 4/8/2022 2:40:20 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -18.7239373644994 (T) = (0 -1537.2144220094967) / Math.Sqrt((1224.5197593867324 / (41)) + (385.2264574045426 / (16))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (16) - 2, .025) and -0.09824972335700226 = (1399.6947955613575 - 1537.2144220094967) / 1399.6947955613575 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS alpine 3.12
Baseline dee2bb50f12e5d6ef16eee7ed944b0fb7ef27053
Compare 285b8259af30e255980b1ebfb2ddd60ca291a83a
Diff Diff

Regressions in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LastIndexOfAnyValues - Duration of single invocation 9.68 ns 14.16 ns 1.46 0.54 False
BinarySearch - Duration of single invocation 9.03 ns 10.39 ns 1.15 0.14 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Span&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Byte>.LastIndexOfAnyValues(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.159845620731966 > 11.656600717395374.
IsChangePoint: Marked as a change because one of 3/16/2022 3:05:50 AM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -19.357725869138704 (T) = (0 -14.196068057031201) / Math.Sqrt((1.1733959313355158 / (42)) + (0.016550021977281348 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.3027240891349547 = (10.897217741983868 - 14.196068057031201) / 10.897217741983868 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.BinarySearch(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.393370663094947 > 9.473804526883557.
IsChangePoint: Marked as a change because one of 1/31/2022 7:28:13 PM, 2/1/2022 2:18:14 AM, 3/17/2022 6:54:53 PM, 3/29/2022 3:04:55 PM, 4/7/2022 3:14:09 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -19.952848366603714 (T) = (0 -10.240215896289447) / Math.Sqrt((0.00018233033663987982 / (35)) + (0.08278754013758374 / (22))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (22) - 2, .025) and -0.13586019021992402 = (9.015384097849884 - 10.240215896289447) / 9.015384097849884 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.12
Baseline dee2bb50f12e5d6ef16eee7ed944b0fb7ef27053
Compare 285b8259af30e255980b1ebfb2ddd60ca291a83a
Diff Diff

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitArrayNot - Duration of single invocation 7.85 ns 10.85 ns 1.38 0.23 False
BitArrayOr - Duration of single invocation 8.83 ns 11.73 ns 1.33 0.61 False
BitArrayAnd - Duration of single invocation 8.74 ns 11.74 ns 1.34 0.59 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Tests.Perf_BitArray*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_BitArray.BitArrayNot(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.845571805248149 > 7.709763432514391.
IsChangePoint: Marked as a change because one of 2/2/2022 7:54:56 PM, 2/15/2022 2:06:03 PM, 2/23/2022 4:28:51 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -13.674706395032752 (T) = (0 -10.67244949209505) / Math.Sqrt((0.02248767927592186 / (42)) + (0.8856408150250037 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.45507466613645986 = (7.3346404418082045 - 10.67244949209505) / 7.3346404418082045 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayOr(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.728589380002509 > 9.498874987226355.
IsChangePoint: Marked as a change because one of 2/12/2022 6:41:57 AM, 2/22/2022 5:41:06 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -5.717326926706129 (T) = (0 -12.748922818187394) / Math.Sqrt((0.23028461872526312 / (42)) + (6.360176386981104 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.41622845710003387 = (9.00202418209627 - 12.748922818187394) / 9.00202418209627 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayAnd(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.741024619579484 > 9.77697683189218.
IsChangePoint: Marked as a change because one of 2/15/2022 2:06:03 PM, 2/21/2022 10:29:02 AM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -11.387721713527105 (T) = (0 -11.539000714157897) / Math.Sqrt((0.18397043435650473 / (42)) + (0.6424268018298204 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.27295633754901966 = (9.06472623905967 - 11.539000714157897) / 9.06472623905967 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.12
Baseline dee2bb50f12e5d6ef16eee7ed944b0fb7ef27053
Compare 285b8259af30e255980b1ebfb2ddd60ca291a83a
Diff Diff

Regressions in System.Memory.Span<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Fill - Duration of single invocation 21.22 ns 28.77 ns 1.36 0.65 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Span&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Int32>.Fill(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.774092619192036 > 23.95949360303189.
IsChangePoint: Marked as a change because one of 2/22/2022 2:43:53 PM, 4/8/2022 7:02:49 PM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -11.485891512345141 (T) = (0 -28.970127171976586) / Math.Sqrt((7.816805707043072 / (42)) + (3.0514221177095733 / (15))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (15) - 2, .025) and -0.3288196960209214 = (21.801398081866232 - 28.970127171976586) / 21.801398081866232 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

area-System.Reflection, tenet-performance, tenet-performance-benchmarks, untriaged, refs/heads/main, RunKind=micro, Regression, CoreClr, x64, alpine 3.12
Milestone: -

@DrewScoggins
Copy link
Member

DrewScoggins commented Apr 14, 2022 •
edited
Loading

Windows-Arm64: dotnet/perf-autofiling-issues#4615

@DrewScoggins
Copy link
Member

Ubuntu-Arm64: dotnet/perf-autofiling-issues#4627

@DrewScoggins
Copy link
Member

We are also seeing some improvement on Win-Arm64: dotnet/perf-autofiling-issues#4621

@michellemcdaniel michellemcdaniel mentioned this issue Apr 18, 2022
Closed
@steveharter
Copy link
Member

This will change again once #67917 is in.

@steveharter steveharter removed the untriaged New issue has not been triaged by the area owner label May 25, 2022
@steveharter steveharter self-assigned this Jul 5, 2022
@steveharter
Copy link
Member

Closing; this code base was updated again and the new perf numbers are good: #69575 (comment)

@ghost ghost locked as resolved and limited conversation to collaborators Aug 12, 2022
@jeffhandley jeffhandley added runtime-coreclr specific to the CoreCLR runtime and removed CoreClr labels Dec 28, 2022
@jeffhandley jeffhandley added arch-x64 and removed x64 labels Dec 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@steveharter steveharter

Labels
arch-x64 area-System.Reflection runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@EgorBo @jeffhandley @jkotas @steveharter @DrewScoggins