Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/arm64: 29 Regressions on 12/2/2022 1:56:04 AM #79336

Closed
performanceautofiler bot opened this issue Dec 6, 2022 · 11 comments
Closed

[Perf] Linux/arm64: 29 Regressions on 12/2/2022 1:56:04 AM #79336

performanceautofiler bot opened this issue Dec 6, 2022 · 11 comments
Assignees
@BrzVlad
Labels
arch-arm64 area-Codegen-Interpreter-mono

Comments

@performanceautofiler
Copy link

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Tests.Perf_Version

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Ctor3 - Duration of single invocation 50.11 ns 86.08 ns 1.72 0.03 False
Ctor2 - Duration of single invocation 47.39 ns 71.32 ns 1.51 0.05 False
Ctor4 - Duration of single invocation 51.76 ns 102.99 ns 1.99 0.10 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Version*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Version.Ctor3

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -102.11659169159766 (T) = (0 -86.74571476872939) / Math.Sqrt((0.12230139019819382 / (7)) + (0.34097203649353724 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.7430184405411777 = (49.76752554711717 - 86.74571476872939) / 49.76752554711717 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.Ctor2

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -130.89007952591638 (T) = (0 -71.46053405222891) / Math.Sqrt((0.12904052784230644 / (7)) + (0.047092010962607275 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.5114678426116724 = (47.278898060280206 - 71.46053405222891) / 47.278898060280206 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.Ctor4

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -69.91395572248736 (T) = (0 -101.93066727444004) / Math.Sqrt((1.471178010925077 / (7)) + (0.8878575041028844 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.9529804518972763 = (52.19236432982046 - 101.93066727444004) / 52.19236432982046 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Tests.Perf_Int16

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 50.75 ns 363.06 ns 7.15 0.10 False
ToString - Duration of single invocation 24.27 ns 50.49 ns 2.08 0.13 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int16*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int16.ToString(value: 32767)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -56.388800046236824 (T) = (0 -369.86237428362614) / Math.Sqrt((0.4866799662928942 / (7)) + (96.02840362269899 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -6.326084703684954 = (50.485680857278204 - 369.86237428362614) / 50.485680857278204 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int16.ToString(value: 0)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -80.48997064927724 (T) = (0 -50.725437409996665) / Math.Sqrt((0.3549137971977588 / (7)) + (0.15089070297483032 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -1.0172795589575365 = (25.145467411670943 - 50.725437409996665) / 25.145467411670943 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Numerics.Tests.Perf_Matrix4x4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CreatePerspectiveFieldOfViewBenchmark - Duration of single invocation 103.09 ns 245.60 ns 2.38 0.19 False
CreatePerspectiveBenchmark - Duration of single invocation 89.33 ns 170.42 ns 1.91 0.17 False
CreatePerspectiveOffCenterBenchmark - Duration of single invocation 101.51 ns 178.94 ns 1.76 0.13 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Matrix4x4*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveFieldOfViewBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -47.407465925003 (T) = (0 -244.7283279584433) / Math.Sqrt((16.955750812608382 / (6)) + (19.593654446125736 / (3))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (3) - 2, .025) and -1.4543703149147356 = (99.71124832763678 - 244.7283279584433) / 99.71124832763678 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -28.27856125928147 (T) = (0 -171.1409131619612) / Math.Sqrt((7.888483020637348 / (6)) + (21.182069958258506 / (3))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (3) - 2, .025) and -0.9164335188808791 = (89.30177409018638 - 171.1409131619612) / 89.30177409018638 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveOffCenterBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -31.55669924340902 (T) = (0 -183.25801278540254) / Math.Sqrt((7.261748400179219 / (7)) + (17.08249335847475 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.807574409516796 = (101.38338528171097 - 183.25801278540254) / 101.38338528171097 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in LinqBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Where01ForX - Duration of single invocation 5.23 secs 5.63 secs 1.08 0.04 False
Where01LinqQueryX - Duration of single invocation 5.42 secs 5.88 secs 1.09 0.03 False
Where01LinqMethodX - Duration of single invocation 5.55 secs 6.02 secs 1.08 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'LinqBenchmarks*'

Payloads

Baseline
Compare

Histogram

LinqBenchmarks.Where01ForX

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -14.936972236005152 (T) = (0 -5631493356.288889) / Math.Sqrt((3694169213402831.5 / (7)) + (43721919026826.195 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.06583450163764645 = (5283647083.70403 - 5631493356.288889) / 5283647083.70403 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### LinqBenchmarks.Where01LinqQueryX

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -13.734392138221246 (T) = (0 -5893148331.555554) / Math.Sqrt((3774102618403431.5 / (7)) + (610818891582227.5 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.06782461999228462 = (5518835416.623129 - 5893148331.555554) / 5518835416.623129 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### LinqBenchmarks.Where01LinqMethodX

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -9.86345996511963 (T) = (0 -5980419743.822223) / Math.Sqrt((2693421689581077.5 / (7)) + (5661386090254048 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.08531989186277468 = (5510282994.590477 - 5980419743.822223) / 5510282994.590477 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Perf_Convert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ChangeType - Duration of single invocation 242.81 ns 550.82 ns 2.27 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Perf_Convert*'

Payloads

Baseline
Compare

Histogram

System.Perf_Convert.ChangeType

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -278.32962679684823 (T) = (0 -549.827499101074) / Math.Sqrt((3.4844707188457495 / (7)) + (2.212745495100914 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -1.286444151406167 = (240.4727440042343 - 549.827499101074) / 240.4727440042343 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Tests.Perf_Byte

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 32.69 ns 318.81 ns 9.75 0.23 False
ToString - Duration of single invocation 5.57 ns 38.47 ns 6.91 0.53 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Byte*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Byte.ToString(value: 255)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -195.61171260977292 (T) = (0 -320.29023052152485) / Math.Sqrt((1.5811890951979486 / (7)) + (5.9246995311702335 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -9.641055212020607 = (30.09948018686256 - 320.29023052152485) / 30.09948018686256 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Byte.ToString(value: 0)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -214.07001758349224 (T) = (0 -38.46763301637973) / Math.Sqrt((0.16976291339258173 / (7)) + (0.00026408657920728166 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -6.58722563160005 = (5.07005259684987 - 38.46763301637973) / 5.07005259684987 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Tests.Perf_SByte

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 48.55 ns 335.43 ns 6.91 0.13 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_SByte*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_SByte.ToString(value: 127)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -131.10599664117586 (T) = (0 -331.373923544455) / Math.Sqrt((2.5541456821294104 / (7)) + (12.777817799214688 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -5.701772953166281 = (49.445710241182674 - 331.373923544455) / 49.445710241182674 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Tests.Perf_Int32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 26.24 ns 50.38 ns 1.92 0.07 False
ToString - Duration of single invocation 58.89 ns 464.55 ns 7.89 0.10 False
ToString - Duration of single invocation 49.52 ns 356.36 ns 7.20 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int32.ToString(value: 4)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -397.7799585017025 (T) = (0 -50.363405275838055) / Math.Sqrt((0.005876323844439421 / (7)) + (0.0086335361594911 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.928832130475525 = (26.110828661601413 - 50.363405275838055) / 26.110828661601413 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.ToString(value: 2147483647)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -98.0488081341939 (T) = (0 -459.62939409938963) / Math.Sqrt((0.8294610613239252 / (7)) + (50.04965875014641 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -6.961932675112278 = (57.72836983865956 - 459.62939409938963) / 57.72836983865956 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.ToString(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -186.46041601920334 (T) = (0 -359.6577200179263) / Math.Sqrt((0.225274352613395 / (7)) + (8.157075364986618 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -6.13886458571935 = (50.38024124135775 - 359.6577200179263) / 50.38024124135775 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Tests.Perf_UInt32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 4.63 ns 38.32 ns 8.27 0.53 False
ToString - Duration of single invocation 28.91 ns 353.59 ns 12.23 0.21 False
ToString - Duration of single invocation 36.10 ns 445.83 ns 12.35 0.20 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt32.ToString(value: 0)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -121.83438545236052 (T) = (0 -38.26640494603868) / Math.Sqrt((0.5208478356031971 / (7)) + (0.01079345134623293 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -8.027440808630944 = (4.2388984605086515 - 38.26640494603868) / 4.2388984605086515 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt32.ToString(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -173.78822541441707 (T) = (0 -352.30509252906205) / Math.Sqrt((1.0149976066430988 / (7)) + (9.942195956545815 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -11.11336904509903 = (29.08398903867308 - 352.30509252906205) / 29.08398903867308 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt32.ToString(value: 4294967295)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -140.96911563937482 (T) = (0 -447.88611016078016) / Math.Sqrt((0.1495139313555134 / (7)) + (25.547906105451304 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -11.443840500940496 = (35.992594900820954 - 447.88611016078016) / 35.992594900820954 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Tests.Perf_UInt16

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 31.52 ns 347.78 ns 11.03 0.22 False
ToString - Duration of single invocation 5.90 ns 38.72 ns 6.56 0.67 False
ToString - Duration of single invocation 33.02 ns 346.31 ns 10.49 0.29 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt16*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt16.ToString(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -154.11913435276162 (T) = (0 -349.97969933045647) / Math.Sqrt((1.4621823256088133 / (6)) + (12.168822896554309 / (3))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (3) - 2, .025) and -10.515321667338233 = (30.3925247979073 - 349.97969933045647) / 30.3925247979073 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt16.ToString(value: 0)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -38.44782026273584 (T) = (0 -39.30773967936462) / Math.Sqrt((0.48601388380289795 / (7)) + (2.1568831003647806 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -6.603963758201222 = (5.169374937771032 - 39.30773967936462) / 5.169374937771032 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt16.ToString(value: 65535)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -143.8955244153231 (T) = (0 -348.9548328742756) / Math.Sqrt((0.8354013201156075 / (6)) + (14.192348263337031 / (3))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (3) - 2, .025) and -10.111598552102034 = (31.40455725051929 - 348.9548328742756) / 31.40455725051929 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Numerics.Tests.Perf_BigInteger

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToStringD - Duration of single invocation 457.76 ns 770.69 ns 1.68 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_BigInteger.ToStringD(numberString: 123)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -88.30284517181845 (T) = (0 -773.216228520436) / Math.Sqrt((29.792041467439912 / (7)) + (25.472283889368697 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.6884220683811353 = (457.95197954371577 - 773.216228520436) / 457.95197954371577 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IntegerFormatting - Duration of single invocation 6.34 ms 45.51 ms 7.18 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.LowLevelPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.LowLevelPerf.IntegerFormatting

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -192.70437043581225 (T) = (0 -45726037.119097225) / Math.Sqrt((750142279.8391557 / (7)) + (125144018659.7205 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -6.238286516149628 = (6317246.079866561 - 45726037.119097225) / 6317246.079866561 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in PerfLabTests.BlockCopyPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CallBlockCopy - Duration of single invocation 116.22 ns 151.20 ns 1.30 0.10 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.BlockCopyPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.BlockCopyPerf.CallBlockCopy(numElements: 10)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -19.574283770227478 (T) = (0 -151.22652466839182) / Math.Sqrt((5.545245131534006 / (7)) + (7.681574524377537 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.31062263138136575 = (115.38525357905851 - 151.22652466839182) / 115.38525357905851 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Remove_IntInt - Duration of single invocation 310.30 ns 326.73 ns 1.05 0.17 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 0, i2: 8)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -4.805163093930034 (T) = (0 -331.47128930395) / Math.Sqrt((20.19095320067902 / (7)) + (30.165116981927095 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.05501470909673039 = (314.18641507637847 - 331.47128930395) / 314.18641507637847 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Threading.Tests.Perf_Lock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReaderWriterLockSlimPerf - Duration of single invocation 447.26 ns 495.72 ns 1.11 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Threading.Tests.Perf_Lock*'

Payloads

Baseline
Compare

Histogram

System.Threading.Tests.Perf_Lock.ReaderWriterLockSlimPerf

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -10.329299861259933 (T) = (0 -490.93573881226456) / Math.Sqrt((70.12852835875252 / (7)) + (19.48928884899319 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.09349763439689746 = (448.95912288190084 - 490.93573881226456) / 448.95912288190084 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot added arm64 untriaged New issue has not been triaged by the area owner labels Dec 6, 2022
@kotlarmilos
Copy link
Member

Regressions caused by ab35b89 and 5af47bb

Decent regression in UInt32ToDecStr

@kotlarmilos kotlarmilos removed the untriaged New issue has not been triaged by the area owner label Dec 7, 2022
@kotlarmilos kotlarmilos transferred this issue from dotnet/perf-autofiling-issues Dec 7, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Dec 7, 2022
@kotlarmilos kotlarmilos added area-Codegen-Interpreter-mono and removed untriaged New issue has not been triaged by the area owner labels Dec 7, 2022
@ghost
Copy link

ghost commented Dec 7, 2022

Tagging subscribers to this area: @BrzVlad
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Tests.Perf_Version

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Ctor3 - Duration of single invocation 50.11 ns 86.08 ns 1.72 0.03 False
Ctor2 - Duration of single invocation 47.39 ns 71.32 ns 1.51 0.05 False
Ctor4 - Duration of single invocation 51.76 ns 102.99 ns 1.99 0.10 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Version*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Version.Ctor3

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -102.11659169159766 (T) = (0 -86.74571476872939) / Math.Sqrt((0.12230139019819382 / (7)) + (0.34097203649353724 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.7430184405411777 = (49.76752554711717 - 86.74571476872939) / 49.76752554711717 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.Ctor2

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -130.89007952591638 (T) = (0 -71.46053405222891) / Math.Sqrt((0.12904052784230644 / (7)) + (0.047092010962607275 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.5114678426116724 = (47.278898060280206 - 71.46053405222891) / 47.278898060280206 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.Ctor4

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -69.91395572248736 (T) = (0 -101.93066727444004) / Math.Sqrt((1.471178010925077 / (7)) + (0.8878575041028844 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.9529804518972763 = (52.19236432982046 - 101.93066727444004) / 52.19236432982046 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Tests.Perf_Int16

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 50.75 ns 363.06 ns 7.15 0.10 False
ToString - Duration of single invocation 24.27 ns 50.49 ns 2.08 0.13 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int16*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int16.ToString(value: 32767)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -56.388800046236824 (T) = (0 -369.86237428362614) / Math.Sqrt((0.4866799662928942 / (7)) + (96.02840362269899 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -6.326084703684954 = (50.485680857278204 - 369.86237428362614) / 50.485680857278204 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int16.ToString(value: 0)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -80.48997064927724 (T) = (0 -50.725437409996665) / Math.Sqrt((0.3549137971977588 / (7)) + (0.15089070297483032 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -1.0172795589575365 = (25.145467411670943 - 50.725437409996665) / 25.145467411670943 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Numerics.Tests.Perf_Matrix4x4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CreatePerspectiveFieldOfViewBenchmark - Duration of single invocation 103.09 ns 245.60 ns 2.38 0.19 False
CreatePerspectiveBenchmark - Duration of single invocation 89.33 ns 170.42 ns 1.91 0.17 False
CreatePerspectiveOffCenterBenchmark - Duration of single invocation 101.51 ns 178.94 ns 1.76 0.13 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Matrix4x4*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveFieldOfViewBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -47.407465925003 (T) = (0 -244.7283279584433) / Math.Sqrt((16.955750812608382 / (6)) + (19.593654446125736 / (3))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (3) - 2, .025) and -1.4543703149147356 = (99.71124832763678 - 244.7283279584433) / 99.71124832763678 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -28.27856125928147 (T) = (0 -171.1409131619612) / Math.Sqrt((7.888483020637348 / (6)) + (21.182069958258506 / (3))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (3) - 2, .025) and -0.9164335188808791 = (89.30177409018638 - 171.1409131619612) / 89.30177409018638 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveOffCenterBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -31.55669924340902 (T) = (0 -183.25801278540254) / Math.Sqrt((7.261748400179219 / (7)) + (17.08249335847475 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.807574409516796 = (101.38338528171097 - 183.25801278540254) / 101.38338528171097 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in LinqBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Where01ForX - Duration of single invocation 5.23 secs 5.63 secs 1.08 0.04 False
Where01LinqQueryX - Duration of single invocation 5.42 secs 5.88 secs 1.09 0.03 False
Where01LinqMethodX - Duration of single invocation 5.55 secs 6.02 secs 1.08 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'LinqBenchmarks*'

Payloads

Baseline
Compare

Histogram

LinqBenchmarks.Where01ForX

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -14.936972236005152 (T) = (0 -5631493356.288889) / Math.Sqrt((3694169213402831.5 / (7)) + (43721919026826.195 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.06583450163764645 = (5283647083.70403 - 5631493356.288889) / 5283647083.70403 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### LinqBenchmarks.Where01LinqQueryX

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -13.734392138221246 (T) = (0 -5893148331.555554) / Math.Sqrt((3774102618403431.5 / (7)) + (610818891582227.5 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.06782461999228462 = (5518835416.623129 - 5893148331.555554) / 5518835416.623129 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### LinqBenchmarks.Where01LinqMethodX

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -9.86345996511963 (T) = (0 -5980419743.822223) / Math.Sqrt((2693421689581077.5 / (7)) + (5661386090254048 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.08531989186277468 = (5510282994.590477 - 5980419743.822223) / 5510282994.590477 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Perf_Convert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ChangeType - Duration of single invocation 242.81 ns 550.82 ns 2.27 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Perf_Convert*'

Payloads

Baseline
Compare

Histogram

System.Perf_Convert.ChangeType

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -278.32962679684823 (T) = (0 -549.827499101074) / Math.Sqrt((3.4844707188457495 / (7)) + (2.212745495100914 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -1.286444151406167 = (240.4727440042343 - 549.827499101074) / 240.4727440042343 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Tests.Perf_Byte

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 32.69 ns 318.81 ns 9.75 0.23 False
ToString - Duration of single invocation 5.57 ns 38.47 ns 6.91 0.53 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Byte*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Byte.ToString(value: 255)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -195.61171260977292 (T) = (0 -320.29023052152485) / Math.Sqrt((1.5811890951979486 / (7)) + (5.9246995311702335 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -9.641055212020607 = (30.09948018686256 - 320.29023052152485) / 30.09948018686256 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Byte.ToString(value: 0)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -214.07001758349224 (T) = (0 -38.46763301637973) / Math.Sqrt((0.16976291339258173 / (7)) + (0.00026408657920728166 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -6.58722563160005 = (5.07005259684987 - 38.46763301637973) / 5.07005259684987 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Tests.Perf_SByte

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 48.55 ns 335.43 ns 6.91 0.13 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_SByte*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_SByte.ToString(value: 127)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -131.10599664117586 (T) = (0 -331.373923544455) / Math.Sqrt((2.5541456821294104 / (7)) + (12.777817799214688 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -5.701772953166281 = (49.445710241182674 - 331.373923544455) / 49.445710241182674 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Tests.Perf_Int32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 26.24 ns 50.38 ns 1.92 0.07 False
ToString - Duration of single invocation 58.89 ns 464.55 ns 7.89 0.10 False
ToString - Duration of single invocation 49.52 ns 356.36 ns 7.20 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int32.ToString(value: 4)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -397.7799585017025 (T) = (0 -50.363405275838055) / Math.Sqrt((0.005876323844439421 / (7)) + (0.0086335361594911 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.928832130475525 = (26.110828661601413 - 50.363405275838055) / 26.110828661601413 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.ToString(value: 2147483647)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -98.0488081341939 (T) = (0 -459.62939409938963) / Math.Sqrt((0.8294610613239252 / (7)) + (50.04965875014641 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -6.961932675112278 = (57.72836983865956 - 459.62939409938963) / 57.72836983865956 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.ToString(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -186.46041601920334 (T) = (0 -359.6577200179263) / Math.Sqrt((0.225274352613395 / (7)) + (8.157075364986618 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -6.13886458571935 = (50.38024124135775 - 359.6577200179263) / 50.38024124135775 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 1076cf52f60a845cf59e316345e2e599a6fcd3ac
Compare 042d3ee8a7f8f5d4c86f1f7814e2a6a8e7bb18b5
Diff Diff

Regressions in System.Tests.Perf_UInt32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 4.63 ns 38.32 ns 8.27 0.53 False
ToString - Duration of single invocation 28.91 ns 353.59 ns 12.23 0.21 False
ToString - Duration of single invocation 36.10 ns 445.83 ns 12.35 0.20 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt32.ToString(value: 0)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -121.83438545236052 (T) = (0 -38.26640494603868) / Math.Sqrt((0.5208478356031971 / (7)) + (0.01079345134623293 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -8.027440808630944 = (4.2388984605086515 - 38.26640494603868) / 4.2388984605086515 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt32.ToString(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -173.78822541441707 (T) = (0 -352.30509252906205) / Math.Sqrt((1.0149976066430988 / (7)) + (9.942195956545815 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -11.11336904509903 = (29.08398903867308 - 352.30509252906205) / 29.08398903867308 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt32.ToString(value: 4294967295)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/1/2022 1:41:45 AM, 12/6/2022 2:12:43 AM falls between 11/26/2022 10:41:09 PM and 12/6/2022 2:12:43 AM.
IsRegressionStdDev: Marked as regression because -140.96911563937482 (T) = (0 -447.88611016078016) / Math.Sqrt((0.1495139313555134 / (7)) + (25.547906105451304 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -11.443840500940496 = (35.992594900820954 - 447.88611016078016) / 35.992594900820954 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: stephentoub
Labels:

area-Codegen-Interpreter-mono, refs/heads/main, Regression, arm64
Milestone: -

This was referenced Dec 7, 2022
Closed
Closed
@stephentoub
Copy link
Member

stephentoub commented Dec 7, 2022 •
edited
Loading

Presumably these are all due to removing the C implementation of UInt32ToDecStr that the mono interpreter was using in preference to it using the managed implenentstion.
#79061 (comment)
@BrzVlad, what would you like to do about it? Accept it since you said it was going away anyway? Me rewrite it and put it back?

@BrzVlad
Copy link
Member

BrzVlad commented Dec 8, 2022

Overall I'm against bring this intrinsic back, especially now that managed and native code is diverging. The regression doesn't seem to be that widespread and there is significant room for improvement on the interpreter side that can help with this.

Testing locally byte.ToString, the regression went down from 7.25x to about 4.5x with #79392. I think the regression can be brought under 3x without complex optimizations and under 2x with more advanced optimizations that are planned for long term.

@stephentoub
Copy link
Member

stephentoub commented Dec 8, 2022 •
edited
Loading

Overall I'm against bring this intrinsic back, especially now that managed and native code is diverging. The regression doesn't seem to be that widespread and there is significant room for improvement on the interpreter side that can help with this.

Ok, thanks, sounds good. I will unassign myself then.

I think the regression can be brought under 3x without complex optimizations and under 2x with more advanced optimizations that are planned for long term

Nice.

@stephentoub stephentoub assigned BrzVlad and unassigned stephentoub Dec 8, 2022
@kotlarmilos kotlarmilos mentioned this issue Dec 9, 2022
Closed
@lewing
Copy link
Member

lewing commented Feb 16, 2023

https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x64_ubuntu%2018.04_CompliationMode%3dwasm_RunKind%3dmicro%2fSystem.Tests.Perf_UInt16.ToString(value%3a%2065535).html we've still got a pretty big regression in number formatting

@lewing
Copy link
Member

lewing commented Feb 16, 2023

for net7 -> p1
image

On the good side there have been some fixes that should help since p1

@kotlarmilos
Copy link
Member

Mitigated by #82093.

@kotlarmilos
Copy link
Member

Mitigated by #83490.

@kotlarmilos kotlarmilos mentioned this issue Apr 4, 2023
Closed
@ghost ghost locked as resolved and limited conversation to collaborators Apr 28, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@BrzVlad BrzVlad

Labels
arch-arm64 area-Codegen-Interpreter-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@lewing @jeffhandley @stephentoub @BrzVlad @kotlarmilos