Regressions in System.Reflection.Attributes #91620

performanceautofiler · 2023-09-05T09:28:08Z

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Reflection.Attributes

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
GetCustomAttributesClassHitInherit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.36 μs	1.53 μs	1.12	0.02	False
GetCustomAttributesMethodOverrideMiss - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.25 μs	1.40 μs	1.12	0.02	False
GetCustomAttributesMethodOverrideMissInherit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.30 μs	1.44 μs	1.11	0.03	False
GetCustomAttributesMethodBaseHitInherit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.28 μs	1.46 μs	1.14	0.03	False
IsDefinedMethodOverrideHitInherit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	546.92 ns	592.26 ns	1.08	0.08	False
GetCustomAttributesMethodBaseHit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.26 μs	1.38 μs	1.10	0.02	False
GetCustomAttributesClassMiss - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	118.04 ns	134.20 ns	1.14	0.18	False
GetCustomAttributesClassHit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.20 μs	1.35 μs	1.13	0.04	False
GetCustomAttributesMethodOverrideHit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.26 μs	1.42 μs	1.13	0.03	False
IsDefinedMethodOverrideHit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	541.04 ns	570.39 ns	1.05	0.09	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Reflection.Attributes*'

Payloads

Baseline
Compare

System.Reflection.Attributes.GetCustomAttributesClassHitInherit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.528459583275599 > 1.4169015459427434.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -10.913268524832555 (T) = (0 -1527.2288803264742) / Math.Sqrt((6785.835897417417 / (15)) + (258.66657330844026 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.18256736678750765 = (1291.451906435786 - 1527.2288803264742) / 1291.451906435786 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodOverrideMiss

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3958803354881135 > 1.2962813860171623.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -10.842527397556537 (T) = (0 -1398.9233398428814) / Math.Sqrt((5392.148814172355 / (15)) + (109.24380105005997 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.17396533504286166 = (1191.6223572238669 - 1398.9233398428814) / 1191.6223572238669 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodOverrideMissInherit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4446282108078565 > 1.3551836266830741.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -11.670162152407054 (T) = (0 -1465.8801602397352) / Math.Sqrt((5394.127497576298 / (15)) + (169.7098343609407 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.18055008256030503 = (1241.6924803906868 - 1465.8801602397352) / 1241.6924803906868 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodBaseHitInherit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4640371721019545 > 1.3489420708351492.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -12.182806737233397 (T) = (0 -1460.9705877867384) / Math.Sqrt((4777.515214218393 / (15)) + (157.50922630949424 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.17764891484169476 = (1240.5824599966868 - 1460.9705877867384) / 1240.5824599966868 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.IsDefinedMethodOverrideHitInherit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 592.258236921529 > 560.4389762828821.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -11.44179021669637 (T) = (0 -580.033412910356) / Math.Sqrt((321.0939044530798 / (15)) + (90.5227602505153 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.11286971698933808 = (521.2051366439626 - 580.033412910356) / 521.2051366439626 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodBaseHit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3846096831637664 > 1.297663247763238.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -12.46802258748777 (T) = (0 -1401.3527985127155) / Math.Sqrt((4297.198613393701 / (15)) + (171.10278204892487 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.18073282478497166 = (1186.8500384648169 - 1401.3527985127155) / 1186.8500384648169 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesClassMiss

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 134.2017579952718 > 129.40321232462932.
IsChangePoint: Marked as a change because one of 5/8/2023 3:52:55 PM, 5/20/2023 9:58:12 PM, 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -3.7967124925090814 (T) = (0 -129.0076364046882) / Math.Sqrt((71.27567471250687 / (15)) + (44.57445500221441 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.08592151224851993 = (118.80014803055491 - 129.0076364046882) / 118.80014803055491 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesClassHit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3547490170266572 > 1.2567756945654949.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -10.848805860668076 (T) = (0 -1366.6844986978397) / Math.Sqrt((5640.9299552756975 / (15)) + (129.66462980546717 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.1839987267588616 = (1154.2955814142397 - 1366.6844986978397) / 1154.2955814142397 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodOverrideHit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4209621382846689 > 1.3027001898736688.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -12.824922425658633 (T) = (0 -1413.7425350811588) / Math.Sqrt((4389.421286301902 / (15)) + (111.80149560835834 / (17))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (17) - 2, .025) and -0.18612212092450317 = (1191.9030175233902 - 1413.7425350811588) / 1191.9030175233902 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.IsDefinedMethodOverrideHit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 570.3874751948115 > 564.3027335366662.
IsChangePoint: Marked as a change because one of 7/28/2023 9:24:36 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -9.168925948836067 (T) = (0 -574.2076647890194) / Math.Sqrt((486.7927472721306 / (15)) + (59.13916857348186 / (17))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (17) - 2, .025) and -0.10584818493372372 = (519.2463781304964 - 574.2076647890194) / 519.2463781304964 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.ReadOnlyMemory<Byte>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Pin - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	28.21 ns	34.09 ns	1.21	0.15	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlyMemory&lt;Byte&gt;*'

Payloads

Baseline
Compare

System.Memory.ReadOnlyMemory<Byte>.Pin(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.08518220088306 > 29.54765601731688.
IsChangePoint: Marked as a change because one of 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -14.43343811946705 (T) = (0 -33.76940903653108) / Math.Sqrt((0.3875209848907069 / (15)) + (2.6249866282744834 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.2151973776543225 = (27.789237911059082 - 33.76940903653108) / 27.789237911059082 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Memory<Char>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Pin - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	25.00 ns	32.02 ns	1.28	0.35	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Memory&lt;Char&gt;*'

Payloads

Baseline
Compare

System.Memory.Memory<Char>.Pin(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.01815470857049 > 28.886769923921584.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -16.87117564116427 (T) = (0 -32.92122084075074) / Math.Sqrt((0.8884468151391294 / (15)) + (0.7180230189215515 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.19238249633462384 = (27.60961431583436 - 32.92122084075074) / 27.60961431583436 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Memory<Byte>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Pin - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	26.48 ns	35.79 ns	1.35	0.13	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Memory&lt;Byte&gt;*'

Payloads

Baseline
Compare

System.Memory.Memory<Byte>.Pin(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.79324923061559 > 28.619763638535943.
IsChangePoint: Marked as a change because one of 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -14.878200331693714 (T) = (0 -33.59812764829385) / Math.Sqrt((0.9729217809265139 / (15)) + (1.8366071497788568 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.22086374004927342 = (27.519965206713277 - 33.59812764829385) / 27.519965206713277 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Tests.Add_Remove_SteadyState<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
ConcurrentStack - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	14.54 ns	16.22 ns	1.12	0.09	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Add_Remove_SteadyState&lt;String&gt;*'

Payloads

Baseline
Compare

System.Collections.Tests.Add_Remove_SteadyState<String>.ConcurrentStack(Count: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.217058109647507 > 15.274684770621837.
IsChangePoint: Marked as a change because one of 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -12.066344419518478 (T) = (0 -16.087218183748288) / Math.Sqrt((0.023524243234109836 / (15)) + (0.2311640740684595 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.09894997815702222 = (14.638717415261356 - 16.087218183748288) / 14.638717415261356 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Threading.Tests.Perf_CancellationToken

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
CancelAfter - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	73.56 ns	81.68 ns	1.11	0.19	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Threading.Tests.Perf_CancellationToken*'

Payloads

Baseline
Compare

System.Threading.Tests.Perf_CancellationToken.CancelAfter

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 81.6812033444421 > 76.80067166807122.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -14.993148953512394 (T) = (0 -84.62210903742988) / Math.Sqrt((3.371249112201878 / (15)) + (6.438477036176775 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.15636131774268336 = (73.17964354136258 - 84.62210903742988) / 73.17964354136258 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

The text was updated successfully, but these errors were encountered:

performanceautofiler · 2023-09-05T09:28:12Z

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in Exceptions.Handling

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
CatchAndRethrowDeep - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	134.84 μs	143.17 μs	1.06	0.04	False
CatchAndThrowOtherDeep - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	137.88 μs	147.53 μs	1.07	0.03	False
CatchAndThrowOtherDeep - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	142.65 μs	152.74 μs	1.07	0.04	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Exceptions.Handling*'

Payloads

Baseline
Compare

Exceptions.Handling.CatchAndRethrowDeep(kind: Software)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 143.17291939711663 > 140.42736462477896.
IsChangePoint: Marked as a change because one of 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -17.996352363217696 (T) = (0 -144392.75476951696) / Math.Sqrt((1787165.3903544426 / (15)) + (5071532.512922934 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.0856751630651855 = (132998.11921813985 - 144392.75476951696) / 132998.11921813985 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.CatchAndThrowOtherDeep(kind: Software)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 147.52951482479787 > 144.28085045434597.
IsChangePoint: Marked as a change because one of 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -34.32254420750751 (T) = (0 -149055.394191505) / Math.Sqrt((637402.2029485131 / (15)) + (1250463.5633604939 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.0834818354589256 = (137570.73659510893 - 149055.394191505) / 137570.73659510893 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.CatchAndThrowOtherDeep(kind: Hardware)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 152.7421743697479 > 146.84007882882887.
IsChangePoint: Marked as a change because one of 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -17.186504674377478 (T) = (0 -151168.96700148913) / Math.Sqrt((3475663.6805127747 / (15)) + (2367653.9394168127 / (17))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (17) - 2, .025) and -0.07439925697076112 = (140700.9228838317 - 151168.96700148913) / 140700.9228838317 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CtorFromCollectionNonGeneric<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
ArrayList - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	12.22 μs	20.32 μs	1.66	0.16	False
Queue - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	27.42 μs	31.09 μs	1.13	0.21	False
Stack - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	26.67 μs	30.16 μs	1.13	0.13	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollectionNonGeneric&lt;Int32&gt;*'

Payloads

Baseline
Compare

System.Collections.CtorFromCollectionNonGeneric<Int32>.ArrayList(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.321774054982818 > 13.086170048504457.
IsChangePoint: Marked as a change because one of 5/23/2023 1:41:12 PM, 7/13/2023 3:06:28 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -29.302496897542813 (T) = (0 -19233.329163791044) / Math.Sqrt((268735.4945460489 / (15)) + (685796.9946775256 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.5639229150855712 = (12298.13117914362 - 19233.329163791044) / 12298.13117914362 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.CtorFromCollectionNonGeneric<Int32>.Queue(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 31.09429215879265 > 27.460831912511715.
IsChangePoint: Marked as a change because one of 5/20/2023 9:58:12 PM, 7/23/2023 3:42:40 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -13.019483257160964 (T) = (0 -30624.434204615627) / Math.Sqrt((1838488.876566457 / (15)) + (283760.619320864 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.18781655494716487 = (25782.12441733297 - 30624.434204615627) / 25782.12441733297 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.CtorFromCollectionNonGeneric<Int32>.Stack(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.162131109022557 > 27.50292026651317.
IsChangePoint: Marked as a change because one of 5/20/2023 9:58:12 PM, 7/23/2023 3:42:40 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -12.391419150166792 (T) = (0 -30655.745199551) / Math.Sqrt((1600493.3219252268 / (15)) + (1030241.2450783377 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.19568742149869894 = (25638.59470991713 - 30655.745199551) / 25638.59470991713 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Array

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
ArrayRetrieve2D - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	2.25 ms	2.56 ms	1.14	0.19	False
ArrayRetrieve3D - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	2.55 ms	2.74 ms	1.07	0.23	False
ArrayRetrieve1D - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	2.06 ms	2.30 ms	1.11	0.17	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Array*'

Payloads

Baseline
Compare

System.Tests.Perf_Array.ArrayRetrieve2D

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.556898214285714 > 2.2746247500000005.
IsChangePoint: Marked as a change because one of 5/8/2023 11:50:12 PM, 7/20/2023 6:02:40 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -9.400245844919876 (T) = (0 -2466870.8861343684) / Math.Sqrt((17943408908.716312 / (15)) + (3000162002.170201 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.16370772437037456 = (2119837.1674202573 - 2466870.8861343684) / 2119837.1674202573 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Array.ArrayRetrieve3D

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.7422992361111116 > 2.5376828883928573.
IsChangePoint: Marked as a change because one of 5/8/2023 11:50:12 PM, 7/22/2023 10:20:55 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -9.426037258005783 (T) = (0 -2755120.4364019474) / Math.Sqrt((22929960296.34341 / (15)) + (2435530584.960624 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.16219630943738378 = (2370615.3719724803 - 2755120.4364019474) / 2370615.3719724803 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Array.ArrayRetrieve1D

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.2953867857142853 > 2.1041535703125005.
IsChangePoint: Marked as a change because one of 5/16/2023 12:30:38 AM, 7/23/2023 3:42:40 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -13.215636567401535 (T) = (0 -2307986.747986897) / Math.Sqrt((10731376517.83205 / (15)) + (1221158819.4495716 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.1908367391969799 = (1938121.8869206603 - 2307986.747986897) / 1938121.8869206603 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Cache

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
IsMatch - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	35.29 ms	56.72 ms	1.61	0.34	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Cache*'

Payloads

Baseline
Compare

System.Text.RegularExpressions.Tests.Perf_Regex_Cache.IsMatch(total: 40000, unique: 7, cacheSize: 0)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 56.71656315789474 > 37.9217145.
IsChangePoint: Marked as a change because one of 7/18/2023 9:23:54 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -11.213967356373555 (T) = (0 -51376407.96697627) / Math.Sqrt((269339751899.43912 / (15)) + (32407902952561.086 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.41709996936755195 = (36254610.87964417 - 51376407.96697627) / 36254610.87964417 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.ReadOnlyMemory<Char>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Pin - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	28.37 ns	33.29 ns	1.17	0.36	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlyMemory&lt;Char&gt;*'

Payloads

Baseline
Compare

System.Memory.ReadOnlyMemory<Char>.Pin(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 33.28556139024886 > 29.553225944957155.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -13.706095048948457 (T) = (0 -33.120315213441444) / Math.Sqrt((1.4221603597106312 / (15)) + (1.8185388500628286 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.22419206243664733 = (27.05483578084827 - 33.120315213441444) / 27.05483578084827 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.ComponentModel.Tests.Perf_TypeDescriptorTests

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
GetConverter - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	175.80 ns	195.58 ns	1.11	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.ComponentModel.Tests.Perf_TypeDescriptorTests*'

Payloads

Baseline
Compare

System.ComponentModel.Tests.Perf_TypeDescriptorTests.GetConverter(typeToConvert: typeof(System.ComponentModel.Tests.SomeEnum))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 195.5816456361724 > 183.93036025270203.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -35.470971027840925 (T) = (0 -199.18669174964492) / Math.Sqrt((1.515775062777477 / (15)) + (6.769023895381382 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.14025683875144893 = (174.6858119857883 - 199.18669174964492) / 174.6858119857883 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

jeffschwMSFT · 2023-09-06T14:43:10Z

@DrewScoggins is there a reason that this perf regression, and recent filed issues, do not have a commit range to help identify the cause? Without the range, these will be assigned the test owner, but that is often not the best first place to start.

ghost · 2023-09-06T14:43:30Z

Tagging subscribers to this area: @dotnet/area-system-reflection
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Reflection.Attributes

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
GetCustomAttributesClassHitInherit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.36 μs	1.53 μs	1.12	0.02	False
GetCustomAttributesMethodOverrideMiss - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.25 μs	1.40 μs	1.12	0.02	False
GetCustomAttributesMethodOverrideMissInherit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.30 μs	1.44 μs	1.11	0.03	False
GetCustomAttributesMethodBaseHitInherit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.28 μs	1.46 μs	1.14	0.03	False
IsDefinedMethodOverrideHitInherit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	546.92 ns	592.26 ns	1.08	0.08	False
GetCustomAttributesMethodBaseHit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.26 μs	1.38 μs	1.10	0.02	False
GetCustomAttributesClassMiss - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	118.04 ns	134.20 ns	1.14	0.18	False
GetCustomAttributesClassHit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.20 μs	1.35 μs	1.13	0.04	False
GetCustomAttributesMethodOverrideHit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.26 μs	1.42 μs	1.13	0.03	False
IsDefinedMethodOverrideHit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	541.04 ns	570.39 ns	1.05	0.09	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Reflection.Attributes*'

Payloads

Baseline
Compare

System.Reflection.Attributes.GetCustomAttributesClassHitInherit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.528459583275599 > 1.4169015459427434.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -10.913268524832555 (T) = (0 -1527.2288803264742) / Math.Sqrt((6785.835897417417 / (15)) + (258.66657330844026 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.18256736678750765 = (1291.451906435786 - 1527.2288803264742) / 1291.451906435786 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodOverrideMiss

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3958803354881135 > 1.2962813860171623.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -10.842527397556537 (T) = (0 -1398.9233398428814) / Math.Sqrt((5392.148814172355 / (15)) + (109.24380105005997 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.17396533504286166 = (1191.6223572238669 - 1398.9233398428814) / 1191.6223572238669 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodOverrideMissInherit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4446282108078565 > 1.3551836266830741.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -11.670162152407054 (T) = (0 -1465.8801602397352) / Math.Sqrt((5394.127497576298 / (15)) + (169.7098343609407 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.18055008256030503 = (1241.6924803906868 - 1465.8801602397352) / 1241.6924803906868 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodBaseHitInherit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4640371721019545 > 1.3489420708351492.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -12.182806737233397 (T) = (0 -1460.9705877867384) / Math.Sqrt((4777.515214218393 / (15)) + (157.50922630949424 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.17764891484169476 = (1240.5824599966868 - 1460.9705877867384) / 1240.5824599966868 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.IsDefinedMethodOverrideHitInherit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 592.258236921529 > 560.4389762828821.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -11.44179021669637 (T) = (0 -580.033412910356) / Math.Sqrt((321.0939044530798 / (15)) + (90.5227602505153 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.11286971698933808 = (521.2051366439626 - 580.033412910356) / 521.2051366439626 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodBaseHit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3846096831637664 > 1.297663247763238.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -12.46802258748777 (T) = (0 -1401.3527985127155) / Math.Sqrt((4297.198613393701 / (15)) + (171.10278204892487 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.18073282478497166 = (1186.8500384648169 - 1401.3527985127155) / 1186.8500384648169 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesClassMiss

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 134.2017579952718 > 129.40321232462932.
IsChangePoint: Marked as a change because one of 5/8/2023 3:52:55 PM, 5/20/2023 9:58:12 PM, 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -3.7967124925090814 (T) = (0 -129.0076364046882) / Math.Sqrt((71.27567471250687 / (15)) + (44.57445500221441 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.08592151224851993 = (118.80014803055491 - 129.0076364046882) / 118.80014803055491 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesClassHit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3547490170266572 > 1.2567756945654949.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -10.848805860668076 (T) = (0 -1366.6844986978397) / Math.Sqrt((5640.9299552756975 / (15)) + (129.66462980546717 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.1839987267588616 = (1154.2955814142397 - 1366.6844986978397) / 1154.2955814142397 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodOverrideHit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4209621382846689 > 1.3027001898736688.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -12.824922425658633 (T) = (0 -1413.7425350811588) / Math.Sqrt((4389.421286301902 / (15)) + (111.80149560835834 / (17))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (17) - 2, .025) and -0.18612212092450317 = (1191.9030175233902 - 1413.7425350811588) / 1191.9030175233902 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.IsDefinedMethodOverrideHit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 570.3874751948115 > 564.3027335366662.
IsChangePoint: Marked as a change because one of 7/28/2023 9:24:36 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -9.168925948836067 (T) = (0 -574.2076647890194) / Math.Sqrt((486.7927472721306 / (15)) + (59.13916857348186 / (17))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (17) - 2, .025) and -0.10584818493372372 = (519.2463781304964 - 574.2076647890194) / 519.2463781304964 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.ReadOnlyMemory<Byte>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Pin - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	28.21 ns	34.09 ns	1.21	0.15	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlyMemory&lt;Byte&gt;*'

Payloads

Baseline
Compare

System.Memory.ReadOnlyMemory<Byte>.Pin(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.08518220088306 > 29.54765601731688.
IsChangePoint: Marked as a change because one of 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -14.43343811946705 (T) = (0 -33.76940903653108) / Math.Sqrt((0.3875209848907069 / (15)) + (2.6249866282744834 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.2151973776543225 = (27.789237911059082 - 33.76940903653108) / 27.789237911059082 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Memory<Char>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Pin - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	25.00 ns	32.02 ns	1.28	0.35	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Memory&lt;Char&gt;*'

Payloads

Baseline
Compare

System.Memory.Memory<Char>.Pin(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.01815470857049 > 28.886769923921584.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -16.87117564116427 (T) = (0 -32.92122084075074) / Math.Sqrt((0.8884468151391294 / (15)) + (0.7180230189215515 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.19238249633462384 = (27.60961431583436 - 32.92122084075074) / 27.60961431583436 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Memory<Byte>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Pin - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	26.48 ns	35.79 ns	1.35	0.13	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Memory&lt;Byte&gt;*'

Payloads

Baseline
Compare

System.Memory.Memory<Byte>.Pin(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.79324923061559 > 28.619763638535943.
IsChangePoint: Marked as a change because one of 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -14.878200331693714 (T) = (0 -33.59812764829385) / Math.Sqrt((0.9729217809265139 / (15)) + (1.8366071497788568 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.22086374004927342 = (27.519965206713277 - 33.59812764829385) / 27.519965206713277 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Tests.Add_Remove_SteadyState<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
ConcurrentStack - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	14.54 ns	16.22 ns	1.12	0.09	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Add_Remove_SteadyState&lt;String&gt;*'

Payloads

Baseline
Compare

System.Collections.Tests.Add_Remove_SteadyState<String>.ConcurrentStack(Count: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.217058109647507 > 15.274684770621837.
IsChangePoint: Marked as a change because one of 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -12.066344419518478 (T) = (0 -16.087218183748288) / Math.Sqrt((0.023524243234109836 / (15)) + (0.2311640740684595 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.09894997815702222 = (14.638717415261356 - 16.087218183748288) / 14.638717415261356 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Threading.Tests.Perf_CancellationToken

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
CancelAfter - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	73.56 ns	81.68 ns	1.11	0.19	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Threading.Tests.Perf_CancellationToken*'

Payloads

Baseline
Compare

System.Threading.Tests.Perf_CancellationToken.CancelAfter

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 81.6812033444421 > 76.80067166807122.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -14.993148953512394 (T) = (0 -84.62210903742988) / Math.Sqrt((3.371249112201878 / (15)) + (6.438477036176775 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.15636131774268336 = (73.17964354136258 - 84.62210903742988) / 73.17964354136258 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	`area-System.Reflection`, `os-windows`, `arch-x64`, `untriaged`, `runtime-coreclr`, `PGO`, `needs-area-label`
Milestone:	-

steveharter · 2023-09-06T20:13:13Z

@DrewScoggins is there a reason that this perf regression, and recent filed issues, do not have a commit range to help identify the cause? Without the range, these will be assigned the test owner, but that is often not the best first place to start.

Also, the apparent regression was from August 29-30 but the graph pics show a perf improvement, not a regression. However, there was a regression in late July or early August but this issue isn't for that so this is confusing.

DrewScoggins · 2023-09-06T20:25:37Z

@DrewScoggins is there a reason that this perf regression, and recent filed issues, do not have a commit range to help identify the cause? Without the range, these will be assigned the test owner, but that is often not the best first place to start.

Those were always found "manually" using the graphs in the issue, and then added. I assume that the triage v-team has just not been grabbing them lately. For this filed issue this is the range, 87f3817...9c3f8b3.

@DrewScoggins is there a reason that this perf regression, and recent filed issues, do not have a commit range to help identify the cause? Without the range, these will be assigned the test owner, but that is often not the best first place to start.

Also, the apparent regression was from August 29-30 but the graph pics show a perf improvement, not a regression. However, there was a regression in late July or early August but this issue isn't for that so this is confusing.

So first, the regression in late July was already filed here, dotnet/perf-autofiling-issues#20306. In the case here we are seeing more regressions. It can sometimes be clearer to see if you click on the Test Report link below the tables. In the case of the System.Reflection tests we can see on 8/29 at 21:19 we see a jump in that tests running time. Also on some of the other tests the scale, because the values used to be much higher, is thrown off and it can be difficult to see the regression from just the image in the issue.

Hope this helps.

steveharter · 2023-09-06T21:42:58Z

Those were always found "manually" using the graphs in the issue, and then added. I assume that the triage v-team has just not been grabbing them lately. For this filed issue this is the range, 87f3817...9c3f8b3.

Where did this range come from? The various "Diff" links above show 86252f7...03d25cd.

If I understand correctly this issue lists the following regressions:

System.Reflection.Attributes 5-12%
System.Memory.ReadOnlyMemory<Byte> 21%
System.Memory.Memory<Char> 28%
System.Memory.Memory<Byte> 35%
System.Collections.Tests.Add_Remove_SteadyState<String> 12%
System.Threading.Tests.Perf_CancellationToken 11%
Exceptions.Handling 6-7%
System.Collections.CtorFromCollectionNonGeneric<Int32> 13-66%
System.Tests.Perf_Array 7-14%
System.Text.RegularExpressions.Tests.Perf_Regex_Cache 61%
System.Memory.ReadOnlyMemory<Char> 17%
System.ComponentModel.Tests.Perf_TypeDescriptorTests 11%

but this issue is currently assigned to System.Reflection.Attributes which is on the lower scale from the regression %.

steveharter · 2023-09-06T21:49:28Z

Also the "test report" for at least the first couple of areas have this larger commit range: 976c2ab...9eb0266

steveharter · 2023-09-06T21:58:19Z

If the smaller commit range 87f3817...9c3f8b3 is used, then I suspect #89311.

steveharter · 2023-09-06T22:00:29Z

Assigning to area-Host based on #89311 and also because that calls out PGO-related issues.

ghost · 2023-09-06T22:00:46Z

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Reflection.Attributes

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
GetCustomAttributesClassHitInherit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.36 μs	1.53 μs	1.12	0.02	False
GetCustomAttributesMethodOverrideMiss - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.25 μs	1.40 μs	1.12	0.02	False
GetCustomAttributesMethodOverrideMissInherit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.30 μs	1.44 μs	1.11	0.03	False
GetCustomAttributesMethodBaseHitInherit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.28 μs	1.46 μs	1.14	0.03	False
IsDefinedMethodOverrideHitInherit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	546.92 ns	592.26 ns	1.08	0.08	False
GetCustomAttributesMethodBaseHit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.26 μs	1.38 μs	1.10	0.02	False
GetCustomAttributesClassMiss - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	118.04 ns	134.20 ns	1.14	0.18	False
GetCustomAttributesClassHit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.20 μs	1.35 μs	1.13	0.04	False
GetCustomAttributesMethodOverrideHit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	1.26 μs	1.42 μs	1.13	0.03	False
IsDefinedMethodOverrideHit - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	541.04 ns	570.39 ns	1.05	0.09	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Reflection.Attributes*'

Payloads

Baseline
Compare

System.Reflection.Attributes.GetCustomAttributesClassHitInherit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.528459583275599 > 1.4169015459427434.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -10.913268524832555 (T) = (0 -1527.2288803264742) / Math.Sqrt((6785.835897417417 / (15)) + (258.66657330844026 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.18256736678750765 = (1291.451906435786 - 1527.2288803264742) / 1291.451906435786 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodOverrideMiss

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3958803354881135 > 1.2962813860171623.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -10.842527397556537 (T) = (0 -1398.9233398428814) / Math.Sqrt((5392.148814172355 / (15)) + (109.24380105005997 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.17396533504286166 = (1191.6223572238669 - 1398.9233398428814) / 1191.6223572238669 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodOverrideMissInherit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4446282108078565 > 1.3551836266830741.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -11.670162152407054 (T) = (0 -1465.8801602397352) / Math.Sqrt((5394.127497576298 / (15)) + (169.7098343609407 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.18055008256030503 = (1241.6924803906868 - 1465.8801602397352) / 1241.6924803906868 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodBaseHitInherit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4640371721019545 > 1.3489420708351492.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -12.182806737233397 (T) = (0 -1460.9705877867384) / Math.Sqrt((4777.515214218393 / (15)) + (157.50922630949424 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.17764891484169476 = (1240.5824599966868 - 1460.9705877867384) / 1240.5824599966868 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.IsDefinedMethodOverrideHitInherit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 592.258236921529 > 560.4389762828821.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -11.44179021669637 (T) = (0 -580.033412910356) / Math.Sqrt((321.0939044530798 / (15)) + (90.5227602505153 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.11286971698933808 = (521.2051366439626 - 580.033412910356) / 521.2051366439626 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodBaseHit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3846096831637664 > 1.297663247763238.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -12.46802258748777 (T) = (0 -1401.3527985127155) / Math.Sqrt((4297.198613393701 / (15)) + (171.10278204892487 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.18073282478497166 = (1186.8500384648169 - 1401.3527985127155) / 1186.8500384648169 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesClassMiss

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 134.2017579952718 > 129.40321232462932.
IsChangePoint: Marked as a change because one of 5/8/2023 3:52:55 PM, 5/20/2023 9:58:12 PM, 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -3.7967124925090814 (T) = (0 -129.0076364046882) / Math.Sqrt((71.27567471250687 / (15)) + (44.57445500221441 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.08592151224851993 = (118.80014803055491 - 129.0076364046882) / 118.80014803055491 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesClassHit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3547490170266572 > 1.2567756945654949.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -10.848805860668076 (T) = (0 -1366.6844986978397) / Math.Sqrt((5640.9299552756975 / (15)) + (129.66462980546717 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.1839987267588616 = (1154.2955814142397 - 1366.6844986978397) / 1154.2955814142397 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodOverrideHit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4209621382846689 > 1.3027001898736688.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -12.824922425658633 (T) = (0 -1413.7425350811588) / Math.Sqrt((4389.421286301902 / (15)) + (111.80149560835834 / (17))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (17) - 2, .025) and -0.18612212092450317 = (1191.9030175233902 - 1413.7425350811588) / 1191.9030175233902 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.IsDefinedMethodOverrideHit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 570.3874751948115 > 564.3027335366662.
IsChangePoint: Marked as a change because one of 7/28/2023 9:24:36 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -9.168925948836067 (T) = (0 -574.2076647890194) / Math.Sqrt((486.7927472721306 / (15)) + (59.13916857348186 / (17))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (17) - 2, .025) and -0.10584818493372372 = (519.2463781304964 - 574.2076647890194) / 519.2463781304964 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.ReadOnlyMemory<Byte>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Pin - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	28.21 ns	34.09 ns	1.21	0.15	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlyMemory&lt;Byte&gt;*'

Payloads

Baseline
Compare

System.Memory.ReadOnlyMemory<Byte>.Pin(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.08518220088306 > 29.54765601731688.
IsChangePoint: Marked as a change because one of 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -14.43343811946705 (T) = (0 -33.76940903653108) / Math.Sqrt((0.3875209848907069 / (15)) + (2.6249866282744834 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.2151973776543225 = (27.789237911059082 - 33.76940903653108) / 27.789237911059082 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Memory<Char>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Pin - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	25.00 ns	32.02 ns	1.28	0.35	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Memory&lt;Char&gt;*'

Payloads

Baseline
Compare

System.Memory.Memory<Char>.Pin(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.01815470857049 > 28.886769923921584.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -16.87117564116427 (T) = (0 -32.92122084075074) / Math.Sqrt((0.8884468151391294 / (15)) + (0.7180230189215515 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.19238249633462384 = (27.60961431583436 - 32.92122084075074) / 27.60961431583436 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Memory<Byte>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Pin - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	26.48 ns	35.79 ns	1.35	0.13	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Memory&lt;Byte&gt;*'

Payloads

Baseline
Compare

System.Memory.Memory<Byte>.Pin(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.79324923061559 > 28.619763638535943.
IsChangePoint: Marked as a change because one of 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -14.878200331693714 (T) = (0 -33.59812764829385) / Math.Sqrt((0.9729217809265139 / (15)) + (1.8366071497788568 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.22086374004927342 = (27.519965206713277 - 33.59812764829385) / 27.519965206713277 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Tests.Add_Remove_SteadyState<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
ConcurrentStack - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	14.54 ns	16.22 ns	1.12	0.09	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Add_Remove_SteadyState&lt;String&gt;*'

Payloads

Baseline
Compare

System.Collections.Tests.Add_Remove_SteadyState<String>.ConcurrentStack(Count: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.217058109647507 > 15.274684770621837.
IsChangePoint: Marked as a change because one of 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -12.066344419518478 (T) = (0 -16.087218183748288) / Math.Sqrt((0.023524243234109836 / (15)) + (0.2311640740684595 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.09894997815702222 = (14.638717415261356 - 16.087218183748288) / 14.638717415261356 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	86252f7b723eeae2a827144ee9c15bc675028ad8
Compare	03d25cd9f3868c92554e27048a57c49a39ce07c9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Threading.Tests.Perf_CancellationToken

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
CancelAfter - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	73.56 ns	81.68 ns	1.11	0.19	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Threading.Tests.Perf_CancellationToken*'

Payloads

Baseline
Compare

System.Threading.Tests.Perf_CancellationToken.CancelAfter

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 81.6812033444421 > 76.80067166807122.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 8/29/2023 4:22:52 PM, 9/4/2023 9:28:57 AM falls between 8/26/2023 7:53:22 PM and 9/4/2023 9:28:57 AM.
IsRegressionStdDev: Marked as regression because -14.993148953512394 (T) = (0 -84.62210903742988) / Math.Sqrt((3.371249112201878 / (15)) + (6.438477036176775 / (18))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (18) - 2, .025) and -0.15636131774268336 = (73.17964354136258 - 84.62210903742988) / 73.17964354136258 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	`area-System.Reflection`, `os-windows`, `arch-x64`, `area-Host`, `untriaged`, `runtime-coreclr`, `PGO`
Milestone:	-

DrewScoggins · 2023-09-06T23:43:38Z

When looking at a report start by clicking on the small chart on the left side of the row.

This will bring up a larger version of the graph. You can zoom, pan, and select an area for zoom using the controls on the graph overlay.

Use the selection tool to focus in on the part of the graph that shows the regression.

After that it will be easy to click the point right before when the regression occurred. When you click that point you will get a fly-out that has several options. Set Baseline Set Compare and Clear Comparison. It works best to click your baseline point first and then click Set Baseline.

Once you have the baseline set you can close the fly-out by clicking on the red X.

Now, click the compare point, in this case you want to click the point that is closest to the baseline point that shows the regression.

After that click the Set Compare button.

This will bring up a link to the runtime repo that will do a comparison between the git commit hashes of the two points that were set as the baseline and compare.

DrewScoggins · 2023-09-06T23:44:55Z

Also the "test report" for at least the first couple of areas have this larger commit range: 976c2ab...9eb0266

We try and narrow down the diff range as much as we can automatically, but it is difficult to always be correct. So we bias towards giving a slightly larger range automatically, to ensure that we don't miss it entirely. But manual inspection can often give a much better range.

elinor-fung · 2023-09-07T17:08:10Z

Assigning to area-Host based on #89311 and also because that calls out PGO-related issues.

PGO was temporarily disabled as part of that PR, since it was updating linker flags. #91297 will re-enable it once we have new PGO data using the new flag.

DrewScoggins · 2023-09-07T17:12:57Z

Thanks for the reminder on that. We have had updated PGO data for the last couple weeks.

elinor-fung · 2024-02-02T18:42:32Z

#89311 was reverted:

So this should no longer be an issue from PGO being disabled.

performanceautofiler bot added arch-x64 os-windows PGO runtime-coreclr specific to the CoreCLR runtime untriaged New issue has not been triaged by the area owner labels Sep 5, 2023

cincuranet removed the untriaged New issue has not been triaged by the area owner label Sep 5, 2023

cincuranet changed the title ~~[Perf] Windows/x64: 27 Regressions on 8/29/2023 9:18:48 PM~~ Regressions in System.Reflection.Attributes Sep 5, 2023

cincuranet transferred this issue from dotnet/perf-autofiling-issues Sep 5, 2023

dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Sep 5, 2023

ghost added the untriaged New issue has not been triaged by the area owner label Sep 5, 2023

jeffschwMSFT added the area-System.Reflection label Sep 6, 2023

steveharter removed the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Sep 6, 2023

steveharter added the area-Host label Sep 6, 2023

steveharter removed the area-System.Reflection label Sep 6, 2023

agocke added this to the 9.0.0 milestone Sep 26, 2023

ghost removed the untriaged New issue has not been triaged by the area owner label Sep 26, 2023

elinor-fung closed this as completed Feb 2, 2024

github-actions bot locked and limited conversation to collaborators Mar 4, 2024

Regressions in System.Reflection.Attributes #91620

Regressions in System.Reflection.Attributes #91620

Comments

performanceautofiler bot commented Sep 5, 2023

Run Information

Regressions in System.Reflection.Attributes

Repro

Payloads

System.Reflection.Attributes.GetCustomAttributesClassHitInherit

ETL Files

Histogram

Description of detection logic

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodOverrideMiss

ETL Files

Histogram

Description of detection logic

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodOverrideMissInherit

ETL Files

Histogram

Description of detection logic

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodBaseHitInherit

ETL Files

Histogram

Description of detection logic

JIT Disasms

System.Reflection.Attributes.IsDefinedMethodOverrideHitInherit

ETL Files

Histogram

Description of detection logic

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodBaseHit

ETL Files

Histogram

Description of detection logic

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesClassMiss

ETL Files

Histogram

Description of detection logic

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesClassHit

ETL Files

Histogram

Description of detection logic

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodOverrideHit

ETL Files

Histogram

Description of detection logic

JIT Disasms

System.Reflection.Attributes.IsDefinedMethodOverrideHit

ETL Files

Histogram

Description of detection logic

JIT Disasms

Docs

Run Information

Regressions in System.Memory.ReadOnlyMemory<Byte>

Repro

Payloads

System.Memory.ReadOnlyMemory<Byte>.Pin(Size: 512)

ETL Files

Histogram

Description of detection logic

JIT Disasms

Docs

Run Information

Regressions in System.Memory.Memory<Char>

Repro

Payloads

System.Memory.Memory<Char>.Pin(Size: 512)

ETL Files

Histogram

Description of detection logic

JIT Disasms

Docs

Run Information