[libc++] Adjust some of the [rand.dist] critical values that are too strict #88669
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for submitting a Pull Request (PR) to the LLVM Project! This PR will be automatically labeled and the relevant teams will be If you wish to, you can add reviewers by using the "Reviewers" section on this page. If this is not working for you, it is probably because you do not have write If you have received no comments on your PR for a week, you can request a review If you have further questions, they may be answered by the LLVM GitHub User Guide. You can also ask questions in a comment on this PR, on the LLVM Discord or on the forums. |
llvmbot
added
the
libc++
|
@llvm/pr-subscribers-libcxx Author: Matt Stephanson (MattStephanson) ChangesIn the MSVC STL, a few of the eval.pass tests under rand.dist are marked XFAIL, but the failures mostly haven't been looked into in detail. I was investigating P0952R2, and it caused many more tests to fail, so I decided to look at the tolerances in these tests systematically. For context, most of these tests are comparing the sample mean, variance, skew, and kurtosis to the analytic values, although a few tests are using the Kolmogorov–Smirnov test. The tests I ran and the results are described here. With the new critical values proposed in this PR, there aren't any tolerance-related failures using the MSVC STL and the new In the rand.dist.sample.plinear test, I also fixed a narrowing warning-treated-as-error that was causing a test failure for MSVC. Patch is 51.38 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/88669.diff 28 Files Affected:
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bernoulli/eval.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bernoulli/eval.pass.cpp
index 9a7af92931dd3e..568bf34f1ea437 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bernoulli/eval.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bernoulli/eval.pass.cpp
@@ -63,8 +63,8 @@ int main(int, char**)
double x_kurtosis = (6 * sqr(d.p()) - 6 * d.p() + 1)/x_var;
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
- assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.02);
+ assert(std::abs((skew - x_skew) / x_skew) < 0.02);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.05);
}
{
typedef std::bernoulli_distribution D;
@@ -99,8 +99,8 @@ int main(int, char**)
double x_kurtosis = (6 * sqr(d.p()) - 6 * d.p() + 1)/x_var;
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
- assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.02);
+ assert(std::abs((skew - x_skew) / x_skew) < 0.02);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.05);
}
return 0;
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bernoulli/eval_param.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bernoulli/eval_param.pass.cpp
index 5584a9d1644896..dfaa9f1c89f991 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bernoulli/eval_param.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bernoulli/eval_param.pass.cpp
@@ -65,8 +65,8 @@ int main(int, char**)
double x_kurtosis = (6 * sqr(p.p()) - 6 * p.p() + 1)/x_var;
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
- assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.02);
+ assert(std::abs((skew - x_skew) / x_skew) < 0.02);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.05);
}
{
typedef std::bernoulli_distribution D;
@@ -103,8 +103,8 @@ int main(int, char**)
double x_kurtosis = (6 * sqr(p.p()) - 6 * p.p() + 1)/x_var;
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
- assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.02);
+ assert(std::abs((skew - x_skew) / x_skew) < 0.02);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.05);
}
return 0;
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval.PR44847.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval.PR44847.pass.cpp
index 572a59b7a1926a..549f3cee4d4a45 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval.PR44847.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval.PR44847.pass.cpp
@@ -163,7 +163,7 @@ int main(int, char**) {
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.04);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.06);
return 0;
}
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval.pass.cpp
index dbdd0972411940..1670df40769101 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval.pass.cpp
@@ -68,7 +68,7 @@ void test1() {
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.04);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.08);
}
template <class T>
@@ -109,8 +109,8 @@ void test2() {
double x_kurtosis = (1-6*d.p()*(1-d.p())) / x_var;
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
- assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.01);
+ assert(std::abs((skew - x_skew) / x_skew) < 0.02);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.08);
}
template <class T>
@@ -151,8 +151,8 @@ void test3() {
double x_kurtosis = (1-6*d.p()*(1-d.p())) / x_var;
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
- assert(std::abs((skew - x_skew) / x_skew) < 0.03);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.3);
+ assert(std::abs((skew - x_skew) / x_skew) < 0.07);
+ assert(std::abs(kurtosis - x_kurtosis) < 2.0);
}
template <class T>
@@ -292,7 +292,7 @@ void test6() {
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
assert(std::abs(skew - x_skew) < 0.02);
- assert(std::abs(kurtosis - x_kurtosis) < 0.01);
+ assert(std::abs(kurtosis - x_kurtosis) < 0.03);
}
template <class T>
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval_param.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval_param.pass.cpp
index 78d6aedde73a5d..adbcb78d10f4f2 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval_param.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval_param.pass.cpp
@@ -72,7 +72,7 @@ int main(int, char**)
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.04);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.08);
}
{
typedef std::binomial_distribution<> D;
@@ -113,8 +113,8 @@ int main(int, char**)
double x_kurtosis = (1-6*p.p()*(1-p.p())) / x_var;
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
- assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.01);
+ assert(std::abs((skew - x_skew) / x_skew) < 0.02);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.08);
}
{
typedef std::binomial_distribution<> D;
@@ -155,8 +155,8 @@ int main(int, char**)
double x_kurtosis = (1-6*p.p()*(1-p.p())) / x_var;
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
- assert(std::abs((skew - x_skew) / x_skew) < 0.04);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.3);
+ assert(std::abs((skew - x_skew) / x_skew) < 0.07);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 2.0);
}
return 0;
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.geo/eval.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.geo/eval.pass.cpp
index 440334ed3488ae..0cdb7fa6312caf 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.geo/eval.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.geo/eval.pass.cpp
@@ -77,7 +77,7 @@ void test1() {
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.01);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.03);
}
template <class T>
@@ -161,7 +161,7 @@ void test3() {
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.02);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.03);
}
template <class T>
@@ -203,7 +203,7 @@ void test4() {
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.02);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.03);
}
template <class T>
@@ -245,7 +245,7 @@ void test5() {
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.02);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.03);
}
template <class T>
@@ -284,8 +284,8 @@ void test6() {
double x_var = x_mean / d.p();
double x_skew = (2 - d.p()) / std::sqrt((1 - d.p()));
double x_kurtosis = 6 + sqr(d.p()) / (1 - d.p());
- assert(std::abs((mean - x_mean) / x_mean) < 0.01);
- assert(std::abs((var - x_var) / x_var) < 0.01);
+ assert(std::abs((mean - x_mean) / x_mean) < 0.02);
+ assert(std::abs((var - x_var) / x_var) < 0.02);
assert(std::abs((skew - x_skew) / x_skew) < 0.01);
assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.02);
}
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.geo/eval_param.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.geo/eval_param.pass.cpp
index 16cb7fb0a45f98..16a5bd4c7a33e7 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.geo/eval_param.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.geo/eval_param.pass.cpp
@@ -72,7 +72,7 @@ int main(int, char**)
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.01);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.03);
}
{
typedef std::geometric_distribution<> D;
@@ -156,7 +156,7 @@ int main(int, char**)
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.02);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.03);
}
return 0;
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.negbin/eval.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.negbin/eval.pass.cpp
index d0f6fbf0a1203f..be08361fc27e21 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.negbin/eval.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.negbin/eval.pass.cpp
@@ -74,7 +74,7 @@ void test1() {
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.02);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.03);
}
template <class T>
@@ -115,8 +115,8 @@ void test2() {
double x_kurtosis = 6. / d.k() + sqr(d.p()) / (d.k() * (1 - d.p()));
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
- assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.01);
+ assert(std::abs((skew - x_skew) / x_skew) < 0.02);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.1);
}
template <class T>
@@ -157,8 +157,8 @@ void test3() {
double x_kurtosis = 6. / d.k() + sqr(d.p()) / (d.k() * (1 - d.p()));
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
- assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.03);
+ assert(std::abs((skew - x_skew) / x_skew) < 0.02);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.08);
}
template <class T>
@@ -243,8 +243,8 @@ void test5() {
double x_kurtosis = 6. / d.k() + sqr(d.p()) / (d.k() * (1 - d.p()));
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
- assert(std::abs((skew - x_skew) / x_skew) < 0.04);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.05);
+ assert(std::abs((skew - x_skew) / x_skew) < 0.02);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.3);
}
template <class T>
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.negbin/eval_param.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.negbin/eval_param.pass.cpp
index 0b03982a737e57..26bc83382f6850 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.negbin/eval_param.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.negbin/eval_param.pass.cpp
@@ -72,7 +72,7 @@ int main(int, char**)
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.01);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.03);
}
{
typedef std::negative_binomial_distribution<> D;
@@ -113,8 +113,8 @@ int main(int, char**)
double x_kurtosis = 6. / p.k() + sqr(p.p()) / (p.k() * (1 - p.p()));
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
- assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.01);
+ assert(std::abs((skew - x_skew) / x_skew) < 0.02);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.1);
}
{
typedef std::negative_binomial_distribution<> D;
@@ -155,8 +155,8 @@ int main(int, char**)
double x_kurtosis = 6. / p.k() + sqr(p.p()) / (p.k() * (1 - p.p()));
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
- assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.03);
+ assert(std::abs((skew - x_skew) / x_skew) < 0.02);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.08);
}
return 0;
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.cauchy/eval.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.cauchy/eval.pass.cpp
index abc0cc531a117e..98f32d00f88381 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.cauchy/eval.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.cauchy/eval.pass.cpp
@@ -45,7 +45,7 @@ int main(int, char**)
u.push_back(d(g));
std::sort(u.begin(), u.end());
for (int i = 0; i < N; ++i)
- assert(std::abs(f(u[i], a, b) - double(i)/N) < .001);
+ assert(std::abs(f(u[i], a, b) - double(i)/N) < .0013);
}
{
typedef std::cauchy_distribution<> D;
@@ -60,7 +60,7 @@ int main(int, char**)
u.push_back(d(g));
std::sort(u.begin(), u.end());
for (int i = 0; i < N; ++i)
- assert(std::abs(f(u[i], a, b) - double(i)/N) < .001);
+ assert(std::abs(f(u[i], a, b) - double(i)/N) < .0013);
}
{
typedef std::cauchy_distribution<> D;
@@ -75,7 +75,7 @@ int main(int, char**)
u.push_back(d(g));
std::sort(u.begin(), u.end());
for (int i = 0; i < N; ++i)
- assert(std::abs(f(u[i], a, b) - double(i)/N) < .001);
+ assert(std::abs(f(u[i], a, b) - double(i)/N) < .0013);
}
return 0;
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.chisq/eval.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.chisq/eval.pass.cpp
index 2a8dfd31aa08eb..559034b2a0ec10 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.chisq/eval.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.chisq/eval.pass.cpp
@@ -70,7 +70,7 @@ int main(int, char**)
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.01);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.04);
}
{
typedef std::chi_squared_distribution<> D;
@@ -109,7 +109,7 @@ int main(int, char**)
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.01);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.03);
}
{
typedef std::chi_squared_distribution<> D;
@@ -148,7 +148,7 @@ int main(int, char**)
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.01);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.03);
}
return 0;
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.chisq/eval_param.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.chisq/eval_param.pass.cpp
index 52864739c9b3cc..74454f29626724 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.chisq/eval_param.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.chisq/eval_param.pass.cpp
@@ -72,7 +72,7 @@ int main(int, char**)
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.01);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.03);
}
{
typedef std::chi_squared_distribution<> D;
@@ -113,7 +113,7 @@ int main(int, char**)
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.01);
+ assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.03);
}
{
typedef std::chi_squared_distribution<> D;
@@ -154,7 +154,7 @@ int main(int, char**)
assert(std::abs((mean - x_mean) / x_mean) < 0.01);
assert(std::abs((var - x_var) / x_var) < 0.01);
assert(std::abs((skew - x_skew) / x_skew) < 0.01);
- assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.01);
+ assert(std::abs((kurtosis - x_kurtosis) / x...
[truncated]
|
mordante
reviewed
Apr 15, 2024
There was a problem hiding this comment.
Thanks for working on this. Can you update the commit message how you determined the new values; I guess empirical, but it would be good to know for certain.
I noticed the same when I worked on P0952R2. (That patch is still WIP and I got lower on my priority list.)
| @@ -151,8 +151,8 @@ void test3() { | |||
| double x_kurtosis = (1-6*d.p()*(1-d.p())) / x_var; | |||
| assert(std::abs((mean - x_mean) / x_mean) < 0.01); | |||
| assert(std::abs((var - x_var) / x_var) < 0.01); | |||
| assert(std::abs((skew - x_skew) / x_skew) < 0.03); | |||
| assert(std::abs((kurtosis - x_kurtosis) / x_kurtosis) < 0.3); | |||
There was a problem hiding this comment.
Oops, I was debating changing it to an absolute test because the expected kurtosis is close to zero (-0.017). 2.0 should be the relative bound.
For reference, if you approximate the binomial distribution as normal, you would expect the sample kurtosis to have a standard deviation of
There was a problem hiding this comment.
Can you add something along these lines in the commit message. That give future readers more information why these changes are valid.
There was a problem hiding this comment.
@MattStephanson can you update the commit message? Then the patch is good to land. Thanks.
There was a problem hiding this comment.
Sorry if I wasn't clear.
When we land this patch the commits will be squashed and merged into one commit. Then we only have the commit message of the review. So what I meant was to edit the top message with the additional information.
There was a problem hiding this comment.
Oh, okay. I think I have it now?
There was a problem hiding this comment.
Yes great! Thanks. I'll land the patch.
libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.lognormal/eval.pass.cpp
Show resolved
Hide resolved
|
Is in not better to have smaller values on this? If so, we may want to keep some testing to ensure our implementation doesn't slip? |
Not exactly. The statistics of a sample (mean, variance, etc.) are themselves random variables with some distribution. So except in degenerate cases, there's always some chance that a correct generator will give an out-of-tolerance result. There's also a tradeoff because if you want a smaller tolerance, you need more samples to justify rejecting the null hypothesis that the sample comes from the desired distribution. Even these larger tolerances that I'm proposing only approximate a 90% confidence interval, so if the underlying U[0,1) engine was truly random, you would still expect these tests to fail 10% of the time even if the distribution generator was completely correct. |
MattStephanson
force-pushed
the
rand-dist-tols
branch
from
82c8a54
to
53830bf
Compare
You can test this locally with the following command:git-clang-format --diff 5927492e8e68c1ee5a0d84cf6c402a900aac4a3c 7f117556001c9d205af692a5ef4d6fa0e432a0e1 -- libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bernoulli/eval.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bernoulli/eval_param.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval.PR44847.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval_param.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.geo/eval.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.geo/eval_param.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.negbin/eval.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.negbin/eval_param.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.cauchy/eval.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.chisq/eval.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.chisq/eval_param.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.lognormal/eval.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.lognormal/eval_param.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.t/eval.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.norm/rand.dist.norm.t/eval_param.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.pois/rand.dist.pois.exp/eval.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.pois/rand.dist.pois.exp/eval_param.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.pois/rand.dist.pois.extreme/eval.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.pois/rand.dist.pois.extreme/eval_param.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.pois/rand.dist.pois.gamma/eval.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.pois/rand.dist.pois.gamma/eval_param.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.pois/rand.dist.pois.poisson/eval.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.pois/rand.dist.pois.poisson/eval_param.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.samp/rand.dist.samp.discrete/eval.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.samp/rand.dist.samp.pconst/eval.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.samp/rand.dist.samp.plinear/eval.pass.cpp libcxx/test/std/numerics/rand/rand.dist/rand.dist.samp/rand.dist.samp.plinear/eval_param.pass.cppView the diff from clang-format here.diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.samp/rand.dist.samp.plinear/eval.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.samp/rand.dist.samp.plinear/eval.pass.cpp
index 8aad0b8e4a..06f57dc69d 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.samp/rand.dist.samp.plinear/eval.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.samp/rand.dist.samp.plinear/eval.pass.cpp
@@ -85,7 +85,7 @@ test1()
bk = b[k];
c = (b[k + 1] * p[k] - b[k] * p[k + 1]) / (b[k + 1] - b[k]);
kp = k;
- }
+ }
assert(std::abs(f(u[i], a, m, bk, c) - double(i) / N) < .0013);
}
}
@@ -136,7 +136,7 @@ test2()
bk = b[k];
c = (b[k + 1] * p[k] - b[k] * p[k + 1]) / (b[k + 1] - b[k]);
kp = k;
- }
+ }
assert(std::abs(f(u[i], a, m, bk, c) - double(i) / N) < .0013);
}
}
@@ -187,7 +187,7 @@ test3()
bk = b[k];
c = (b[k + 1] * p[k] - b[k] * p[k + 1]) / (b[k + 1] - b[k]);
kp = k;
- }
+ }
assert(std::abs(f(u[i], a, m, bk, c) - double(i) / N) < .0013);
}
}
@@ -239,7 +239,7 @@ test4()
bk = b[k];
c = (b[k + 1] * p[k] - b[k] * p[k + 1]) / (b[k + 1] - b[k]);
kp = k;
- }
+ }
assert(std::abs(f(u[i], a, m, bk, c) - double(i) / N) < .0013);
}
}
@@ -292,7 +292,7 @@ test5()
bk = b[k];
c = (b[k + 1] * p[k] - b[k] * p[k + 1]) / (b[k + 1] - b[k]);
kp = k;
- }
+ }
assert(std::abs(f(u[i], a, m, bk, c) - double(i) / N) < .0013);
}
}
@@ -343,7 +343,7 @@ test6()
bk = b[k];
c = (b[k + 1] * p[k] - b[k] * p[k + 1]) / (b[k + 1] - b[k]);
kp = k;
- }
+ }
assert(std::abs(f(u[i], a, m, bk, c) - double(i) / N) < .0013);
}
}
diff --git a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.samp/rand.dist.samp.plinear/eval_param.pass.cpp b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.samp/rand.dist.samp.plinear/eval_param.pass.cpp
index 4601c36019..b847678aae 100644
--- a/libcxx/test/std/numerics/rand/rand.dist/rand.dist.samp/rand.dist.samp.plinear/eval_param.pass.cpp
+++ b/libcxx/test/std/numerics/rand/rand.dist/rand.dist.samp/rand.dist.samp.plinear/eval_param.pass.cpp
@@ -87,7 +87,7 @@ int main(int, char**)
bk = b[k];
c = (b[k + 1] * p[k] - b[k] * p[k + 1]) / (b[k + 1] - b[k]);
kp = k;
- }
+ }
assert(std::abs(f(u[i], a, m, bk, c) - double(i) / N) < .0013);
}
}
|
mordante
approved these changes
Apr 20, 2024
There was a problem hiding this comment.
Is in not better to have smaller values on this?
I kind of agree, however these are statistical tests and there is no motivation why these tolerances were picked. For example 45a9997
I noticed the tolerances not to work when implementing a new paper.
If so, we may want to keep some testing to ensure our implementation doesn't slip? But otherwise, this change seems reasonable.
I think we could add a test where we test some specific expected values. To me this is outside the scope of this patch; unless @MattStephanson likes to do it.
I'm happy with this patch after addressing my last comment. Please give @EricWF a few days to respond before landing this patch.
- Most critical values are determined empirically by running each test 51 times with a different PRNG seed and finding the smallest symmetric interval around the median that contains 90% of the sample means, variances, etc. - For the Kolmogorov-Smirnov tests, the alpha=0.1 critical value for large N is 1.224/sqrt(N). - For normally distributed variates, the sample kurtosis is distributed as Normal(0, 24/N). For N=1e5, this gives a 90% confidence interval of 0+/-0.0255. For Binomial(40, 0.25), which is approximately normal, the kurtosis is -0.0167, so the relative 90% CI is large, on the order of 0.0255/0.0167 = 153%. In most cases the distribution of the sample kurtosis isn't known analytically, but similarly large relative tolerances can be expected if the kurtosis is near zero.
MattStephanson
force-pushed
the
rand-dist-tols
branch
from
53830bf
to
788fc37
Compare
|
I noticed your email is hidden, could you make that public? |
Okay. |
|
@MattStephanson Congratulations on having your first Pull Request (PR) merged into the LLVM Project! Your changes will be combined with recent changes from other authors, then tested Please check whether problems have been caused by your change specifically, as How to do this, and the rest of the post-merge process, is covered in detail here. If your change does cause a problem, it may be reverted, or you can revert it yourself. If you don't get any reports, no action is required from you. Your changes are working as expected, well done! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adjust some of the [rand.dist] critical values that are too strict
Most critical values are determined empirically by running each test 51
times with a different PRNG seed and finding the smallest symmetric interval
around the median that contains 90% of the sample means, variances, etc.
For the Kolmogorov-Smirnov tests, the alpha=0.1 critical value for large N
is 1.224/sqrt(N).
For normally distributed variates, the sample kurtosis is distributed as
Normal(0, 24/N). For N=1e5, this gives a 90% confidence interval of
0+/-0.0255. For Binomial(40, 0.25), which is approximately normal, the
kurtosis is -0.0167, so the relative 90% CI is large, on the order of
0.0255/0.0167 = 153%. In most cases the distribution of the sample kurtosis
isn't known analytically, but similarly large relative tolerances can be
expected if the kurtosis is near zero.