"Pending" tests of "status quo"

[som-snytt]

It's useful to have a test that documents the status quo even if the status quo is undesirable.

For example,
#23276

I chose a random commit to see how tests/pending is used:
3125665 deletes a pos test that had a comment that it should fail and see tests/pending/neg/i5690.scala, which was moved to neg.

I don't think pending is useful if it is used to park tests without testing them. (That is, it is useful as a staging area only.)

I should be alerted if my commit changes behavior in any way, including dubious behavior. (That might be a progression or a regression.)

This question (about testing the status quo) came up again because of
tests/untried/neg/t4989.scala
which generates more errors than Scala 2, but possibly not enough errors. If it were a tried test, even without resolving any behavioral issues, at least the test would have history to demonstrate whether the behavior has changed. (History is easier than bisecting.)

Is there any argument against a test category for status-quo, or perhaps dubious behavior, to signal that a test has problematic behavior? Or an annotation that a test result is known to be dubious, which would have more force than a TODO comment.

My own experience is that particular test categories are useful only for managing how tests are run, rather than characterizing the tests. For example, tests/warn is more useful than pos or neg because it permits zero or more warnings, and not because it counts "warnings" instead of "errors". So I'm not convinced that a separate test kind is desirable, only that such tests are run somehow.

[odersky]

I agree, adding a status-quo is a useful addition. The convention could be that a test that should fail but that compiles should have // should error comments. Likewise a test that fails but should be OK should have // error but should be ok comments.