Test failure should return correct "fail" value

[johscheuer]

I followed the steps described here: https://www.openpolicyagent.org/docs/how-do-i-test-policies.html#test-results and I wondered about the "fail": false actually I would expect that when a test case fails the value ist set to: "fail": true?

{
  "location": {
    "file": "pass_fail_error_test.rego",
    "row": 4,
    "col": 1
  },
  "package": "data.example",
  "name": "test_ok",
  "duration": 1291364
}
{
  "location": {
    "file": "pass_fail_error_test.rego",
    "row": 9,
    "col": 1
  },
  "package": "data.example",
  "name": "test_failure",
  "fail": false,
  "duration": 597117
}
{
  "location": {
    "file": "pass_fail_error_test.rego",
    "row": 14,
    "col": 1
  },
  "package": "data.example",
  "name": "test_error",
  "error": {
    "code": "eval_internal_error",
    "message": "div: divide by zero",
    "location": {
      "file": "pass_fail_error_test.rego",
      "row": 15,
      "col": 5
    }
  },
  "duration": 527246
}

[tsandall]

Yes, this is confusing. The current implementation sets the fail field as follows:

• If the test rule is undefined, the field is set to false
• If the test rule generates a value that is not true, the field is set to the value

We should just set the test fail field to true if the test failed in all cases. I'll submit a PR today.