Support very large test results

[ben-manes]

My project produces millions of test executions over a 2 hour build. I was hoping to use this action for a summary of the results, which would be a lightweight as just the counts at the suite level. Unfortunately the action failed with an out of memory error (exit code 137). Can the results be processed in a streaming fashion rather than read fully into memory, which avoids having a limit?

A possible workaround that I might try is to delete all the test methods prior to processing, e.g.

xml ed -L -d "/testsuite/*" test.xml

[EnricoMi]

This are 6.31 GB of test results, wow! Let me see what I can do ...

[ben-manes]

haha, yes. I tried it on a local download and the cli tool crashed as it's not streaming either.

$ find . -type f -exec xml ed -L -d "/testsuite/*" {} \;
./TEST-com.github.benmanes.caffeine.cache.ExpirationTest.xml:124476.115: internal error: Huge input lookup
ache.testing.RemovalListeners$RejectingRemovalListener.onRemoval(RemovalListener

[EnricoMi]

Since you are running with check_run_annotations: none, you can safely remove all those <testcase> nodes under the root <testsuite>. The action takes the statistics from the <testsuite> and only needs the individual <testcase> nodes for annotations.

[ben-manes]

yes, exactly. I need to find the right incantation of a shell command to do that. Something like this should work, though not quite right yet: find . -name "*.xml" -exec sh -c 'grep testsuite {} > {}' \;

[ben-manes]

I think will do it as the preprocessing step.

find . -type f -name "*.xml" -exec sh -c 'grep testsuite {} > {}.out && mv {}.out {}' \;

closing and off to give it a try.

[EnricoMi]

Thanks for reporting, I am curious about any workaround that works for you.

[ben-manes]

kicked off a job, but it will take ~2 hours for the full run.

[EnricoMi]

Can you test the action from this branch:

uses: EnricoMi/publish-unit-test-result-action/composite@branch-xml-ignore-testcases

That should not require you modifying the XML files.

I may make this available through a new option. This branch is just a PoC.

[ben-manes]

Sure, here's that job

[EnricoMi]

The first job looks ok, the parsing errors will go away when you point files to test result xml files only:

 files: '**/*.xml'

The reported 0 tests will go away when you rerun the second job (assuming it has the same output as the first job - hasn't finished yet). There is a fix in the branch that did not get picked up by the second job as I pushed it after your workflow started.

[ben-manes]

I kicked off another job with that fix. 🙁

[EnricoMi]

And, as a side note: with matrix strategy, you have to give the jobs individual check_name (include ${{ matrix.java }}) so that you get two results, otherwise the two results will overwrite each other.

[ben-manes]

right, I intend to use an artifact and separate workflow_run job, but that only runs if on the master branch. So I inlined these for debugging.

[ben-manes]

Looks like it attached to the examples workflow, oddly enough, but still got a nice summary.

[EnricoMi]

Your browser is running on macOS, right? On Ubuntu, this renders nicer:

Amazing how different Github looks across OS.

[ben-manes]

Yep, that looks much nicer. Odd as probably both on chrome?

[EnricoMi]

Here is Ubuntu Chrome:

[EnricoMi]

OK, so the branch works equally well, no need to preprocess the files then.

[EnricoMi]

This has been released as v1.31, available via v1.

[ben-manes]

Thanks! I pushed the switch over to this new config option.

[EnricoMi]

Thanks for testing, I am watching that job.

[EnricoMi]

Yay, this worked:

Next run, the lowest row will disappear, only leaving the test row.

8.5m tests, this is definitively top 1 in this action's high score.

[ben-manes]

This is great! I had to estimate counts before since it is so huge. 🙂

For this project the complexity is very high and beyond what I can mentally track. I decided to brute force testing was a good safety net, even if sometimes excessive. The ~2000 tests methods are parameterized to run for every valid configuration (a cartesian product of the specification constraints), which catches a lot of simple mistakes during development that happen only in some cases. One of those settings is the backing implementation, where Google's Guava is a reference to assert similar behavior. I also ported unit test suite from the jdk, eclipse, apache, and google to fill in gaps due to lack of imagination. I then throw in many static analyzers, etc in hopes of catching whatever remains. Adding features or tests for bug fixes inches those numbers further. All of that is to say that the project is too complex for my meager brain so I copped out and burn cpu cycles instead 😄