Automatic testing

[AerysL]

Given that we have a number of tests which we know the expected outcome, I was thinking that it might be advantageous to setup some sort of automatic testing, so it is known if a change incorrectly changes the results of the tests. We could integrate such testing into the makefile, so feedback would be near realtime (if the tests are done on relatively small problems).

If such tests are deemed worthy to invest time into, I think we could do one of two directions:

1. Run our own tests without a framework. This would be very similar to our current setup with I think two main differences:

   • Smaller problems would be selected, this would speed up the testing so that it could be done on every make with only a few additional seconds.
   • Check the expected vs actual output, making sure they are within some sort of tolerance.

2. Adopt some sort of framework. This would have the advantage of being more robust and expandable, however would likely require more investment into testing infrastructure in addition to learning the framework and would likely add more dependencies to the EVSL project.

[liruipeng]

Maybe we can do regression test for EVSL. We can write some shell scripts to do so, and run it automatically every day/week, or manually when one does a push. We have scripts for Hypre that run every night. If needed, we can make changes from them.