Add regression tests for theory matrices

[Zaharid]

It would be good to have regression tests on things like the 3 point covariance matrix, to ensure that further changes do not screw it up.

This depends on #113 to a large extend.

[voisey]

@Zaharid, could you please remind me how to go about doing this? In particular, which file do these regression tests go in?

[Zaharid]

@voisey Have a look at:

https://github.com/NNPDF/nnpdf/blob/master/validphys2/src/validphys/tests/test_regressions.py

The idea is that you define a function starting with test_ and decorate it with make_table_comp(<loader>).

The function should return a pandas frame. <loader> should be something that reconstructs the table that has been dumped as csv (setting the columns right and so on). parse_exp_mat will probably work for you as a loader.

The test works as follows: If no saved table is found in the designed path, if will be written to the path and the test will fail. if it exists, the computed version will be compared with the loaded version and the test will fail if the results are (too) different. In this way we will see it if some change in the code unintentionally changes the result, as the computed version will no longer match the stored version.

[voisey]

Thanks for this but I'm still a little unclear on how this works exactly. Is the idea that in this function I use the theory_covmat action from theorycovariance.py to return a theory_covmat dataframe, so that we can test future covmats against this?

[Zaharid]

@voisey sorry, this is another thing that is more folklore than documented. The tests are run by a program called pytest (that you can install with conda). You can see how we run it here:

nnpdf/conda-recipe/run_test.sh

Line 7 in 4efae63

pytest --pyargs validphys

if you run pytest in the git directory, you will see something like:

$ pytest 
================================================ test session starts =================================================
platform linux -- Python 3.7.0, pytest-3.8.0, py-1.6.0, pluggy-0.7.1
rootdir: /home/zah/nngit/nnpdf, inifile:
plugins: hypothesis-3.68.0
collected 18 items                                                                                                   

validphys2/src/validphys/tests/test_calcutils.py .                                                             [  5%]
validphys2/src/validphys/tests/test_fitdata.py .                                                               [ 11%]
validphys2/src/validphys/tests/test_fitveto.py ..                                                              [ 22%]
validphys2/src/validphys/tests/test_loader.py .                                                                [ 27%]
validphys2/src/validphys/tests/test_regressions.py ........                                                    [ 72%]
validphys2/src/validphys/tests/test_tableloader.py ..                                                          [ 83%]
validphys2/src/validphys/tests/test_utils.py .                                                                 [ 88%]
validphys2/src/validphys/tests/test_weights.py ..                                                              [100%]

you can be more selective with e.g. $ pytest -k test_regressions.

This causes, that among other things, the functions that start with test_ are executed (it also does complicated black magic with the assert statement to have it print very rich errors on failure).

Going back to test_regressions, yes, you need to execute manually the actions that reportengine performs in order to get to a covmat, which is itself a dataframe.

[voisey]

Thanks for the info. I'll play around with this. It seems hypothesis is a requirement for pytest but it's not installed as a dependency when one does 'conda install pytest'. Should I open an issue for this?

[Zaharid]

No, thst is fine. It's a requirement for the nnpdf tests, as specified by our recipe.

…

On Fri, 5 Oct 2018, 16:36 Cameron Voisey, ***@***.***> wrote: Thanks for the info. I'll play around with this. It seems hypothesis is a requirement for pytest but it's not installed as a dependency when one does 'conda install pytest'. Should I open an issue for this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#117 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFabUk3mdjRp-WvGM2GGPKyqaB9mUYsIks5uh3x5gaJpZM4SfCwh> .

[voisey]

Ok, good.

So far I have something like this:

from validphys import theorycovariance
@make_table_comp(parse_exp_mat)
def test_thcovmat(convolution_results):
    th, exps = convolution_results[0]
    eindex = results.experiments_index(exps)
    return theorycovariance.theory_covmat(th, eindex)

Is this going at all in the right direction? Am I correct in using convolution_results? I need to provide theory_covmat with theoryids_experiments_central_values rather than what I currently call 'th' but I'm not sure how to do this.

[Zaharid]

The start looks good.

Roughly you can replace collect with a list comprehension over the relevant items. You can get theory objects with loade.check_theoryID. Note that this is to be run on each commit so we should aim at needing two theories or so.

[scarlehoff]

There were already some tests and the rest was done in #1899