Development of the python test framework

[clanmills]

The test suite is written in python and works quite well. Here's my prioritised list of test concerns for v1.00

1. Integration of python_tests and bash_tests

Dan's test suite in tests/system_tests.py is excellent code. Leo's code in tests/bash_tests works quite well. I would like much stronger integration between system_tests and bash_tests.

2. Running the suite on a dirty machine

There is something wrong in the test suite when the build machine is "not clean". If you build/install v0.27.3, then run the test suite on 'main', you will experience errors. I think it's being caused by incorrect setting of environment strings such as PATH, LD_LIBRARY_PATH and DYLD_LIBRARY_PATH. @clanmills owns this issue: #1516

3. Documentation
I am happy to rewrite the documentation in tests/README-TESTS.md to simplify and clarify everything. I would like to document the API provided in bash_tests. I ran out of time to work on this for v0.27.4. I'll be happy to finish that work and release it in v1.00. @clanmills owns this issue and is working on it in PR: #1553.

4. Canon Lens Testing

@webmeister submitted a couple of PRs last Winter concerning Canon Lenses. I've invited Alex to join Team Exiv2 and to see his ideas developed to improve our testing of Canon lenses. #1428.

5. Testing Localisation

I added a test for NLS in v0.27.4. It doesn't work very well because the localisation files are expected to be installed on the platform. It would be good to use an environment string to locate the translation files. And there's a complication that in the build directory, the localisation is stored in files such as po/sk.po. At run time, libintl searches /usr/local/share/locale/sk/LC_MESSAGES/exiv2.mo. Let's fix this. Luis discusses this here: #549.

6. Downloading Large Files

When the project was hosted on SVN, we had a secondary server svn:://dev.exiv2.org/team/testimages on which we stored large images. This was done to reduce the size of the download necessary to build the library and sample applications. Larger images for eps and video testing were downloaded on demand. The "basic test" didn't download. Tests such as $ make test video used svn to download the image "just in time". We should adopt as similar approach as suggested in #896.

7. Deeper Testing

There are a number of sites from which we can obtain test images. For example raw.pixls.us and exiftool.org. There are numerous GitHub projects with test images for HEIF, JP2 and other formats. Let's investigate how we can use those archives to "beaf up" our test coverage. It's entirely possibly that a "full deep test" could take hours to download 10,000+ images, followed by 10+ minutes to read all the metadata. That's OK. We might only use this facility as part of the release process.

8. Lens Recognition Testing

The greatest cause of pain in Exiv2 is lens recognition. Once we have all the ExifTool test images available, it would be wonderful to run a test to identify differences between lens recognition in Exiv2 and ExifTool. Again, this may not be used with great frequency, however it is very desirable to know and understand the differences.

Summary

This is quite an extensive list of test topics which deserve attention. I don't expect everything to be completed for v1.00. However, let's work to improve our test capability.

---

Dropped or Closed

1. CTest
There has been a discussion about using CTest. I don't have an opinion about this. I think the python scripts are working well and I don't know what advantage CTest would provide. It would be useful to look at this and quickly decide to use CTest or reject it. #1068. I've discussed this with @piponazo and we're not going to pursue this.

[1div0]

I will have a look at it by the EoW.

[clanmills]

That'll be great, Peter. I have a number of things in mind and I'll edit this issue report with more details.

Today, I will finish crawling through all the open issues which we intended for 'master/0.28'. The list for v1.00 is not overwhelming. Alex is going to call a Team meeting on the weekend of 24 April or 1 May to discuss v1.00.

I've only found one issue that will have to wait for v1.1 or something and that concerns previews and thumbnails. That task is too large for our resources.

I'm delighted by the change in fortunes of Exiv2 in the past year. Thank You for working with me and contributing to this important and valuable project.