Integration Testing

Producing Trusted reference Hashes

Optionally, it may or may not be a good idea to first manually inspect the samples (via e.g. darktable), make a note which are seemingly currently decoded correctly, and which are not.

For best results the Trusted Hashes should be produced in most mundane environment - stable mainstream hardware (little-endian, x86; no overclocking), stable software stack, and most importantly a trusted compiler. You also shouldn’t use -Oomg-optimize -fmoar-performance compilation flags for this.

Warning

Trusted baseline hashes are a the very foundation for any further integration testing. It is always important to have good, stable foundation. It will not be productive if those hashes are produced incorrectly, be it either because the hardware is faulty (RAM/disk bit flips), or the library was miscompiled.

Other than that, generating said hashes is pretty trivial.

Specifying location of Reference Sample Archive

In order to make use of build system integration of integration testing, we must first tell it where the sample set is located, for example:

$ cmake -DRAWSPEED_ENABLE_SAMPLE_BASED_TESTING:BOOL=ON \
        -DRAWSPEED_REFERENCE_SAMPLE_ARCHIVE:PATH="~/raw-camera-samples/raw.pixls.us-unique/" \
        <path to rawspeed repo checkout>

Note

The location of the samples must be writable if you intend to produce hashes.

Other required CMake flags

We also need to build the code that is actually responsible for these integration tests:

$ cmake -DBUILD_TOOLS:BOOL=ON \
        <path to rawspeed repo checkout>

Producing hashes

After that is done, we can finally create the hashes, and for that there is a rstest-create build target:

$ cmake --build . -- rstest-clean # get rid of any pre-existing hashes, just in case.
[1/1] Running utility command for rstest-clean
$ cmake --build . -- rstest-create
<maybe actually building library's sources and rstest if you didn't build it yet>
[0/1] Running utility command for rstest-create
<full path to a sample>: starting decoding ...
<full path to a sample>:  <> MB / <> ms

Total decoding time: <>s

All good, all hashes created!

And that’s it, we’ve got the hashes! They were placed next to the samples in the archive, with .hash suffix appended. Maybe you want to use some kind of layered file system (overlayfs e.g.) to separate those from the actual samples, up to you.

Performing Integration Testing

Important

Do ensure that the library is actually re-compiled with the changes you want to test. To err on the safe side, sometimes it is useful to remove the entire build directory and make a fresh build!

After you have performed the changes you wanted to - modified the library, or changed hardware/software/compiler/compiler flags - and you want to validate that those changes did not cause any regressions in the sample set, it is time to actually make use of the Trusted Reference Hashes that we have created previously.

For that, there is a rstest-test build target. If everything is good you may see:

$ cmake --build . -- rstest-test
[0/1] Running utility command for rstest-test
<full path to a sample>: starting decoding ...
<full path to a sample>:  <> MB / <> ms
Total decoding time: <>s

All good, no tests failed!

Or, if there are issues, you may see:

$ cmake --build . -- rstest-test
[0/1] Running utility command for rstest-test
<full path to a sample>: starting decoding ...
<full path to a sample>:  <> MB / <> ms
<full path to a sample> failed: hash/metadata mismatch
Total decoding time: <>s

WARNING: the following <> tests have failed:
<full path to a sample> failed: hash/metadata mismatch
See rstest.log for details.
<...>
ninja: build stopped: subcommand failed.

Unless the process crashed, it should have created <full path to a sample>.hash.failed, and outputted the diff between the existing <full path to a sample>.hash Trusted Hash and the actual result <full path to a sample>.hash.failed into rstest.log file in root of the build dir.