Add the option to slice out straws/tubes/layers/pixels

[nvaytet]

A request from users: while playing around with the direct beam iterations, it would be nice to be able to just run it on a single layer/tube etc to speed up development.

This PR adds the option to slice out the data at the correct point in the workflow in a way that retains identical results.

[SimonHeybrock]

I am a bit unhappy about this change. I think the previous version, which said "we compute the solid angle based on calibrated data" was clearly expressing the behavior and the intent of the pipeline. As it is now, it could be anything. If we later change the order of slicing and calibration this will be completely broken.

[nvaytet]

Would simply changing the name to SlicedCalibratedMaskedData help or is that not what you mean?

[SimonHeybrock]

Would help partially. I still wonder though if this is correctly expressing the intent. Solid angle could be computed before slicing or after, so this choice is due to cost?

[SimonHeybrock]

See above... losing information that this is calibrated.

Could the slicing be combined with masking, in terms of the graph structure, to avoid this?

[nvaytet]

I think the slicing has to be done after the masking because we need all the detector pixels to run the beam center finder, and the masking needs to be done before doing the beam center finder.

If we run the beam center finder on a single layer we will get a different offset.

[SimonHeybrock]

Is there actually a compute cost reason for changing this in the workflow? Or could we consider, e.g., returning reduced data per pixel (or in event mode)? Maybe that would take too much memory?

[nvaytet]

As in the PR description, it was because of performance. When Judith asked for this I said she could just inspect the results at the end in a single layer or tube. But she said it would be nice if it was faster when playing around with parameters.

But I am open to alternatives

[SimonHeybrock]

Hmm, isn't loading the files generally the slowest part? If we slice in the workflow it will mean the complete files will be loaded N times of we want to look at N tubes?

Edit: If this is mainly about the direct-beam iteration I suppose file-load is not the performance bottleneck. Would it be possible to load files, make a selection, and call the direct-beam workflow on that?

[nvaytet]

You can always cache parts of the workflow by setting an intermediate result on the pipeline?
E.g.

res = pipeline.compute(CalibrateMaskedData[SampleRun])
pipeline[CalibrateMaskedData[SampleRun]] = res

But I am not sure if this is dangerous and shouldn't be recommended to the users?

If this is mainly about the direct-beam iteration I suppose file-load is not the performance bottleneck.

Note that we are calling the pipeline inside a loop (and it fact we are computing it twice inside the loop: once for full wavelength range and once for wavelength bands). I think that means we are loading all the files at every loop.
We should probably have better caching inside the iteration loop?

Presumably we can cache the solid angle, the numerator, the transmission fraction and monitors, and only evolve the direct beam?

[nvaytet]

I'm guessing a similar thing can be done to speed up the beam_center_finder_from_iofq?

[SimonHeybrock]

I am just trying that, but it only helps a bit (3 seconds per iteration). There is another source of bad performance in the wavelength-bands pipeline, I am trying to track that down now.

In other words: We should check if this scientist request is actually an XY problem (this solution was suggested because performance is bad).

[nvaytet]

My guess is this the merge_spectra function is the bottleneck: https://github.com/scipp/esssans/blob/main/src/esssans/i_of_q.py#L238

[SimonHeybrock]

Yes, you are correct. But it is not the event-mode one, but just the histogram mode. I have some ideas how to improve, will try to look into that tomorrow.

[SimonHeybrock]

I think this has been superseded, please reopen if I am wrong.