Implement capture sharpening inside demosaic module

[jenshannoschwalm]

This has been discussed many times here and on pixls.us so here we go. As we have D&S giving good results i was somewhat reluctant to work on this and do a pr but as CS works so good on RT and after a lot of testing (and using this on my personal images for a while with 3 auto applied presets for my fixed-lens camera - one for high-iso with CS turned off and two presets for wide open and "normal") i am pretty confident it's worth it and would be a significant improvement.
(It took quite a while to spot a very bad bug in GPU code leading to system instability and crashes so unfortunately not for 5.2)

Please test in depth and let's decide if this is for master. It's a lot of processing - so not a superfast module.

---

Capture sharpening has been implemented to work inside the demosaic module so it's raw only.
Credits to: Ingo Weyrich (heckflosse67@gmx.de), he implemented the original algorithm for rawtherapee, this implementation is based on his work, especially the convolution kernels.
CPU and OpenCL code paths are both available.
Demosaic module gets more parameters so there is a version bump, one float has been reserved.

The "mini manual" (regularly updated to latest commit)

Capture sharpening (CS) tries to recover details lost due to in-camera blurring, which can be caused by diffraction, the anti-aliasing filter or other sources of gaussian-type blur. Prerequisites are

• good white balance parameters (same requirement as for highlights reconstruction or demosaic)
• no chromatic aberration, you might want to add the "raw chromatic aberration" module
• low noise as noise will be amplified by CS

Controls

1. capture sharpen
   switches CS on if above zero and defines the strength of overall effect.
   CS works in an iterative process, this defines the number of iterations, mostly a setting of 10 will be enough.
   As downscaling after demosaicing reduces the visibility of CS we reduce iterations in such cases for performance.
2. radius
   defines the basic convolution gaussian sigma.
   This should not be set by "creative means" but to the blurring radius of the optical system and sensor, too large values will lead to artifacts like halos.
   Calculating a correct radius is provided internally. This will be done either if you
   a) click on the button besides the slider
   b) activate capture sharpen the first time after resetting to demosaic defaults or developing old edits.
3. contrast threshold
   As sensor noise will be amplified by CS we take some care about this by a per pixel variance analysis and use a logistic function with this threshold to avoid CS in noisy areas. The default is good for low iso images.
4. corner boost
   Increase the radius in image corners. We assume a circle of 1/2 of image size to be "sharp" (only use main radius), locations outside this center circle get an increased convolution radius.

[HansBull]

A lovely enhancement, and reasonably fast even on my aging i7-2670QM!

[da-phil]

Nice effort, I like it 👍
However for some images I noticed a color shift the more iterations I choose for CS. Can you also reproduce it? Try very colorful images and maybe bump up the saturation a little bit in order to make the difference more visible.
In my case I also overlayed the non-CS with the CS image and used the layer mode "difference" to also see the color cast quantitatively.

[jenshannoschwalm]

However for some images I noticed a color shift the more iterations I choose for CS. Can you also reproduce it?

I couldn't yet, could you somehow share such raw files? (And possibly the xmp too? It might be related to white balance coeffs. A pre-requisite would be a correct WB)

As CS is an iterative process any error would be amplified by more iterations.

In my case I also overlayed ...

I don't understand your "reproducer procedure" here.

[TurboGit]

Just did a quick test, for a consistent UI I would propose to make the sliders non sensitive when sharpen is 0 but not to hide them.

[jenshannoschwalm]

I can do so of course. EDIT latest commit changes this but for my taste it's looking worse. We also have the other optinoal sliders in this module only visible if "relevant"

[da-phil]

In my case I also overlayed ...

I don't understand your "reproducer procedure" here.

What I meant was exporting an image without CS and one with CS turned on with at least 15 iterations, then compare the images overlayed in separate layers in gimp, using the layer mode "difference".
For some images I also noticed that the image with CS turned on appear slightly brighter.

I'll try to find images which I can share for you to reproduce.

[jenshannoschwalm]

What I meant was exporting an image without CS and one with CS turned on with at least 15 iterations, then compare the images overlayed in separate layers in gimp, using the layer mode "difference".

You are aware of the --dump-pipe module option?

[jenshannoschwalm]

Squashed and force-pushed an updated version

1. convolution kernels support sigma-steps of 0.01 instead of 0.02
2. slightly improved CPU convolution performance
3. the per-pixel variance anylysis is done in a circle with a diameter of 5 instead of 3 for improved stability in noisy regions and at sharp luminance transitions.

[da-phil]

What I meant was exporting an image without CS and one with CS turned on with at least 15 iterations, then compare the images overlayed in separate layers in gimp, using the layer mode "difference".

You are aware of the --dump-pipe module option?

Nope, never used it, what is it good for?

[jenshannoschwalm]

What I meant was exporting an image without CS and one with CS turned on with at least 15 iterations, then compare the images overlayed in separate layers in gimp, using the layer mode "difference".

You are aware of the --dump-pipe module option?

Nope, never used it, what is it good for?

It writes pfm files to tmp directory while processing the specified module in the pipe, it's meant for debugging exactly such issues :-)

darktable --dump-pipe demosaic would be your friend (use HQ processing mode)

[jenshannoschwalm]

Latest version includes a variance lut for performance plus some readability maintenance. (Some NaN issues spotted while testing this are in a bugfix PR for 5.2)

[jenshannoschwalm]

Latest version got some subtle perf gain, more important is improved stability for high-noise underexposed images. Right now i think it's good as is and i would appreciate testing. Please bear in mind, i have squashed and force-pushed the update to keep number of commits low.

I am still not convinced about the UI.

1. Is it really better to make the unused sliders insensitive instead of just hiding?
2. Any other idea?

[wpferguson]

@jenshannoschwalm do you have suggested settings or at least a good starting point?

[jenshannoschwalm]

Normally with a strength of 10 would be fine. Evaluate at 100% zoom.

[jenshannoschwalm]

Latest version - as usual load the whole pr for testing -

1. Does automatic calculation of radius if capture sharpen has been enabled
2. Some improved tooltips
3. The maximum radius has been increased to 2.0 as this is still acceptable with the convolution kernels.

[jenshannoschwalm]

make it working again with current master, convolution stuff has not changed, the "avoid-noisy-areas" threshold has been improved.

[jenshannoschwalm]

force-pushed rebased on master and minor code maintenance, nothing algo-wise changed.

[jenshannoschwalm]

1. use a different default contrast threshold after a lot of testing - definitely better for medium ISO images while still perfect for low ISO
2. Second commit while being here cleans up some demosaicer code.

[jenshannoschwalm]

Ready to go from my side. Latest force-push just has some more aggressive compiler options for CPU performance

[jenshannoschwalm]

Release note suggestion:
Implemented capture sharpen (CS) inside the demosaic module so only available for raw images.
xtrans and bayer sensors are supported, CPU and OpenCL code paths are provided.
When done with dual demosaicing capture sharpen is only applied to the main demosaicer.
Capture sharpening (CS) tries to recover details lost due to in-camera blurring, which can be caused by diffraction,
the anti-aliasing filter or other sources of gaussian-type blur.
The main parameters are "capture sharpen" defining the iterations and "radius".
To improve results for noisy images the effect can be controlled via the "contrast threshold" parameter.
As many lenses are less sharp in the corners we can do some additional sharpening there via "corner boost"

[jenshannoschwalm]

Minor code change to make params check happy.