Access to HDR/High bit depth/additional channels in images and video - PNG and JPEG-XL and H264

[wayfarer3130]

Several of the image formats and video formats now defined allow defining both additional channels and higher bit depths (9+). Access to this data as raw data is critical for some types of applications such as:

1. Video editing - so that the original data can be set or modified
2. Medical images - so that the native image decoder can be used instead of re-implementing one in WASM
3. Overlay display - to control additional overlays such as identification of a given area
4. Custom rendering such as rendering an object with reflectivity values encoded in the raw image
5. Creation/publication of HDR data
6. Viewing of scenes at specified lighting levels - the transform would have to be on the raw data, so that at high brightness levels, none of the dark data is visible, while at low lightness levels, only the dark data is visible (eg games showing a door opening)

It would be extremely helpful to have the webcodecs API include access to the raw image data, any number of channels as not just byte data but also 16 or 32 bit data, as well as 1 bit data (because accessing the data as individual 1 bit in 8 bit objects is very inefficient).

[chcunningham]

Support for higher bit depths is definitely on the road map. Supporting 1 bpp formats should also be doable. Which 1 bpp format(s) are most common?

I'm a little confused about "additional channels"... do mean something more than an alpha channel? Can you point to an example of a codec API that does what you want?

Rendering HDR VideoFrames/Images is somewhat blocked on Canvas support (context here and here).

[wayfarer3130]

By additional channels, I mean things like region identification (segmentation in medical parlance), or additional wavelengths of light (infrared, fluoroscopy etc). The idea is to do things like being able to click on a location and have an overlay that indicates what type of object is at that location, or being able to correct for lighting types in an image, or show what the image would look like under black light. Yes, I know one can add those as extra images, but it would be nice to have it all in one place, as the data actually correlates to the image object. The segmentation might be a single bit - for example "heart" might be a single channel, and there might be a bunch of channels for different things, or it might be an identifier lookup based on, say value 3 int he "identifier" channel. Bill

…

On Tue, 18 Jan 2022 at 13:04, chcunningham ***@***.***> wrote: Support for higher bit depths is definitely on the road map. Supporting 1 bpp formats should also be doable. Which 1 bpp format(s) are most common? I'm a little confused about "additional channels"... do mean something more than an alpha channel? Can you point to an example of a codec API that does what you want? Rendering HDR VideoFrames/Images is somewhat blocked on Canvas support (context here <https://github.com/WICG/canvas-color-space/blob/main/CanvasColorSpaceProposal.md#additional-color-spaces-high-bit-depth-and-high-dynamic-range> and here <https://github.com/w3c/ColorWeb-CG>). — Reply to this email directly, view it on GitHub <#427 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGT56XKXJMNVNZUPK7O4PYDUWWTUVANCNFSM5KOF64TQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[sandersdan]

I'm familiar with this sort of use in the FITS image format for astronomy, but I'm not aware of any image format supported by browsers with these sorts of features. Are you asking for just the API portion so that a polyfill could implement this, for browsers to support a new format, or is there an existing, supported format that we simply need to expose more features from?

[wayfarer3130]

JPEG XL supports all of those features, and so does JPEG XR. I'm voting on JPEG XL because of the backwards compatibility with JPEG and the higher compression ratios at very similar CPU cost to decompress. It also supports HDR lossless images, which is really what I'm interested in for display in a browser. The extra channels aren't really required, but if one is going to handle direct access to the HDR data before converting it to 8 bit RGB, it would be really nice to allow access to all the channels/data, and not just grayscale or RGB data. The access for medical images is used to show the detail in, for example bone or in muscle - the ranges for each are so different that showing bone data would make the muscle completely black or white, or if you show both, then you lose so much detail in both of them that neither one is useful. For something like a game, imagine using lossless high bit depths to store HDR to allow displaying the same image from day to night time. Then, if you also have extra channels, then you can do things like using an extra channel to identify items in the game that can be clicked on, or, to automatically allow highlighting certain objects by using a different channel as an alternate alpha. There are lots of things one can do with additional image bits and channels. Thanks, Bill Bill

…

On Tue, 18 Jan 2022 at 14:23, Dan Sanders ***@***.***> wrote: I'm familiar with this sort of use in the FITS image format for astronomy, but I'm not aware of any image format supported by browsers with these sorts of features. Are you asking for just the API portion so that a polyfill could implement this, for browsers to support a new format, or is there an existing, supported format that we simply need to expose more features from? — Reply to this email directly, view it on GitHub <#427 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGT56XOAAUIKDCW4MOAHV53UWW4ZRANCNFSM5KOF64TQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[dalecurtis]

Duplicate of #384