What is HDR?
HDR (High Dynamic Range) is a term with several definitions in imaging:
Stacking images with bracketed exposures (darker and brighter images) to increase the dynamic range of the output image beyond what a single frame could capture
Encoding the image with a colour profile that encodes a broader range of colours and luminance (like REC2020, or Dolby Vision) than SDR (sRGB or REC709), to allow the image to map onto an HDR display (a display that can render very bright and dark things at the same time)
Adding a gain map to an SDR image to optionally supplement the RGB data of each pixel with additional offset RGB values that would allow the image to render in HDR.
In this case, we are talking about the latter two, and serving them over the web.
Challenge #1 - Hardware Fragmentation
Greg Benz's blog on HDR has an excellent article describing the landscape of HDR support on various devices.
The TL;DR is that unless you have a recent flagship Android phone, iPhone, Macbook Pro 14 or 16in, a Pro Display XDR, or a handful of other devices, you won't be able to view HDR content at all.
This isn't to say that HDR is useless, though, because recent iPhones and flagship android phones make up a large portion of web users, and of those willing to pay for expensive photoshoots, penetration of high end HDR capable devices is quite high, not to mention the near-ubiquity of Macbooks Pro among media professionals.
Challenge #2 - Software Inconsistency
- Safari (and thus WebKit, all iOS browsers) will not render HDR images in HDR, even when the same image in Photos will render in HDR
- Chrome and chromium-based browsers support HDR well, with AVIF and JPG+Gain Map
- HDR related CSS media queries are unreliable and inconsistent, IE query for `dynamic-range: high` does not garuntee that the user's device will be able to render an HDR image properly. These queries are also not very well supported or consistent between browsers.
Challenge #3 - Operationalizing HDR for Web
Modern web developers often rely on a smart CMS to optimize their images for the many diverse devices that will receive them.
These systems can do things like transcoding, resizing, and smart cropping images so that they look right and load fast on all screen sizes.
Given a high quality source image, a tool like Cloudinary can encode it in the best format, at the right dimensions for the user's display, while smart cropping to faces or other points of interest in the image.
The issue is that most CMS (content management systems such as Sanity, Gatsby, Cloudinary) have undefined behaviour when processing images with HDR colour space or gain maps. This is because:
- gain maps are too new
- image processing pipelines in these services are often not colour managed
- processing algorithms may be hard-coded to work in sRGB/rec709, resulting in gamma-related colour distortions in certain processing steps like downsampling, convolutions.
This results in an unwanted tradeoff - do we want correctly sized and cropped images, or HDR?
Solution #1 - Wait
We expect HDR displays to become commonplace on consumer devices. This will take time, but technology does trickle down. Since the iPhone 12 series, HDR displays have been on all new iPhones.
Solution #2 - Clever Client-Side Software
We can solve many of the HDR rendering problems with a few simple subproblems, and a way to identify which situation we are in:
HDR Unsupported
If HDR media queries all fail, we can be very sure that HDR content will not render on the current device. We defer to standard SDR rendering
HDR Supported - Chromium
If HDR is supported, and we are on a Chromium browser, we can render HDR content using JPG+Gain Map or AVIF. If AVIF is supported, it is preferred due to better compression efficiency.
HDR Supported - WebKit (Safari)
If HDR is supported, but we are on WebKit, we need to use the "video workaround", where HDR photos are encoded as 1 frame HDR videos. This way, the content presents as HDR even on iOS.
Solution #3 - Clever Server-side Software (custom CMS)
Once we know what device we are on (see solution #2), we can query our backend for the right image. This is determined by:
- Codec support (AVIF, JXL, WebP, JPG+GainMap, AV1 Video)
- Image dimensions (aspect ratio)
- Size preference (high DPI vs data savings)
- HDR type (none or WebKit or Chromium)
Example
- HDR on Chromium on Pixel 8 Pro: `JPG+GainMap 9:16, data savings, Chromium -> image-720-1280-gainmap.jpg`
- HDR on Chromium on Macbook Pro 14in: `AVIF REC2020, 16:9, DPI, Chromium -> image-2560-1440-rec2020.avif`
- SDR on Chromium on XPS 15: `AVIF sRGB, 16:9, DPI, Chromium -> image-2560-1440-srgb.avif`
Backend Design
To be able to generate this type of output, we need to make a system similar to Cloudinary, but somewhat more robust from a colour management perspective.
- Stores a high quality "master image" which has the highest resolution and broadest colour space
- Generate image metadata for use when serving -- face coordinates, points of interest, dominant colours for creating a blur hash
- Upon request, check AWS CloudFront for an asset already in CDN matching request, if it doesn't exist, run the following (AWS Lambda)
- Convert the master image to linear gamma (if not already linear)
- Apply crop if needed, taking into account points of interest
- Apply colourspace and gamma transform to put image into requested format
- Generate correctly encoded image and push to CloudFront
Development Challenges
Many libraries I have tried to use for image encoding do not allow for tight control of things like colour space metadata, end to end colour management, especially when working with AVIF HDR. Currently the .exr based workflow is extremely data inefficient, and AVIF images cannot be easily served at all. Google's libultrahdr is great though.
Project Status
This project is currently under construction, for the re-design of sashaphoto.ca. The goal is to serve HDR images with a photographer workflow that is the same as class leading products like Cloudinary.
When the tool is complete and battle tested on sashaphoto.ca, it will be released.
Want to work together? Don't hesitate to reach out to hello@thebigsasha.com