Some thoughts on ProRes RAW and ProRes RAW HQ

Just before NAB 2018, Apple announced ProRes RAW and ProRes RAW HQ, a new video codec aimed at professionals. When ProRes was originally launched it supported the YCbCr color space with 4:2:2 and 4:4:4:4 chrominance subsampling. ProRes is popular as an intermediate codec for transcoding video from the camera for editing or as a mezzanine codec for storing high quality edited video before transcoding to a delivery format. The 4:2:2 version is preferable for general purpose compression at high quality and 4:4:4:4 for applications such as chromakey or when on screen text and graphics will be added.

The key new ingredient to ProRes RAW is that it compresses Bayer filter images. Using a color filter is a common way to make a color camera using a single sensor. Other techniques include using a color wheel (bigger and not good for higher frame rates) or multiple image sensors (more expensive and bigger). The process of demosaicing - obtaining a full RGB image from Bayer - is performed after decompression. Final Cut Pro X has GPU optimized decoding for ProRes RAW so demosaicing can be performed in software without introducing latency.

Cropped and Zoomed In

 

Bayer Version of Previous Image

 

ProRes RAW will be useful for storing video direct from camera onto an external recorder, raw workflows and HDR (High Dynamic Range). ProRes 4:2:2 and 4:4:4:4 will continue to be useful as intermediate codecs; there won't be much point using ProRes RAW for encoding rendered output.

How does it work ? It is quite easy to imagine that ProRes RAW could simply fill the gaps in the Bayer images using a low pass filter, encode each color channel as a prediction from a chosen reference channel e.g. green, using quantized DCT as in JPEG/ProRes and finally when decompressing recover the Bayer image ignoring the filled in gaps.

Apple's white paper on ProRes RAW shows graphs of ProRes RAW with half the data rate compared to ProRes 422, that is because Bayer requires 1 sample per pixel but YCbCr 4:2:2 requires the equivalent of 2 samples per pixel. ProRes RAW HQ is shown with half the data rate of ProRes 4444, in this case YCbCr plus alpha requires 4 samples per pixel but RAW requires 1 sample. There is no form of ProRes RAW with alpha channel.

 

The other graphs show ProRes RAW fitting more frames into memory and being faster to work with than other codecs, something you would expect given that Apple optimized their codec after all.