Determining RGB Format in libavif During Decoding

When working with libavif for AVIF image decoding, the library doesn’t automatically store or retrieve the RGB format from the original encoding. You must manually set the RGB format using avifRGBImage before calling avifImageYUVToRGB(), as the decoded avifImage only contains YUV data without format information. The avifImageYUVToRGB function preserves whatever format you set in the avifRGBImage structure rather than inferring it from the original encoding.

Contents

• Understanding libavif RGB Format Handling During Decoding
• How to Set RGB Format When Decoding AVIF Images
• Differences Between Encoding and Decoding Format Management
• Practical Examples of RGB Format Detection in libavif

Understanding libavif RGB Format Handling During Decoding

When working with libavif for AVIF image decoding, it’s crucial to understand how RGB format handling works. Unlike encoding where you explicitly set the RGB format, the decoding process doesn’t automatically preserve this information. When you decode an AVIF file, the library gives you an avifImage that contains YUV data, but it does not store the RGB format that was used during encoding.

The avifRGBFormat field belongs to the avifRGBImage you pass to avifImageYUVToRGB(), not to the decoded image itself. This is a fundamental concept that many developers overlook when working with libavif. The libavif documentation clearly states that the decoded image only contains YUV information without the original RGB format metadata.

This design choice makes sense from a technical standpoint because AVIF images are stored in YUV format internally, and the RGB conversion is performed during decoding based on the format you specify. The library separates the format decision from the conversion process, giving you control over the output format.

How to Set RGB Format When Decoding AVIF Images

To properly handle RGB format during decoding, you need to manually set it in the avifRGBImage structure before calling avifImageYUVToRGB(). By default avifRGBImageSetDefaults() sets rgb->format to AVIF_RGB_FORMAT_RGBA, and avifImageYUVToRGB() will convert to whatever format you have set at that moment.

Here’s the recommended approach:

avifRGBImage rgb;
avifRGBImageSetDefaults(&rgb, image); // sets format to RGBA by default
rgb.format = AVIF_RGB_FORMAT_GRAY; // choose grayscale if needed
avifImageYUVToRGB(image, &rgb); // conversion uses the format you set

The key insight is that you control the RGB format through the avifRGBImage structure, not by retrieving it from the original encoded data. This approach gives you flexibility to choose different RGB formats based on your application’s needs, such as converting to grayscale for certain use cases or maintaining the full RGBA format for others.

When deciding which format to use, you should consider factors like:

• The source image’s yuvFormat and depth
• Your application’s requirements for color representation
• Performance considerations for different formats
• Memory usage implications of different formats

Differences Between Encoding and Decoding Format Management

The way RGB formats are handled differs significantly between encoding and decoding in libavif, which often confuses developers new to the library.

During encoding, you explicitly set the RGB format when creating the encoder and providing input image data. The encoder takes your RGB data and converts it to YUV for storage. The format you choose affects how the color information is processed and compressed.

During decoding, however, the process is reversed. The library extracts YUV data from the AVIF file, but doesn’t automatically know or store what RGB format was used during encoding. Instead, you must specify the desired output RGB format when setting up the avifRGBImage structure before calling avifImageYUVToRGB().

This asymmetry is intentional and reflects the different purposes of encoding and decoding:

• Encoding: Takes RGB input and produces compressed YUV output
• Decoding: Takes compressed YUV input and produces RGB output based on your specifications

The libavif decode example demonstrates this complete workflow, showing how to handle both 8-bit and 16-bit depth RGB data by using appropriate pixel pointer types.

Practical Examples of RGB Format Detection in libavif

Let’s look at a practical example of how to handle RGB format during decoding:

// Initialize decoder and decode the AVIF file
avifDecoder* decoder = avifDecoderCreate();
avifResult result = avifDecoderReadFile(decoder, "input.avif", NULL);

if (result == AVIF_RESULT_OK) {
 avifImage* image = decoder->image;
 
 // Set up RGB image structure
 avifRGBImage rgb;
 avifRGBImageSetDefaults(&rgb, image);
 
 // Choose appropriate RGB format based on your needs
 if (image->yuvFormat == AVIF_PIXEL_FORMAT_YUV400) {
 // Grayscale input, use grayscale output
 rgb.format = AVIF_RGB_FORMAT_GRAY;
 } else {
 // Color input, use RGBA
 rgb.format = AVIF_RGB_FORMAT_RGBA;
 }
 
 // Allocate memory for RGB pixels
 avifRGBImageAllocatePixels(&rgb);
 
 // Perform YUV to RGB conversion
 avifImageYUVToRGB(image, &rgb);
 
 // Now you can work with the RGB pixels
 // rgb.pixels contains the RGB data
 // rgb.rowBytes contains the row stride
 
 // ... process RGB data ...
 
 // Clean up
 avifRGBImageFreePixels(&rgb);
}

avifDecoderDestroy(decoder);

In this example, we’re checking the input image’s YUV format to determine the appropriate RGB output format. If the input is grayscale (YUV400), we output grayscale RGB. For color images, we use RGBA format.

Another common scenario is handling different bit depths:

// Handle different bit depths
avifRGBImage rgb;
avifRGBImageSetDefaults(&rgb, image);

if (image->depth > 8) {
 // 16-bit depth
 rgb.format = AVIF_RGB_FORMAT_RGBA_16;
 rgb.depth = 16;
} else {
 // 8-bit depth
 rgb.format = AVIF_RGB_FORMAT_RGBA;
 rgb.depth = 8;
}

avifRGBImageAllocatePixels(&rgb);
avifImageYUVToRGB(image, &rgb);

// Process the appropriate pixel type
if (image->depth > 8) {
 uint16_t* pixels = (uint16_t*)rgb.pixels;
 // ... process 16-bit pixels ...
} else {
 uint8_t* pixels = (uint8_t*)rgb.pixels;
 // ... process 8-bit pixels ...
}

Remember that there is no API to query the original RGB format from the decoded image; you must set it manually based on your application’s requirements or the characteristics of the input image.

Conclusion

In summary, determining the avifRGBFormat when decoding with libavif requires a different approach than many developers expect. The library doesn’t store or provide access to the original RGB format from encoding, so you must manually set it in the avifRGBImage structure before calling avifImageYUVToRGB(). This design choice gives you control over the output format based on your application’s needs rather than relying on whatever format was used during encoding.

To properly handle RGB format during decoding:

1. Initialize the avifRGBImage structure using avifRGBImageSetDefaults()
2. Manually set the desired RGB format (e.g., AVIF_RGB_FORMAT_GRAY, AVIF_RGB_FORMAT_RGBA)
3. Call avifImageYUVToRGB() with your configured avifRGBImage
4. Work with the resulting RGB data in your chosen format

This approach provides flexibility while maintaining clear separation between the library’s internal YUV representation and your desired RGB output format.

Sources

1. libavif header documentation — Official API documentation explaining RGB format handling: https://github.com/AOMediaCodec/libavif/blob/main/include/avif/avif.h
2. libavif decode example — Complete decoding workflow demonstrating RGB format management: https://github.com/AOMediaCodec/libavif/blob/main/examples/avif_example_decode_file.c