Fixing the R/B swap the right way

If you’ve ever looked at a GPU render and seen blue where red should be, you’ve met the R/B swap problem. For etnaviv this has been a long-standing source of complexity. We were solving it in the shader, but the proprietary blob driver had a simpler approach all along. As part of my work at Igalia, I finally sat down and did it properly.

The problem

Vivante GPUs have a quirk: the Pixel Engine (PE) always writes pixels in BGRA byte order. When your API says “render to R8G8B8A8_UNORM”, what actually lands in memory is B, G, R, A. Every byte of every pixel, every frame. The hardware just works that way.

The question is: where do you fix it?

The etnaviv driver was doing it in the shader. Before the fragment shader writes its output, a NIR lowering pass swaps the R and B channels:

   alu->src[0].swizzle[0] = 2;   /* .r reads from .b */
   alu->src[0].swizzle[2] = 0;   /* .b reads from .r */

This works, until it doesn’t. The shader key needs a frag_rb_swap bitmask per render target. The blend color needs per-RT R/B swapping to match. And it falls apart entirely for scalar outputs - if a shader writes a single float, there’s no .z component to swizzle into .x. That’s exactly the NIR validation failure we hit:

Test case 'dEQP-GLES3.functional.fragment_out.basic.fixed.rgb8_lowp_float'..
NIR validation failed after etna_lower_io in ../mesa/src/gallium/drivers/etnaviv/etnaviv_compiler_nir.c:1296
1 errors:
shader: MESA_SHADER_FRAGMENT
source_blake3: {0x4d463d73, 0x4b27d742, 0x27a92b64, 0x375c010f, 0xb2ce3767, 0x2adc55cc, 0x6da8105b, 0x5b9fce29}
name: GLSL1
prev_stage: MESA_SHADER_VERTEX
inputs_read: 32
outputs_written: 4
perspective_varyings: 32
max_subgroup_size: 128
min_subgroup_size: 1
api_subgroup_size_draw_uniform: true
first_ubo_is_default_ubo: true
known_interpolation_qualifiers: true
flrp_lowered: true
inputs: 1
outputs: 1
decl_var shader_in INTERP_MODE_SMOOTH none highp float packed:var0 (VARYING_SLOT_VAR0.x, 0, 0)
decl_var shader_out INTERP_MODE_NONE none mediump float out0 (FRAG_RESULT_DATA0.x, 0, 0)
decl_function main () (entrypoint)

impl main {
    block b0:  // preds:
    32    %3 = load_const (0x00000000)
    32    %4 = @load_input (%3 (0x0)) (base=0, range=1, component=0, dest_type=float32, io location=VARYING_SLOT_VAR0 slots=1)  // packed:var0
    32    %2 = deref_var &out0 (shader_out mediump float)
    32    %5 = mov %4.z
error: src->swizzle[i] < num_components (../mesa/src/compiler/nir/nir_validate.c:217)

               @store_deref (%2, %5) (wrmask=x, access=none)
               // succs: b1
    block b1:
}

FATAL ERROR: Test program crashed

What the blob does

Looking at command stream traces from the proprietary driver, the answer is almost disappointingly simple. Instead of this:

  Texture format: A8B8G8R8     (read BGRA as BGRA)
  PE format:      A8B8G8R8     (write BGRA)
  Shader:         swap R <-> B

The blob does this:

  Texture format: A8R8G8B8     (read BGRA as RGBA - hardware swaps on read)
  PE format:      A8B8G8R8     (write BGRA - unchanged)
  Shader:         nothing

That’s it. Tell the texture sampler the data is A8R8G8B8, and it will correctly interpret the BGRA bytes as RGBA channels. The PE keeps writing BGRA because that’s what it does. No shader modification needed.

In our format table, the change is a single field:

  -  VT(R8G8B8A8_UNORM, UNSIGNED_BYTE, A8B8G8R8, A8B8G8R8)
  +  VT(R8G8B8A8_UNORM, UNSIGNED_BYTE, A8R8G8B8, A8B8G8R8)
                                        ^^^^^^^^
                                     texture format

The data flow

To understand why this works, trace a red pixel through the pipeline:

                          BGRA-internal byte order
                         +------------------------+
                         |                        |
  API: glClear(1,0,0,1)  |   Memory: [0,0,255,255]|   CPU: expects [255,0,0,255]
  "red = 1.0"            |   (B=0, G=0, R=255,    |   "RGBA order"
                         |    A=255)              |
                         +------------------------+

  +-----------+     +--------+     +--------+     +---------+
  | Shader    |     | PE     |     | Memory |     | Sampler |
  | out=RGBA  | --> | writes | --> | stores | --> | reads   |
  | (1,0,0,1) |     | BGRA   |     | BGRA   |     | as      |
  |           |     |        |     | bytes  |     | A8R8G8B8|
  +-----------+     +--------+     +--------+     +---------+
       |                |               |              |
    R=1.0            B=0x00          [00 00 FF FF]   R=1.0
    G=0.0            G=0x00                          G=0.0
    B=0.0            R=0xFF                          B=0.0
    A=1.0            A=0xFF                          A=1.0

The shader writes (1,0,0,1). The PE swaps R/B on write, so memory gets [0,0,255,255] in BGRA order. The sampler, told the format is A8R8G8B8, reads those same bytes back as (1,0,0,1). Round-trip complete, no shader involvement.

The CPU boundary problem

GPU-to-GPU is clean. But what happens at the CPU boundary - glReadPixels, glTexSubImage, glBlitFramebuffer to a CPU-mapped buffer? The CPU expects RGBA byte order. Memory has BGRA. Something needs to swap.

This is where the hardware copy/resolve engines come in - RS and BLT. Both can perform R/B swapping during their copy operations. The RS engine has a swap_rb bit. The BLT engine has per-side swizzle fields. We just need to activate this at the right moment.

The key insight: only transfer blits (tiled-to-linear copies for CPU access) need the swap. GPU-internal blits - glBlitFramebuffer between two render targets, TS resolve, mipmap generation - are all operating on data already in BGRA order on both sides. Swapping there would be wrong.

So we gate it with a context flag:

ctx->in_transfer_blit = true;
etna_copy_resource_box(pctx, trans->rsc, &rsc->base, ...);
ctx->in_transfer_blit = false;

And in the RS blit path:

.swap_rb = ctx->in_transfer_blit &&
           translate_pe_format_rb_swap(blit_info->src.format),

The texture shadow trap

With the basic approach working, tests passed on GC7000 (BLT engine). But GC2000 (RS engine) had a regression: fbo-blit showed blue where red should be.

After adding debug prints and tracing the code paths, the culprit was the texture shadow. Some resources can’t be sampled directly by the texture unit - for example, a render target might use a layout the sampler doesn’t understand. For these, the driver allocates a second copy of the resource in a sampler-compatible tiled layout. This is the “texture shadow”.

The shadow is a workaround that hurts performance - it means extra memory and extra copies. Ideally we wouldn’t need it at all. But while it exists, the driver uses it as a shortcut for CPU transfers: read directly from the shadow and detile in software, skipping the blit engine:

Passing probes:  PATH=temp_resource+RS_blit    swap_rb=1   -> correct
Failing probes:  PATH=texture_shadow                       -> R/B swapped

The texture shadow path does a software detile - raw byte copy, no R/B swap. With BGRA-internal byte order, that gives you BGRA bytes on the CPU side. Wrong.

The fix: skip the texture shadow shortcut for formats that need R/B swap, forcing through the blit engine path which handles the conversion:

if (rsc->texture && !etna_resource_newer(rsc, etna_resource(rsc->texture)) &&
    !translate_pe_format_rb_swap(prsc->format)) {
   /* Use texture shadow - safe, no R/B swap needed */
   rsc = etna_resource(rsc->texture);
} else {
   /* Use blit engine - handles R/B swap correctly */
   ...
}

Results

This is a net-negative patch series - 15 files changed, 76 insertions, 113 deletions. The etnaviv: Remove RB swap logic in the fragment shader contains all that’s needed:

1. blt: Use img->swizzle for CONFIG SWIZ fields - preparation for per-image swizzle support
2. Add translate_pe_internal_format helper - maps RGBA pipe formats to BGRA equivalents for clear color packing
3. Use BGRA-internal texture format with BLT/RS R/B swizzle - the main change
4. Compute blend color directly in etna_set_blend_color - simplifies blend color, no deferred update needed

Fixes the NIR validation failure with scalar fragment outputs. And as a nice side effect, removing the shader-based swap means fewer shader variants, fewer instructions and less overhead. glmark2-es2-wayland improves from ~835 to ~874 FPS - a 4.7% performance increase.

Sometimes matching what the hardware vendor does is the right answer. The blob driver figured this out years ago. We just needed to look at the traces.

That’s where I thought the story ended

The texture-format trick has a hidden assumption baked into it: that the GPU both writes and reads every resource. The PE writes BGRA, the sampler is told the format is A8R8G8B8, and the byte order cancels out. It’s a closed loop, and as long as the data never leaves the GPU, nobody outside ever sees the BGRA bytes.

dmabuf breaks the loop.

When a buffer is shared with another process - a Wayland compositor, a video decoder, a camera - the byte order is no longer our private business. It’s mandated by the DRM FourCC. An external producer writes honest RGBA bytes into the buffer. Then our sampler, still convinced the format is A8R8G8B8, reads them as BGRA. Red and blue swap. And on the way out, a transfer blit happily swaps data that was already correct. The optimization that made GPU-internal rendering clean made buffer sharing wrong.

  GPU-internal (closed loop):       dmabuf (loop broken):

  PE writes BGRA                    external producer writes RGBA
       |                                  |
  sampler reads as A8R8G8B8         sampler reads as A8R8G8B8
       |                                  |
  cancels out -> correct            reads RGBA as BGRA -> swapped

Step back out for shared resources

The first fix is the obvious one: when a resource is shared, don’t play the trick. Use the native A8B8G8R8 texture format so the sampler reads RGBA bytes as RGBA, skip the R/B swizzle in the BLT and RS transfer blits, and re-enable the texture shadow shortcut that the swizzle had forced us to disable. Internal resources keep the BGRA-internal optimization untouched.

That handles imports. But there’s a case it doesn’t cover: a resource we rendered into and then export. The PE wrote BGRA, because that’s all the PE knows how to do. The external consumer expects native order.

Here a second kind of shadow shows up. Just as the sampler gets a texture shadow when it can’t read the base layout, the PE gets a render shadow - a render-compatible copy - when it can’t draw into the base layout. (On some GPUs, like GC2000, no single tiling satisfies both the texture engine and the pixel engine, so a resource can end up carrying both shadows.) When an exported resource is flushed, that render shadow is resolved back to the base. So we hook etna_flush_resource() and do the R/B swap during that copy - using the BLT destination swizzle or the RS SWAP_RB bit. The swap rides along on a copy we were doing anyway.

One buffer, two byte orders

Now the same shared buffer can be in one of two states. Just imported, or just flushed for export? Native RGBA. Freshly rendered by the PE, not yet flushed? PE-internal BGRA. A static texture format chosen at sampler-view creation can’t be right for both.

So the format choice becomes dynamic. A shared_native_order flag tracks which order the bytes are currently in, and the sampler-view format follows from it:

Both texture paths - state-based and descriptor-based - pre-compute the native format variant at sampler-view creation, so picking the right one at emit time costs a single branch, not a per-frame format recompute.

The shader swap comes back

Which brings us to LINEAR_PE GPUs, and a twist I didn’t see coming.

On these GPUs, a linear shared resource is render-compatible. There is no render shadow - the PE writes straight into the buffer that gets handed to the compositor. So after a draw, the buffer holds BGRA, and flush_resource() would have to issue a full-surface blit to swap it. That’s real bandwidth, on every frame, that the old shadow-based path never paid.

There’s a cheaper place to do the swap: in the shader, on the way out. Which is exactly the thing this whole series set out to delete.

So it comes back - but only for this one case, and done properly. A per-RT frag_rb_swap bitmask in the shader key drives a NIR lowering pass that swaps channels 0 and 2 on the fragment output. The original shader swap fell over on scalar outputs, because there was no .z to swizzle from. This one widens the output variable to vec4 first, padding the missing components with undef, then applies the swizzle with an adjusted writemask:

   /* Pad source to 4 components (undef for missing) */
   nir_def *padded = nir_pad_vec4(&b, src);

   /* Swap R and B channels */
   unsigned swiz[] = {2, 1, 0, 3};

That’s the scalar rgb8_lowp_float crash from the top of this post - fixed, in the one path that now needs a shader swap at all.

Wiring it up is a check in etna_draw_vbo():

   if (VIV_FEATURE(screen, ETNA_FEATURE_LINEAR_PE)) {
      for (i = 0; i < pfb->nr_cbufs; i++) {
         struct etna_resource *rsc = etna_resource(pfb->cbufs[i].texture);
         if (rsc->shared && rsc->layout == ETNA_LAYOUT_LINEAR &&
             translate_pe_format_rb_swap(pfb->cbufs[i].format))
            key.frag_rb_swap |= (1 << i);
      }
   }

The bitmask is per-RT, so an MRT setup with a mix of shared and private targets does the right thing for each. And because the shader produced native bytes directly, shared_native_order stays true and flush_resource() skips its blit entirely.

The fixes for all of this live in a follow-up series, etnaviv: Fix dmabuf R/B byte order for PE_FORMAT_RB_SWAP formats.

So was removing the shader swap a mistake?

No - but it wasn’t the whole answer either.

The shader swap was wrong as the universal solution. It cost a shader-key dimension, per-RT blend-color fixups, and it crashed on scalar outputs. The texture-format trick is genuinely better for the common case, where a resource lives and dies on the GPU.

What the dmabuf work showed is that there is no single right place to fix R/B order. There’s a place that’s cheapest for each path: the texture format for GPU-internal resources, a transfer-blit swap at the CPU boundary, a flush-time swap for exported render targets, and - for LINEAR_PE, where the PE writes straight into a shared buffer - the shader, after all. The trick isn’t picking one. It’s knowing which boundary you’re standing on, and swapping there.