Lei Zhou
Case study · Mixed-criticality and virtualisation

GPU virtualisation for a software-defined vehicle cockpit

Moving an Android Automotive guest from virgl to gfxstream, with Vulkan, GLES through ANGLE and zero-copy rendering from guest memory, then optimising it to the customer’s performance targets.

Linaro · technical delivery lead and architect
Context

Two criticality levels, one GPU

Any platform that runs a critical workload beside a general-purpose operating system on one GPU meets this problem: a vehicle cockpit, a robot whose control stack shares the GPU with perception and a user interface, or an edge AI system beside its control software. Here it was a cockpit.

The platform is an Arm automotive SoC running a type-1 hypervisor. In the product architecture, the instrument cluster, whose functions are safety-relevant, renders on the host, and the infotainment system is an Android Automotive guest that reaches the same GPU through virtio-gpu. The GPU has no hardware virtualisation isolation, so any separation between the two has to come from software.

The guest’s graphics ran on virgl, and the guest had only software Vulkan. The target was gfxstream: the guest’s Vulkan and GLES calls streamed to a renderer on the host, with hardware Vulkan in the guest.

The challenge

Three constraints that the textbook design does not meet

  • Compatibility at the boundaries. The guest’s Android release was fixed: it was the customer’s functioning production guest, built from an older AOSP release, and moving it to a newer release would have brought BSP bring-up work with it. The VMM’s renderer binding was written for a newer renderer interface, and the renderer that matches the older guest and the newer renderer come from different release lines. Whether any renderer could serve both sides was the main design risk.
  • No runtime memory sharing. This hypervisor stack cannot map host memory into a guest at runtime. gfxstream’s shared memory (its command ring, host-visible Vulkan memory and host-allocated buffers) assumes exactly that.
  • Performance targets. The customer set targets for virtualisation loss against host-native rendering and for the host CPU spent on GPU virtualisation. The Vulkan zero-copy path also had to work on both the proprietary and the open-source host GPU driver.
My role

Architecture, then delivery

I was the technical delivery lead and architect, and I owned both the migration and the performance optimisation through to delivery. That included the migration architecture: the compatibility strategy at each boundary and the zero-copy memory design.

The approach

Read the contracts before building across them

  • Each boundary checked from source first. The renderer interface entry point by entry point, with struct layouts on the target architecture; the guest protocol by opcode table, handshake, capability fields and extension allowlists.
  • Every decision recorded with its rejected alternatives and the conditions that would re-open it.
  • Function before performance. The whole functional chain was proven end to end before optimisation started, so the riskiest unknowns surfaced first and each step changed one variable.
  • Optimisation by attribution. Per-frame ledgers that sum to the measured frame time, a predicted counter for every theory, and keep-or-drop rules written before the data.
Managing risk

Each major risk had a named mitigation before work depended on it

RiskMitigation
No renderer serves both the production guest and the VMMBoth contracts compared from source before building; a compatibility adapter designed and held as the fallback; the conditions that would re-open the decision written down with it
The platform cannot share host memory with the guestSharing reversed to guest-owned memory; runtime sharing in the hypervisor recorded as the long-term recommendation
The biggest unknowns surface lastCoherency, decode coverage and the memory budget investigated early, in parallel with the critical path; function proven end to end before optimisation
A protocol change hangs the guestFeature negotiation, mixed old and new versions on each side, and a targeted stress test before the change was kept
An optimisation that only appears to helpA predicted counter for each theory and a keep-or-drop rule written before the data
What we built

A gfxstream path that fits the platform

  • A renderer that serves both contracts. A newer renderer release that matches the VMM’s interface and decodes every call the production guest can reach, so the guest stayed on its release, with changes confined to its graphics libraries and kernel. A compatibility adapter was designed and kept as the documented fallback.
  • Zero copy from guest memory. The guest allocates graphics memory; the VMM exports those pages as a dma-buf; the host GPU driver imports it. The guest owns cache maintenance on the non-coherent GPU, maps coherent memory types write-combined, and keeps the command ring in its own memory.
  • GLES through ANGLE on Vulkan, so all guest rendering takes the Vulkan path.
  • A display path with one GPU copy into a small ring of dma-buf scanout buffers, instead of one GPU copy and two CPU copies.
  • Overhead removed where it was found: two busy loops in the VMM’s GPU device, a watchdog’s bookkeeping on the guest render thread, a doorbell after every ring write, and shader modules re-created every frame.

The full design, the compatibility analysis and the optimisation method are in the companion article, From virgl to zero-copy gfxstream.

Results

On target, and on par with the previous stack

FunctionVulkan and GLES (through ANGLE) end to end on gfxstream, rendering in place in guest memory; display down from three copies to one GPU copy
OverheadMean virtualisation loss within the customer’s target at every measured resolution
Host CPUWithin target at idle and under the application benchmark
ParityApplication benchmark on par with the previous stack, now with hardware Vulkan in the guest
DriversVulkan zero copy working on both the proprietary and the open-source host GPU driver
DeliveryPatch series against the customer’s own trees; upstream backports keep their upstream authorship