Two criticality levels, one GPU
Any platform that runs a critical workload beside a general-purpose operating system on one GPU meets this problem: a vehicle cockpit, a robot whose control stack shares the GPU with perception and a user interface, or an edge AI system beside its control software. Here it was a cockpit.
The platform is an Arm automotive SoC running a type-1 hypervisor. In the product architecture, the instrument cluster, whose functions are safety-relevant, renders on the host, and the infotainment system is an Android Automotive guest that reaches the same GPU through virtio-gpu. The GPU has no hardware virtualisation isolation, so any separation between the two has to come from software.
The guest’s graphics ran on virgl, and the guest had only software Vulkan. The target was gfxstream: the guest’s Vulkan and GLES calls streamed to a renderer on the host, with hardware Vulkan in the guest.
Three constraints that the textbook design does not meet
- Compatibility at the boundaries. The guest’s Android release was fixed: it was the customer’s functioning production guest, built from an older AOSP release, and moving it to a newer release would have brought BSP bring-up work with it. The VMM’s renderer binding was written for a newer renderer interface, and the renderer that matches the older guest and the newer renderer come from different release lines. Whether any renderer could serve both sides was the main design risk.
- No runtime memory sharing. This hypervisor stack cannot map host memory into a guest at runtime. gfxstream’s shared memory (its command ring, host-visible Vulkan memory and host-allocated buffers) assumes exactly that.
- Performance targets. The customer set targets for virtualisation loss against host-native rendering and for the host CPU spent on GPU virtualisation. The Vulkan zero-copy path also had to work on both the proprietary and the open-source host GPU driver.
Architecture, then delivery
I was the technical delivery lead and architect, and I owned both the migration and the performance optimisation through to delivery. That included the migration architecture: the compatibility strategy at each boundary and the zero-copy memory design.
Read the contracts before building across them
- Each boundary checked from source first. The renderer interface entry point by entry point, with struct layouts on the target architecture; the guest protocol by opcode table, handshake, capability fields and extension allowlists.
- Every decision recorded with its rejected alternatives and the conditions that would re-open it.
- Function before performance. The whole functional chain was proven end to end before optimisation started, so the riskiest unknowns surfaced first and each step changed one variable.
- Optimisation by attribution. Per-frame ledgers that sum to the measured frame time, a predicted counter for every theory, and keep-or-drop rules written before the data.
Each major risk had a named mitigation before work depended on it
| Risk | Mitigation |
|---|---|
| No renderer serves both the production guest and the VMM | Both contracts compared from source before building; a compatibility adapter designed and held as the fallback; the conditions that would re-open the decision written down with it |
| The platform cannot share host memory with the guest | Sharing reversed to guest-owned memory; runtime sharing in the hypervisor recorded as the long-term recommendation |
| The biggest unknowns surface last | Coherency, decode coverage and the memory budget investigated early, in parallel with the critical path; function proven end to end before optimisation |
| A protocol change hangs the guest | Feature negotiation, mixed old and new versions on each side, and a targeted stress test before the change was kept |
| An optimisation that only appears to help | A predicted counter for each theory and a keep-or-drop rule written before the data |
A gfxstream path that fits the platform
- A renderer that serves both contracts. A newer renderer release that matches the VMM’s interface and decodes every call the production guest can reach, so the guest stayed on its release, with changes confined to its graphics libraries and kernel. A compatibility adapter was designed and kept as the documented fallback.
- Zero copy from guest memory. The guest allocates graphics memory; the VMM exports those pages as a dma-buf; the host GPU driver imports it. The guest owns cache maintenance on the non-coherent GPU, maps coherent memory types write-combined, and keeps the command ring in its own memory.
- GLES through ANGLE on Vulkan, so all guest rendering takes the Vulkan path.
- A display path with one GPU copy into a small ring of dma-buf scanout buffers, instead of one GPU copy and two CPU copies.
- Overhead removed where it was found: two busy loops in the VMM’s GPU device, a watchdog’s bookkeeping on the guest render thread, a doorbell after every ring write, and shader modules re-created every frame.
The full design, the compatibility analysis and the optimisation method are in the companion article, From virgl to zero-copy gfxstream.