We noticed a big jump in GPU memory when we switch to Vulkan on Qualcomm hardware. Further analysis shows that the driver is very sensitive to multiple allocations instead of large memory pools. It shows a behavior where multiple engine side allocations result in driver side memory buckets of sizes 8/16/33/66/132 MB which are attributed to the process but never freed over the lifetime of the process even if the engine side allocation was temporary. This happens across all memory types (host visible, GPU only and even PSO uploads can trigger those driver side buckets).
When trying to understand why Vulkan is 300+ MB larger we found this pattern in the driver GPU allocations.
We run with a hypervisor so detail GPU memory information is visible in HGSL for us not KGSL. It shows distinct sizes for the driver side buckets (see rounded MB sizes above).
Please let us know what we can do to mitigate this issue.
[Attachment Removed]
Hi Konstantin,
The memory gap between GLES and Vulkan on Android is known and real. It is not caused by a single inefficiency but by the cumulative effect of Vulkan’s explicit design:
- Transient render targets often receive full memory backing instead of remaining purely tile-resident.
- Sub-allocation introduces allocator slack and fragmentation.
- Multiple frames in flight require duplicated transient resources.
- PSO caches and compiled pipeline data remain resident.
- Resource eviction and residency management are less aggressive than in GLES.
As a result, Vulkan typically consumes noticeably more memory than GLES on Android, even when rendering identical content. The largest opportunities for reducing the gap on 5.5:
- Improving allocator efficiency and defragmentation (CL 36229276 with more allocator improvements coming over the summer months**)**.
- Reducing frames-in-flight (r.OneFrameThreadLag) .
- Introducing more aggressive texture and resource eviction mechanisms comparable to those available (planned future development).
- Push more render targets to memoryless (no imminent development plans).
How much of the current measured overhead is your hardware platform able to absorb currently?
Best regards.
[Attachment Removed]
Hi Stéphane,
thank you for your reply.
Unfortunately, we are unable to absorb any of it, in fact we are trying to find ways to safe RAM and have another ticket open for that. Due to this RAM contention and the numbers we measured our switch to Vulkan is blocked.
Would upgrading to 5.7 or 5.8 improve Vulkan RAM usage at all?
I would like to discuss the distinct pattern I noticed where the Qualcomm driver creates “helper” memory buckets with very distinct size patterns which are not directly attributable to any game resource. I’ve tested this with a bare bones Vulkan application. These buckets appear and double in size when the driver experiences certain stressors like many individual texture allocations instead of a single shared texture memory pool. When those textures are deallocated the driver side bucket sticks around till the Vulkan device is destroyed. In the worst case I see driver side buckets of 132 MB for various memory types (GPU only, host-visible, etc.).
Our UE5 project suffers from those buckets a lot. I think we see multiple 132 MB and 66 MB buckets in the HGSL memory details.
Are you suggesting there is nothing wrong on the drive side? I would have hopped that these driver buckets shrink when client side objects are destroyed.
Kind regards,
Konstantin
[Attachment Removed]
Hi Konstantin,
You are correct. UE does indeed allocate internal buckets from the driver for performance reasons. However, we do have a fallback to force every resource allocation into its own driver allocation. Can you try sett
r.Vulkan.SingleAllocationPerResource=1and let us know if you see an improvement on this front? Android paging requirements may play here.
Could you additionally share the output of:
r.Vulkan.DumpMemoryand
r.Vulkan.DumpMemoryFull
Best regards.
[Attachment Removed]
Hi Stéphane,
I’m not talking about UE requested/allocated memory pools. The driver opaquely creates it’s own memory buckets on top of memory allocated by UE in doubling sizes of 8/16/33/66/132 MB when it is used in certain ways.
These allocations will only show up in HGSL output but fully count against the process PSS/RSS. These “helper” allocations can not be queried through any public Vulkan API. The worst part about those buckets is there permanent nature. They do not get freed once they are present, even if UE deletes all it’s resources. I’ll provide the output of HGSL along side your requested data when I run in Vulkan mode and point out the problematic buckets.
pid=$(pidof PRCESSNAME) && cat /sys/kernel/debug/hgsl/clients/$pid/obj_types && cat /sys/kernel/debug/hgsl/clients/$pid/mem
[Attachment Removed]
Hi Konstantin,
We understand the underlying driver issue. I assume you didn’t see any change in behaviour with r.Vulkan.SingleAllocationPerResource=1? Were you able to gather the output of the 2 console commands mentioned earlier?
Best regards.
[Attachment Removed]