nvidia-drm: don't accumulate override EDID modes in detect (OOM with drm.edid_firmware)

[gianmarcotoso]

Summary

With an override/firmware EDID set on a connector (drm.edid_firmware= or debugfs edid_override), every detect on that connector while it is disconnected leaves the override EDID's modes in connector->probed_modes. Each later probe doubles the number of CEA modes in that list. A compositor re-probing on hotplug events (for example, a DisplayPort monitor going to sleep) can use up all memory within about a minute and hang the system.

This PR drops stale probed modes before nvidia-drm calls drm_edid_override_connector_update() from detect.

Root cause

On kernels without drm_connector::override_edid (the !NV_DRM_CONNECTOR_HAS_OVERRIDE_EDID path), __nv_drm_detect_encoder() calls drm_edid_override_connector_update() to get the override EDID and pass it to NVKMS. The kernel designed that helper for .get_modes(), and it also adds the EDID's modes to connector->probed_modes.

Normally drm_helper_probe_single_connector_modes() drains probed_modes through drm_connector_list_update(). But when detect reports connector_status_disconnected, the helper jumps to exit before get_modes() and before drm_connector_list_update(), so the modes stay in the list.

The next detect calls the helper again. drm_edid_connector_add_modes() → add_alternate_cea_modes() walks the whole probed_modes list and adds an alternate-clock copy (60 / 59.94 Hz) of every CEA mode it finds, including the leftovers. The CEA mode count therefore roughly doubles on every probe. With a common EDID that has a CTA extension (VICs 3, 4, 16, 63, 97 in my case), about 25 probes are enough to fill 30 GB of kmalloc-128 (sizeof(struct drm_display_mode)).

detect is also called once per possible encoder, so a single probe can call the helper more than once.

Fix

Before calling drm_edid_override_connector_update() in __nv_drm_detect_encoder(), free whatever is in connector->probed_modes. This is safe:

• In the probe helper, detect runs before get_modes() and drm_connector_list_update(), so at that point probed_modes only holds leftovers from earlier detect calls.
• The modes nvidia-drm reports are built in nv_drm_connector_get_modes() from nvKms->getDisplayMode(). NVKMS already gets the override EDID through pDetectParams->overrideEdid, so dropping these duplicates changes nothing visible to userspace.
• The NV_DRM_CONNECTOR_HAS_OVERRIDE_EDID path never calls the helper and is unchanged.

How it showed up

• RTX 3090 (GA102), open kernel modules 615.71.09, Linux 7.2.9 (Arch), GNOME Shell / mutter 51 on Wayland.
• Samsung Odyssey G93SC over DisplayPort, with drm.edid_firmware=DP-1:edid/g93sc.bin,DP-2:edid/g93sc.bin,DP-3:edid/g93sc.bin.
• When the monitor went to sleep (HPD low, DPCONN> Zombie? : 1, Lost device), mutter re-probed the connectors. Within ~60 s the system was unresponsive and the kernel log showed:

KMS thread: page allocation failure: order:0, mode:0xc0de0(GFP_KERNEL|__GFP_HIGH|__GFP_ZERO|__GFP_COMP|__GFP_NOMEMALLOC)
...
 __kmalloc_cache_noprof+0x35d/0x470
 drm_mode_duplicate+0x23/0xa0
 _drm_edid_connector_add_modes.part.0+0xb0d/0x12d0
 drm_edid_connector_add_modes+0x41/0x70
 drm_edid_override_connector_update+0x9a/0xa0
 __nv_drm_connector_detect_internal+0x14b/0x350 [nvidia_drm]
 drm_helper_probe_single_connector_modes+0x587/0x6a0
 drm_mode_getconnector+0x363/0x520
...
Mem-Info:
 ... slab_unreclaimable:7533319 ...
Unreclaimable slab info:
Name                      Used          Total
kmalloc-128         30105088KB   30105088KB

followed by OOM kills of every user process.

Reproduction

The connector does not need a monitor attached. It only needs an override EDID with CTA modes and repeated probes while it is disconnected:

# boot with e.g. drm.edid_firmware=DP-1:edid/<some-edid-with-cta-ext>.bin and nothing on DP-1
for i in $(seq 10); do echo detect | sudo tee /sys/class/drm/card1-DP-1/status >/dev/null; done
sudo awk '$1=="kmalloc-128"{print $2}' /proc/slabinfo   # grows ~2x per probe without the fix

Do not run many more iterations on an unpatched driver: the growth is exponential.

Related

• nvidia-drm: trigger connector detect on hotplug events #1127 switches the hotplug work to drm_helper_hpd_irq_event(), which runs detect on every HPD edge. With that change, the growth described here would happen without any userspace probing, so this fix is a prerequisite for it on systems that use an override EDID.

Testing

• With this change on 615.71.09: 15 forced probes on a disconnected connector with an override EDID leave kmalloc-128 unchanged (9556 objects before and after).
• Two DisplayPort monitor sleep/wake cycles: no memory growth (kmalloc-128 stays around 1 MB), and the override EDID modes are still reported afterwards.
• The patch applies cleanly on main (615.78.08).

[CLAassistant]

All committers have signed the CLA.