Installing Displaylink drivers on Ubuntu 22.04 breaks gdm3/Xserver

[user_12344321]

Originally installing the DisplayLink drivers worked fine.

After some time of using it suddenly (supposedly after some unattended upgrade), Ubuntu would boot into a blinking _ and not start the GUI.

After some debugging, I figured out:
* the issue exists independent of whether displaylink display is attached or not
* the issue only exists when displaylink drivers are installed
* the computer boots as expected when they are uninstalled
* the issue persists independent of the window manager gdm3 and lightdm won't start
* the issue exists for 5.5, 5.6 and 5.6.1 of the displaylink drivers
* the issue exists for both Xserver and Wayland, i.e. changing WaylandEnable=false in /etc/gdm3/custom.conf

Ubuntu 22.04 with 5.15.0-58-generic, Thinkpad laptop with and ryzen 4000 series and gpu

I attached a lot of logs and the output from the debug info tool while having set lightdm as the window manager.
Journalctl shows lightdm.service and gpu-manager.service crashing.
I attached the logs of them.