Ubuntu 20.04 freeze on any display changes or unplugging external display

[omegatheta]

With DisplayLink 5.4.1, a 5.11.0-37-generic Linux kernel, Ubuntu 20.04 and Nvidia 470.63.01 drivers, changing display config by GUI or Xrandr or even unplugging external display causes system freeze and reboot is the only option. System won't boot if external monitors are connected. Nvidia settings says PRIME displays cant be controlled by it.
X.org X Server version 1.20.11
X Protocol Version 11, Revision 0
pixman 0.38.4

Also attaching the DL Support tool zip file.