I'm dealing with the same issue on Ubuntu 20.04 with DisplayLink 5.4.0-55.153. I have also applied the official
xserver-xorg-core fix from the DisplayLink team. I have not tried the fix that @gavine99 recommended because no "cursor_events" file exists in any of my "/sys/devices/platform/evdi.%/" directories.
The issue does not appear in screen recordings, but I can take a camera recording (gross) if someone requests it.