DisplayLink does not "pass the video through" - this is how it works:
https://www.synaptics.com/products/d.../corporate/how
But basically, some CPU/GPU resources are used by the DisplayLink driver. Try Windowed or Borderless Windowed mode - it could work better.