I tried to summarize the status, cause, workarounds and fixes for this issue over here.

Unfortunately, I think I mentioned this before, graphics drivers cannot be dynamically unloaded on Mac OS X.
It's technically possible to uninstall the driver at every shut down/restart and then install and load it when needed. But then the only way to avoid the issue would be to reboot so I don't see a big benefit in exchange for all this fiddling.

Actually, even if the toggle switch in the System Preferences uninstalled/reinstalled the driver and then prompted for a reboot, there is still a benefit. It's all about user experience.

If there were a System Preferences app for DisplayLink, the steps would be roughly:
1. Open System Preferences
2. Open DisplayLink
3. Toggle on/off DisplayLink
4. Enter password and press enter
5. Restart

Currently, to install, the steps are:
1. Open folder where .dmg is saved
2. Double click DisplayLink installer .dmg to mount it
3. Double click DisplayLink Software Installer.pkg
4. Click Continue (This package will run a program to determine if the software can be installed.)
5. Click Continue (Welcome)
6. Click Continue (Software License Agreement)
7. Click Agree (Software License Agreement)
7. Click Install
8. Enter password and press enter
9. Click Continue installation (prompt to restart after installing)
10. Click Restart

So, instead of 10 painful steps, there would be 5 relatively easy steps.

Also, I would be able to remove the .dmg file from my Downloads folder.

I just installed the latest OSX updates and now my display adapter works all the time, I can unplug it, plug it in, sleep my mac, anything and no more blank/snowy screen which I never could before (had to uninstall the displaylink driver).