I've now updated the OS X to Big Sur 11.0.1 as apparently there's a problem with factory restore that can pretty much brick brand new M1 devices. I wanted to reload my MBA from scratch to reinstall DL drivers again (without the add-on) but one person has reported that a workaround after updating to 11.0.1 from Apple still doesn't seem to resolve the problem so can't really try it atm.
Source:
https://support.apple.com/en-us/HT211983
If you have another Mac with Catalina and USB-C then there is a way to restore bricked M1 otherwise it's a lottery and a trip to Apple store in COVID times or sending back for replacement.