Front USB ports all weird

My front USB ports occasionally decide that they don't want to recognize anything new that gets connected to them. This is an issue since they're on the front where you swap stuff out all the time. Restart fixes it but I would rather not restart every time I need to connect a different hard drive or something.