It won't hurt anything. They wouldn't sell KVM switches that do the same thing if it did.
The only caveat is that maybe the video card won't correctly recognize a new monitor being plugged in unless you reboot the machine.
If you don't take your time and then end up bending a connector it may cause a problem, but if it did it would do the same thing when you powered back up anyway.