It sounds like you already have your KVM, but if you would rather not invest in a mixer, you may want to consider getting a KVM with audio. They seem pretty common these days.
Anyone seen a KVM capable of handling 5.1 sound? Mine has the single green (front stereo) jack, but it would be nice if I could switch all the channels and retain the Surround capability.
