I'm paranoid so I do not allow internet access that way. However, in the past I did. I did some of the stuff mentioned above. Limited the users who could use it then on top of that I used my firewall (Iptables) to restrict IPs. My purpose was to log in from work to do some stuff so I knew the IP range I'd be coming from. So basically to use ssh you'd have to be on my user list and be logging in from one of the approved IPs.
