If the IPs are dynamically assigned, such an attempt is pointless. What you can do is to use portknocking. This has been suggested and discussed controversial recently here on the list.
Controversial is correct. From what I've read, portknocking is useless, worse than useless, really, since it induces an entirely unjustified sense of security. I will never use it.
