Can't you do the installation remotely? Having just the X libs there isn't as big of a risk as having an actual setuid-root X server.
quite right actually - pump X over to a remote X server does work but i'd still like 1 day to be able to install oracle with just some config files or someother such like