I get a message saying that rivfb must be disabled in the kernel
Actually, it says that rivafb must not be enabled, not quite the same thing. It seems that rivafb is a loadable driver, not usually loaded unless you have that card. I've been using the same NVIDIA driver on the same Fedora distro with no problems, presumably because Linux never feels the need to load the rivafb driver.
Unlike some Linux-heads, I do not relish the notion of building my own kernel over every little thing. It pleases me when these things work as well as the nvidia driver did.
But if I run "X" from the command line in runlevel 3 to test, it works but then my prompt doesn't come back when I exit X.
Try "Ctl-Alt-F1" after you exit X. Text mode Linux runs in multiple terminals; Fedora seems to set up 6 by default. X goes into the seventh terminal, but leaves you there when you exit.