I'd recompile things like Video/Audio applications that REQUIRE performance. Perhaps X as well. Other than that ... nope.
For what it's worth, I actually ran benchmarks on X performance, using x11perf -all, for default RH9 build versus rebuilt with --target ahtlon (Thunderbird 1.4 cpu). The result? Absolutely no difference whatsoever. Variations of about 1-3%, just as often up and down.
But YMMV, I guess.
/Peter