Remove all the modules you don't need.
You missed my point. The point is for the PHP packages support all the extensions, but separated and need not be loaded except when user requested it via php.ini, e.g:
extension=mysql.so extension=pgsql.so ;extension=gd.so ;extension=imap.so ;etc.
Also consider using a cache system, e.g. php-mmcache
Won't work (at least optimally) in multiuser CGI environment e.g. in hosting environment.
-- dave