Not sure about where to find and revert all left-over changes applied by the agent, but since you mention `xorg.conf`, you might as well start there. I suggest you check out the `-configure` switch to `Xorg`.
	Code:
	-configure
               When this option is specified, the Xorg server loads all  video
               driver  modules,  probes for available hardware, and writes out
               an initial xorg.conf(5) file based on what was detected.   This
               option  currently  has  some problems on some platforms, but in
               most cases it is a good way to bootstrap the configuration pro-
               cess.   This option is only available when the server is run as
               root (i.e, with real-uid 0).
 You may also want to look at the CENTOS spec file and patches for the X server to see if they do anything fancier out-of-the-box.
Good luck.
PS: Of course, you can also (a) reinstall the guest or (b) give in and reinstall the agent using the CURSES interface without X, but these are things you're most likely trying to avoid.