[saudor]

There are guides out there that should force installation of OS X on unsupported hardware. After all, I even got Mavericks running on an AMD windows PC. I think there is an ongoing thread on macrumors concerning the mac Pro 2006 and El Capitan. (Mavericks should definitely be possible)

The hackintosh forums are a good resource to get non Mac graphics card working with OSx. (These tend to be cheaper). I hear the ATI 5770 (I picked one up for $20 used a year ago) is pretty easy as macs actually use the card.

Do you have access to a PC that you could place the card in? Or even another Mac. The former might prove difficult if the firmware on the card pertains to a mac

Do you get the same symptoms from a snow Leopard installer? But sounds like a graphics issue.