I could be wrong, but I believe CPU is more influential. Graphics cards are predominantly for 3D work. They deal with vectors. Video is (usually) flat, so the file is read, decoded and displayed. For playback, the CPU certainly does the lions share of the work, for editing, if you start copy-pasting clips left, right and centre, then RAM and disk speed will come into play.
Never done much video on anything, but I own a dual 1GHz G4, with 2GB RAM and an nVidia GeForce FX5200. I only ever splice things together in QT Pro, but it doesn't feel all that quick to me. Sometimes I am copy-pasting up to 1GB at a time though.
If you I were to fit a SATA card and boot drive to my G4, it would probably beat the iMac quite easily. But only you know if you can stretch to a pile of RAM, a new HD and a SATA card to go with the G4.