You're right: with the motherboard fried, the VGA hookup isn't going to work. However, I do have a couple of ideas, assuming you're at least a little knowledgeable of working with the insides of a computer. (All involve gutting the machine, BTW - if you're unprepared and don't desire learning how to do this, just don't.)
#1.) You could replace the motherboard with a new one. There's two directions which could work here, given the space: buy a Macbook/Macbook Pro/iMac Motherboard to replace the motherboard in there (don't do anything older than G5, it's just not worth it), and do your homework to see where the hookups go to. The other direction is to buy and build a Hackintosh and buy some hookups to convert the LCD to something you can plug into your motherboard or Graphics Card. In either case, once you get done, you can buy an input Device (via USB) to hook your XBox 360 to.
#2.) You can look online for used and new LCD parts and converters to convert that LCD screen to something the XBox 360 can plug into. This way is more difficult and may be more expensive in the long run, however, especially if you don't know what you're doing.
In either case, you're going to have to do your homework. There are sites online that will have specific info Apple won't supply on their web site (after all, they don't want to make it easy for people to modify their hardware, especially in the way you want to modify this machine.) Chances are good that someone may have done something similar and has posted something about it - even if they didn't, though, there should be enough info for someone with a little electronics knowledge to get themselves into trouble on this.
My opinion: Find a decent Micro-ATX Intel motherboard (not a motherboard FROM Intel, but one compatible with their chips), an LCD converter board and a decent Graphics card, build it into a Hackintosh. (If you're lucky to have one with the webcame attachment, try to hack a cheap USB webcame to use it as well.) Once you're done, get a USB TV-IN Device and run your 360 into it. Good luck!