The Mac motherboard are similar in size to atx. You can use that size or smaller, like mini or micro atx. Careful on the set up of the board though as that optical drive bay seems to always be in the way!
You could try, but Im thinking no. Mac board in those cases are around 10x11.5". You could push it a little farther, but you will run into problems with pci slots, etc. You could put the board on the inside of the case (not on the swing down door. That could work with some cutting. A new cooling design would be needed then.