I am going to venture a guess that there is something corrupt with the actual contact file.

What I ended up doing was exporting each category out as its own file, recreated the server and then reimported them back in.

While not the most elegant of solutions, it helped in my situation where my contacts were hosed to the point where anyone starting a new calendar file and flowing the server based contacts into the new calendar would crash the mac.

John Gibson
Administrator of a Now Contact / Up to Date Server
Praying heavily for the day that Nighthawk ships.
Mr. Green