The speed problem is mostly with the hosting company's server. The main sites are slow as heck as well (about the same as the forums). Part of the problem is that Drupal is just an unruly mess when it comes to server databases.

When I ran The Graphic Mac on Drupal I had much the same problem. When I took the exact same posts, images, and ads and ported it all over to WordPress, the speed increase was 10-fold at minimum - real night & day comparison.

However, WordPress is not nearly as flexible as Drupal - so it's a trade-off.
The Graphic Mac - Tips, reviews & more on all things OSX & graphic design.