Although my site doesn’t really get much traffic, I tried to improve the site response.
The best way I found was to cache your site using the WP-Cache-2 Plugin by Ricardo Galli.
For a blog site, most contents usually won’t change very often. Even if you are really energetic and passionate, one post a day would be probably the most you can do.
However, the comments on your post can change much more frequently. If you set the cache to expire in 1 hour, new post or new comments in that hour would not show up because the old cache is still there. I guess this is still a bug for the plugin, but should be able to get fixed.
Other than the bug, the caching solution worked very well. I used Apache Benchmark (ab) tool to test the speed before and after enabling the caching plugin. The speed boost is almost 4 times with caching (from 800ms/request to 200ms/request)!
If you can bear with the cache expiration bug, this plugin is highly recommended for a busy WP site!