Fri, 30 Jun 2006

ApacheCon '06 : httpd for 50,000 users

Colm MacCarthaigh (HEANet)

Colm presented this talk to a room bursting with people; extra chairs were brought in and 50,000 users is not a lot in terms of computing power today, so how do we make the webserver handle this many, and more.

listen() + fork() / thread_create
poll() / select() / etc...
accept()
stat(), readdr(), stat()
read() + write() / sendfile()
close()

Generally, we find that the bottleneck in serving files fast is the time spent on system calls. So really, the question, "Which is the fastest webserver?" is not really relevant - it's far more dependant on the way that the system is configured. We're limited by network latency / jitter, storage performance and kernel performance. Jumbo frames can help in some places, but cannot realistically be deployed in all cases.

Be sure to use lots of benchmarking tools, to see if you are getting measurable performance improvements. Apache Bench (ab) shipping with Apache 2.2 should be used, as the ab that ships with previous versions has limitations and you may hit the limit of what ab can handle before hitting the limits of what your webserver can handle. Benchmark with the same files (this seems obvious, but is stated here anyway :-) ) Autobench is also a good product and produces nice looking graphs.

IoZone, Postmark, Bonnie++ and dbench are good tools for testing filesystems. Benchmarking a VM is a much harder thing to do. Compiling a kernel or similar gives some measurements, but these are difficult to compare to anything useful. HEANet use the deadline scheduler, and use a custom script to try and get metrics from the VM.

If you have .htaccess file enabled, Apache checks the requested directory, and every parent directory is checked for .htaccess files - for every request. This can really slow things down.

Use mod_cache to cache local content. HEANet put an XFS (fastest file system) on RAID0 array of 15k RPM disks on a machine running mod_cache. They see something like only 30~40GB of unique content downloaded per day, so this works extremely fast.

We found that compiling, optimising for size (-Os) gives the best performance, since memory is more expensive for us than CPU time.

Mount the web data with noatime since that saves a write for every hit. This can do anything up to double your filesystem performance. Increase the i-cache size. Try different block sizes on the filesystem.

To build a fast webserver, it largely boils down to making a system with lots of memory, and lots of fast disk. Opteron gave the fastest throughput, though Niagara was only about 10% slower and there's a new patch which may make it lots faster. Remember, keep benchmarking and use the configuration that works the best for you.
posted at: 14:19 | path: /technical | permanent link to this entry