How to handle heavy network traffic on FreeBSD (p2)

2011-11-05 / Saturday by WoOh

So we know the basics from the How to handle heavy web traffic on FreeBSD (part 1.), and it's time to go further. The basics are essencial but usually not enough for a production environment. We need more. Let's see what else could we do to improve the speed and decrease the loads.

3. vmstat, the supreme utility to figure out what you need to do

The best utility in the FreeBSD system is the vmstat, which can trace almost all performance related issues. Use like this:

# vmstat -z
ITEM                        SIZE     LIMIT      USED       FREE  REQUESTS  FAILURES
...
hostcache:                136,       15372,    13759,     1613,  13011725,  213476301
...

As you can see, there is a column called: "FAILURES". It's an indicator, which tells us: something is not well configured in our system. It usually means we have to increase a kernel variable via sysctl. The LIMIT and the USED column can give us a hint, how ofter does the server is reaching the limit of the variable. If the FAILURES column shows a very big number, you should immediately increase the related values in the /etc/sysctl.conf. In this example we can see the hostcache, which has a default value: 15372. That could be enough for most servers in the internet, but not for a heavy loaded server. Some variables cannot be set via sysctl.conf, so you have to put them in /boot/loader.conf.

Most of the time, you should tune these variables:

/etc/sysctl.conf
net.local.stream.sendspace
net.local.stream.recvspace
net.inet.tcp.local_slowstart_flightsize
net.inet.tcp.nolocaltimewait
net.inet.tcp.hostcache.expire
kern.maxfiles
kern.maxfilesperproc
kern.maxvnodes
net.inet.tcp.maxtcptw

/boot/loader.conf
kern.maxusers
kern.ipc.nmbclusters
kern.ipc.maxsockets
kern.ipc.maxsockbuf
net.inet.tcp.tcbhashsize
net.inet.tcp.hostcache.hashsize
net.inet.icmp.icmplim

The net.inet.tcp.maxtcptw is always a pain in the ass just as net.local.stream.sendspace. You always have to monitor your vmstat -z output and do the relevant changes in your setup. It could be good for today, but won't be enough for tomorrow (when you get more and more traffic, you'll hit the limits again and again)

4. PostgreSQL

When using PostgreSQL, the default config is not usable in a production enviroment. It has a very low max connection number and if you want to increase it, it'll hit the limits. PostgreSQL uses kern.ipc.* (semaphores) so the semaphore related variables should be increased. Here is a config of a server, which handles maximum 120 connections. Above 120 is not really supported by the PostgreSQL team, and they're right, because if you have 120+ connections, you should consider to optimize the queries to run faster, or build a cluster (with slony for example).

/boot/loader.conf
kern.ipc.semmni=1024
kern.ipc.semmns=2048
kern.ipc.semmnu=1024

/etc/sysctl.conf
kern.ipc.semmap=256
kern.ipc.shmall=32768
kern.ipc.shmmax=134217728
kern.ipc.somaxconn=16384
net.inet.tcp.recvspace=78840
net.inet.tcp.sendspace=78840
kern.ipc.shmmax=67108864
kern.ipc.shmmni=200
kern.ipc.shmseg=128
kern.ipc.semmni=70

After setting these variables, you have to reboot your box to activate them.

5. Nginx instead of Apache with FastCGI/php-cgi.

If you're ready with the tuning, but still have problems with the memory usage, you should consider switching to nginx from apache. Nginx uses a lot less memory than apache. I'd say 70% less memory even with php-cgi. Using this setup, you can track easily the problems, because you'll be able to check what's going on in a specific php-cgi process using procstat utility. So changing to nginx, you'll get memory, speed, stabiity. And you'll be able to detect the probems faster than before. Your web server won't crash or eat up your memory. You can segment your web pages with individual php-cgi processes. A bigger site can get a dedicated php-cgi process, while smaller sites can share a process. How to define more php-cgi backends to your nginx?

/usr/local/etc/nginx.conf
upstream mybackend {            
    server    127.0.0.1:9000;            
    server    127.0.0.1:9001;
}

This will do the work :)

In the next part, i'll show you a real life example, how to speed up your server. The max throughput was 30,000 registered AV / 5 min before the optimisation. After the optimisation, it is able to handle more than 700,000 registered AV / 5 min.