How to handle heavy web traffic on FreeBSD (part 1.)

2011-11-03 / Thursday by WoOh

The question often comes up: What should I do, if my server has a heavy traffic and it rejects the connection. For example the web servers (nginx, apache) throw white page, but the server load is quite low. There are no error logs at all and you don't know what to do. FreeBSD out of the box is not configured for heavy traffic. It's time to fine-tuning your server. Let's see what you should do.

I was looking for several documents and HOWTOs about the fine-tuning of my heavy-loaded FreeBSD servers. Some of them do approximately 1 million AV / 5 min, so they do more than 200-300Mbit/sec, but some of them do only 2500-4000 PI / 2 min. But I found nothing. Everything was about:

You have to recompile the whole kernel, to enable this or this to the kernel...

But this was bullshit. No need to recompile the kernel it is good as it is. You only have to tune some sysctl variables. So let's start with the basics.

1. net.inet.icmp.icmplim

Look at /var/log/messages for the following error messages

Limiting closed port RST response from 205 to 200 packets/sec

If you find these messages, it means: your users get white page when accessing your web page at the time you get these errors in the messages log. But what is this message means? It's all about a self-protecting mechanism in FreeBSD. It controls the maximum number of ICMP "Unreachable" and also TCP RST packets that will be sent back every second. You can get the current value via sysctl:

# sysctl net.inet.icmp.icmplim

It has 200 as an initial value (Out of the box). If your server is under heavy load you should increase this value to:

/etc/sysctl.conf:
net.inet.icmp.icmplim=8192

My FreeBSD boxes have 80,000 - 120,000 concurent connections, so the value 200 is very "weak" for these boxes. But you should tune it yourself. Always increase just a little bit 1028, 2028 and so on, and stop at the value when you're not getting any errors.

2. limit states / timeout interval (packet filter)

If you're working with a firewall which acts as a round-robin balancer, you'll find yourself in such a situation: Your servers behind the firewall are bored, no error messages, no bottlenecks, but somehow your connection to your server is slow. Sometimes you have to wait 2-3 sec when you type:

# telnet yourserver.ip 80
Trying yourserver.ip...
Connected to yourserver.ip.
Escape character is '^]'.

So you feel: something is not ok, but you don't have a clue what's going on. You should check your state table via:

# pfctl -si | grep 'current entries'
current entries                    42464

You can check what is your limit via:

# pfctl -sm
states        hard limit   1000

You will never go over the hard limit. So if you see that your current entries are hitting the hard limit, you should consider to increase this value to a higher value for example: 2000 or ever more. (I use 120000 on my FreeBSD boxes). It can be done via the /etc/pf.conf

pf.conf:
set limit states 120000
set timeout interval 7
pfctl -f /etc/pf.conf

As you can see it has another config variable: timeout interval which tells to the packet filter how many seconds should it keep the entries in the table. After you set up these variables, your system will work better than before, but won't solve everything. I'm gonna write about the tricks in my next post.