I recently had a problem where a website runs normally all month, but after an “update notification Email” was sent, the website ground to a halt.

The first useful command I found was `free` which displays the memory status.

             total       used       free     shared    buffers     cached
Mem:       8176976    1245760    6931216      24060     190016     696392
-/+ buffers/cache:     359352    7817624
Swap:      4194300          0    4194300

This is the current status, which shows a 0 in the swap space usage. This is good! At the time of my problem, I was showing 200000 swap in use.

Digging deeper, my /etc/apache2/mods-enabled/mpm_prefork.conf file had the following

        StartServers              5
        MinSpareServers           5
        MaxSpareServers          10
        MaxRequestWorkers       256
        MaxConnectionsPerChild    0

Pretty standard I thought, but then I found the following script on http://www.stephenwalker.com/2013/09/finding-apache-average-memory-usage-on-debian-wheezy/

sudo ps -ef | grep apache2 | grep -v ^$USER | awk '{ print $2 '} | xargs sudo pmap -d | grep ^mapped: | awk '{ print $4 }' | cut -dK -f1 | awk '{SUM += $1} END { print SUM/NR }'

I ran this to discover my apache process’s were averaging about 170Mb. My 4Gb server could only handle 4096/170 = 24 simultaneous requests in memory. This is why it was disk swapping and grinding to a halt.

I changed MaxRequestWorkers to 20, and this stopped the disk swapping, and seemingly fixed the website speed.

However, a few days later I had the website grinding to a halt again. Further investigation, using

sudo netstat -plan | grep 443

showed I had about 300 users waiting for requests. It was clear that my 20 MaxRequestsWorkers was not enough for my peaks. My CPU load was running at 0.4 so processing power was not the bottle neck. Searching for a solution, I began to realise that 170Mb was rather excessive as others were saying their processes run at 10-20Mb.

To solve the bottleneck issue, I would have to reduce the memory of the apache process in order to allow more simultaneous connections. The standard way is to remove unwanted modules. I removed the mysql php extensions and removed the apache module mod_access_compat. This save 2Mb, not curing my problem but allowing me a minimal configuration once solved.

mod_access_compat

This module is added to apache 2.4 for backwards compatibility in the apache configurations files. It is a simple task though to update your configurations thus removing the need for the module. The following is the main change I had to perform.

Order allow,deny
Allow from all

#change to 
Require all granted

The final solution

I looked at the processor usage on another linux box, and found that was running at 10Mb. This finally led me to the answer. I ran phpinfo() on both machines, and compared them side by side.

I discovered that the machine with the problem had the php command get_browser() enabled. This involves downloading a file called browscap.ini which contains information on all the browsers. Although only 17Mb, each process loads the information into memory taking up 150Mb. Dont ask me why!

Removing this option took my process down to 8Mb !!!

I can now handle 256 (a system default) without the need for disk swapping.