Rate limiting with nginx

I just love Nginx, it's amazing at how little memory it consumes and how well it performs, much of this is owed to nginx’s use of the event driven mode.

One of my teams repeated problems that come up is the simple fact that some users and bots are a little excessive of the servers.

 http {

limit_conn_zone  $binary_remote_addr zone=conn_limit_per_ip:50m;   limit_req_zone   $binary_remote_addr zone=req_limit_per_ip:50m rate=1r/s;

server {

location / {

limit_conn conn_limit_per_ip 10;             limit_req zone=req_limit_per_ip burst=10 nodelay;

}

}

}

The directives that count here: limit_conn_zone, limit_req_zone, limit_conn and limit_req.

We first use limit_req_zone to set up at least a rate limit zone, which will then be enabled by placing them inside specific nginx location directives.

We start by setting up our first zone named ‘default’, give it 50 megabytes of memory to track our sessions, and set a rate at 1 request per second.

We then implement it in the ‘/’ location, and give it a ‘burst’ of 10.

Every time that a bot exceed the rate of 1 request per second, they have to pay a token.  Once they’ve spent all of their tokens, they are given an HTTP 503 error message. 

503 means  the server is currently unavailable (because it is overloaded or down for maintenance). Generally, this is a temporary state.

As you experience an excessive bot you will see the following in your log

2016/09/01 10:06:29 [error] 109154#109154: *42450 limiting requests, excess:
10.195 by zone "req_limit_per_ip", client: ip.of.attacher, server: default,
request: "GET

*42450 means:

This is a connection number, also available as $connection.

109154#109154 means:

This is nginx worker PID (also available as $pid) and thread identifier.

10.195 means:

This is number of requests accumulated in the bucket.  If this 
number is more than burst defined (10 in our case), further 
request will be rejected.

Number of requests in the bucket is reduced according to the rate 
defined and current time, and may not be integer.  The ".195" 
means that an additional request will be allowed in about 195 
milliseconds assuming rate 1r/s.

You can get more information at the following location(s) :
http://www.checkupdown.com/status/E503.html
http://nginx.org/en/docs/http/ngx_http_limit_conn_module.html
http://nginx.org/en/docs/http/ngx_http_limit_req_module.html

HTH
Brent

Another way to calculate PHP5-FPM max_children

This is a slight update from my previous post, but I stumbled across the following command  / method to help calculate the value of "pm.max_children" for PHP5-FPM.

echo "pm.max_children = $(( $(awk '/MemTotal:/ { printf "%d\n", ($2*0.66) }' /proc/meminfo) / $(ps --no-headers -o "rss,cmd" -C php5-fpm | awk '{ sum+=$1 } END { printf ("%d\n", sum/NR) }') ))"

Now whats interesting is that the suggested value return, is a ridiculously high value, but I think what comes out of this, is that it illustrates how much more RAM you can give  PHP5-FPM.

But (un)fortunately real world exists, and there are other existing services in play (e.g. MySQL could be installed).

So if you use the above command, I highly suggest you adjust the PHP5-FPM value sparingly, and thereafter recheck your Munin / Cacti etc

HTH
Brent

Adjusting PHP5-FPM child processes (Apache)

Every now and then, on a clients server, we get the following dreaded message:


WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 4 idle, and 48 total children

So we will start off, by determining the non-swapped physical memory usage by each PHP5-FPM processes (notice its in kilo Bytes)

ps -ylC php5-fpm --sort:rss
ps --no-headers -o "rss,cmd" -C php5-fpm | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024,"M") }'

Thereafter I ran the following command to give me the total memory, of my current PHP5-FPM configuration.

ps -ylC php5-fpm --sort:rss | awk '!/RSS/ { s+=$8 } END { printf "%s\n", "Total memory used by PHP-FPM child processes: "; printf "%dM\n", s/1024 }'

In my case I still had plenty of RAM left.

Remember the appropriate value for pm.max_children can be calculated as:

pm.max_children = Total RAM of your server / Max child process size - in my case it was 67MB

pm.max_children = 70
pm.start_servers = 20
pm.min_spare_servers = 20
pm.max_spare_servers = 35
pm.max_requests = 500

Debian upgrade Wheezy to Jessie

Good day Guys

Here is my tip / howto to upgrade Debian Wheezy to Jessie.

I highly recommend first ensuring that your server is update.

So run:

# apt-get update

# apt-get upgrade

# apt-get dist-upgrade 

# apt-get -f install

Fix any outstanding issues. 

If all is good. You can start by changing you sources.list to reflect Jessie

sed -i 's/wheezy/jessie/g' /etc/apt/sources.list

sed -i 's/wheezy/jessie/g' /etc/apt/sources.list.d/*

The second sed is to ensure all other third pary repos reflect Jessie too.

There after you are good to go.

Run:

# rm -rf /etc/apt/preferences.d/*

# apt-get update

# apt-get upgrade

# apt-get dist-upgrade

# apt-get -f install 

This can take awhile.

After the upgrade it is recommended to reboot your system: 

# reboot

After your server has come up, for safety sake, run another 

# apt-get update

# apt-get upgrade
# apt-get -f install

You should have a new and shiny new OS, and you can verify, by using either 
hostnamectl or 
lsb_release -a

HTH
Brent

Insecure VPN?

Today I was reading that 90% of All SSL VPNs are Insecure.

• 77% of the SSL VPNs still use the insecure SSLv3 or the even less secure SSLv2
• 76% use an untrusted SSL certificate
• 74% have an insecure SHA-1 signature
• 41% use an insecure key length of 1024 for RSA certificates
• 10% rely on versions of SSL that are still vulnerable to the  Heartbleed attack

If you want to test if your VPN supports SS3, just run the following:

 openssl s_client -connect <server>:<port> -ssl3

If the connection succeeds, sslv3 is enabled. If it fails, it is disabled.

HTH
Brent

A tip to speed up Mozilla Firefox

Sometime some domains Firefox needs to resolve each domain name, one for an IPv4 address and once for an IPv6 address.

This results in lots of DNS requests, slowing down your web access.

If you are like 99.999% of the population without IPv6 access, translating domain names in IPv6 addresses is useless.

Try disabling this functionality, by typing about:config into the address bar.

Type ipv6 into the search bar and toggle network.dns.disableIPv6 to true.

HTH
Brent

Running a local Debian mirror (i386 and amd64 only)

For a while now, I have been running a Debian mirror. When you manage as many hosts like I am part of a team do, you tend to think about how it all negatively affect the upstream providers, and really just the community in general.

So with out further ado, here is my take and steps on setting up a mirror, using Raphael Geissert's tool ftpsync.

First you need to create a user and ensure the switch to that user.

sudo adduser  \
--system \
--shell /bin/bash \
--gecos 'User for FTPsync' \
--group \
--disabled-password \
ftpsync

sudo  -i -u ftpsync

git clone https://github.com/rgeissert/ftpsync /home/ftpsync/

Next on the list if to create the configs. Generally people just copy sample config, but there is quite a few options, but you only need a hand few, so therefore I am just going to share what I have.

cat ~/etc/ftpsync.conf
MIRRORNAME=`hostname -f`
TO="/var/www/mirror/debian"
RSYNC_PATH="debian"
RSYNC_HOST=ftp.nl.debian.org
LOGDIR="${BASEDIR}/log"
ARCH_EXCLUDE="alpha arm arm64 armel armhf hppa hurd-i386 ia64 kfreebsd-amd64 kfreebsd-i386 m68k mipsel mips powerpc s390 s390x sh sparc source ppc64el"
MAILTO="your@addressgoes.here"

Next on the list is to have a cron entry to kick off you new shiny mirroring tool. And for that you can just crontab -e , as user ftpsync.

And then add:

5 * * * * /home/ftpsync/bin/ftpsync sync:all >/dev/null 2>&1

And that’s that. You now just need to sit and watch the location on where you defined your $TO variable grow.

One thing I would like too add, at the time of writing, mirroring just i386 and amd64 is using on my VM 373Gigs (Hence the reason why I only mirror these two architectures). Show maybe just make sure you enough space allocated.

Two last parts
You then need to install and configure nginx or apache to offer and share the docroute of '/var/www/mirror/'.
Next you need to edit  (or create)

echo 'deb http://your.mirror.co.za/debian/ wheezy main contrib non-free' > /etc/apt/sources.list.d/debian.list

HTH
Brent

P.s. Remember there too are other repos. i.e.

backports
security
volatile etc