Thursday, September 29, 2016

Puppetserver refuses to start with /tmp mounted noexec.

Im in the process of investigating and moving our code base to Puppet 4.

I came across something very interesting today, in that Puppet 4 does not like /tmp to be mounted noexec.

Now, we mount /tmp with nosuid and noexec as per the recommendation in the Securing Debian Manual.

After much hours of trouble shooting I came across "Puppet Server run issue when /tmp/ directory mounted noexec" (And I dont know why I didnt read this in the first place.)

As per the recommendation, you basically need to add to /etc/default/puppetserver.


JAVA_ARGS="-Xms2g -Xmx2g -Djava.io.tmpdir=/var/tmp"

HTH

Brent

Tuesday, September 20, 2016

Puppetdb dies at random / silently

If you ever have it that the connection to puppet-db is refused or you find the daemon died.

Even if trawling through the puppetdb logs reveal nothing.

It might be that PuppetDB is running out of heap? Check
/var/log/puppetdb for the presence of the  file 'puppetdb-oom.hprof' for an indication that this is happening.

You may find the following to help.
https://docs.puppet.com/puppetdb/2.2/configure.html#configuring-the-java-heap-size

HTH
Brent

Friday, August 26, 2016

Rate limiting with nginx

I just love Nginx, it's amazing at how little memory it consumes and how well it performs, much of this is owed to nginx’s use of the event driven mode.

One of my teams repeated problems that come up is the simple fact that some users and bots are a little excessive of the servers.

 http {

  limit_conn_zone  $binary_remote_addr zone=conn_limit_per_ip:50m;
  limit_req_zone   $binary_remote_addr zone=req_limit_per_ip:50m rate=1r/s;


    server {

        location / {

            limit_conn conn_limit_per_ip 10;
            limit_req zone=req_limit_per_ip burst=10 nodelay;

        }

     }

}
 


The directives that count here: limit_conn_zone, limit_req_zone, limit_conn and limit_req.

We first use limit_req_zone to set up at least a rate limit zone, which will then be enabled by placing them inside specific nginx location directives.

We start by setting up our first zone named ‘default’, give it 50 megabytes of memory to track our sessions, and set a rate at 1 request per second.

We then implement it in the ‘/’ location, and give it a ‘burst’ of 10.

Every time that a bot exceed the rate of 1 request per second, they have to pay a token.  Once they’ve spent all of their tokens, they are given an HTTP 503 error message. 

503 means  the server is currently unavailable (because it is overloaded or down for maintenance). Generally, this is a temporary state.

As you experience an excessive bot you will see the following in your log

2016/09/01 10:06:29 [error] 109154#109154: *42450 limiting requests, excess:
10.195 by zone "req_limit_per_ip", client: ip.of.attacher, server: default,
request: "GET
 
*42450 means:
This is a connection number, also available as $connection.
 
109154#109154 means:
This is nginx worker PID (also available as $pid) and thread identifier.
 
10.195 means:
This is number of requests accumulated in the bucket.  If this 
number is more than burst defined (10 in our case), further 
request will be rejected.
 
Number of requests in the bucket is reduced according to the rate 
defined and current time, and may not be integer.  The ".195" 
means that an additional request will be allowed in about 195 
milliseconds assuming rate 1r/s.

You can get more information at the following location(s) :
http://www.checkupdown.com/status/E503.html
http://nginx.org/en/docs/http/ngx_http_limit_conn_module.html
http://nginx.org/en/docs/http/ngx_http_limit_req_module.html

HTH
Brent

Monday, June 6, 2016

Another way to calculate PHP5-FPM max_children

This is a slight update from my previous post, but I stumbled across the following command  / method to help calculate the value of "pm.max_children" for PHP5-FPM.

echo "pm.max_children = $(( $(awk '/MemTotal:/ { printf "%d\n", ($2*0.66) }' /proc/meminfo) / $(ps --no-headers -o "rss,cmd" -C php5-fpm | awk '{ sum+=$1 } END { printf ("%d\n", sum/NR) }') ))"

Now whats interesting is that the suggested value return, is a ridiculously high value, but I think what comes out of this, is that it illustrates how much more RAM you can give  PHP5-FPM.

But (un)fortunately real world exists, and there are other existing services in play (e.g. MySQL could be installed).

So if you use the above command, I highly suggest you adjust the PHP5-FPM value sparingly, and thereafter recheck your Munin / Cacti etc

HTH
Brent

Adjusting PHP5-FPM child processes (Apache)

Every now and then, on a clients server, we get the following dreaded message:


WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 4 idle, and 48 total children

So we will start off, by determining the non-swapped physical memory usage by each PHP5-FPM processes (notice its in kilo Bytes)

ps -ylC php5-fpm --sort:rss
ps --no-headers -o "rss,cmd" -C php5-fpm | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024,"M") }'

Thereafter I ran the following command to give me the total memory, of my current PHP5-FPM configuration.

ps -ylC php5-fpm --sort:rss | awk '!/RSS/ { s+=$8 } END { printf "%s\n", "Total memory used by PHP-FPM child processes: "; printf "%dM\n", s/1024 }'

In my case I still had plenty of RAM left.

Remember the appropriate value for pm.max_children can be calculated as:

pm.max_children = Total RAM of your server / Max child process size - in my case it was 67MB


pm.max_children = 70
pm.start_servers = 20
pm.min_spare_servers = 20
pm.max_spare_servers = 35
pm.max_requests = 500

Thursday, June 2, 2016

Debian upgrade Wheezy to Jessie

Good day Guys

Here is my tip / howto to upgrade Debian Wheezy to Jessie.

I highly recommend first ensuring that your server is update.

So run:

# apt-get update
# apt-get upgrade
# apt-get dist-upgrade 
# apt-get -f install
 
Fix any outstanding issues. 

 
If all is good. You can start by changing you sources.list to reflect Jessie
 
sed -i 's/wheezy/jessie/g' /etc/apt/sources.list
sed -i 's/wheezy/jessie/g' /etc/apt/sources.list.d/*
 
The second sed is to ensure all other third pary repos reflect Jessie too.
 
There after you are good to go.
 
Run:
# rm -rf /etc/apt/preferences.d/*
# apt-get update
# apt-get upgrade
# apt-get dist-upgrade
# apt-get -f install 

This can take awhile.

After the upgrade it is recommended to reboot your system: 
 
# reboot
 
After your server has come up, for safety sake, run another 

# apt-get update
# apt-get upgrade
# apt-get -f install
You should have a new and shiny new OS, and you can verify, by using either 
hostnamectl or 
lsb_release -a

HTH
Brent

Thursday, April 21, 2016

Insecure VPN?

Today I was reading that 90% of All SSL VPNs are Insecure.

  • 77% of the SSL VPNs still use the insecure SSLv3 or the even less secure SSLv2
  • 76% use an untrusted SSL certificate
  • 74% have an insecure SHA-1 signature
  • 41% use an insecure key length of 1024 for RSA certificates
  • 10% rely on versions of SSL that are still vulnerable to the  Heartbleed attack
If you want to test if your VPN supports SS3, just run the following:

 openssl s_client -connect <server>:<port> -ssl3

If the connection succeeds, sslv3 is enabled. If it fails, it is disabled.

HTH
Brent

Tuesday, March 29, 2016

A tip to speed up Mozilla Firefox

Sometime some domains Firefox needs to resolve each domain name, one for an IPv4 address and once for an IPv6 address.

This results in lots of DNS requests, slowing down your web access.

If you are like 99.999% of the population without IPv6 access, translating domain names in IPv6 addresses is useless.

Try disabling this functionality, by typing about:config into the address bar.

Type ipv6 into the search bar and toggle network.dns.disableIPv6 to true.

HTH
Brent