Vagrant maintenace i.e. Box clean up

I use vagrant alot,

If you do not prune and / or removed unsed box you can quiet easily see yourself waisting space.

Case in point.

bclark@bclark:~$ du -sh .vagrant.d/boxes/
5.7G    .vagrant.d/boxes/


If you want to see what boxes you have:

vagrant box list

i.e.
bclark@bclark:~$ vagrant box list
debian/stretch64 (virtualbox, 9.5.0)
debian/stretch64 (virtualbox, 9.7.0)
debian/stretch64 (virtualbox, 9.8.0)
debian/stretch64 (virtualbox, 9.9.0)
ubuntu/bionic64  (virtualbox, 20190212.1.0)
ubuntu/trusty64  (virtualbox, 20180814.0.0)
ubuntu/xenial64  (virtualbox, 20190325.0.0)


To clean up you can use:

vagrant box remove debian/stretch64 --box-version 9.5.0

HTH
Brent

Modsecuritys upload / scanning testing

If you ever need to test Modsecuritys upload / scanning function, here a method.

curl -k -i -F 'upload=@"//tmp/bad.txt"' https://$domain/


HTH
Brent

Easiest way to scp files from host to Vagrant guest (or visa versa)

If you running a vagrant instance, and you forgot to set up file sharing, but *now* you need to copy a file off to a guest, or on a guest.

Here is a method, I used to save my bacon.

Try this:

vagrant plugin install vagrant-scp (From vagrant-scp)

And then the method is

 
vagrant scp <some_local_file_or_dir> [vm_name]:<somewhere_on_the_vm>

To get "vm_name" you need to run:

vagrant global-status

In my instance:

bclark@bclark:~/Vagrant_Projects/Imapproxy$ vagrant global-status
id       name         provider   state    directory                                          
--------------------------------------------------------------------------------------------- 
5250600  default      virtualbox running  /home/bclark/Vagrant_Projects/Imapproxy

Then to copy the files, I run:

vagrant scp 5250600:/etc/dovecot /path/to/where/you/want/it/stored

HTH
Brent

If you are having to use rsync, and transferring a lot of data. I find -z (--compress) is a great bandwith and time saver.

But the caveat is compressing already compressed data. It does not make sense to (re)compress. Therefore '--skip-compress=' is to the rescue.

The rsync man pages say, the list needs to be seperated by forward slashes (/). I dont know why forward slashes, when pretty much every other script / program makes use of commas.

Here is a list I found and all you need to do is:

export RSYNC_SKIP_COMPRESS=3g2/3gp/3gpp/7z/aac/ace/amr/apk/appx/appxbundle/arc/arj/asf/avi/bz2/cab/crypt5/crypt7/crypt8/deb/dmg/drc/ear/gz/flac/flv/gpg/iso/jar/jp2/jpg/jpeg/lz/lzma/lzo/m4a/m4p/m4v/mkv/msi/mov/mp3/mp4/mpeg/mpg/mpv/oga/ogg/ogv/opus/png/qt/rar/rpm/rzip/s7z/sfx/svgz/tbz/tgz/tlz/txz/vob/wim/wma/wmv/xz/z/zip


rsync -vaz --skip-compress=$RSYNC_SKIP_COMPRESS /source /target

HTH

Verify all account password hashes are shadowed

Heres how to verify that all your account password hashes, are shadowed. 

 

The command below should return “x”: 

 

# cut -d: -f2 /etc/passwd|uniq 

HTH 

Quicky secure Apache

LAMP stack which stands for Linux, Apache, MySQL/MariaDB and PHP/Python/Perl is a very popular model for serving websites.

This is my take on quickly and briefly apache on your server.

Securing Apache:

The default for Apache reveals a little too much about the server. Lets start by hiding some of that information, by editing:

sudo apt-get install libapache2-modsecurity

sudo vi /etc/apache2/conf-available/custom_security.conf

Paste the following:

ServerSignature Off

ServerTokens Prod 

TraceEnable Off
Options all -Indexes
Header unset ETag
Header always unset X-Powered-By
FileETag None

Run:

 
sudo a2enmod headers

sudo a2enconf custom_security.conf

sudo /etc/init.d/apache2 restart

Configuring mod_security:

sudo a2enmod security2

Configure the module and enable the OWASP ModSecurity Core Rule Set (CRS):

sudo mv /etc/modsecurity/modsecurity.conf-recommended /etc/modsecurity/modsecurity.conf

vi /etc/modsecurity/modsecurity.conf 

Paste:

 
SecRuleEngine On
SecResponseBodyAccess Off
SecRequestBodyLimit 8388608
SecRequestBodyNoFilesLimit 131072
SecRequestBodyInMemoryLimit 262144

sudo rm -rf /usr/share/modsecurity-crs
sudo git clone https://github.com/SpiderLabs/owasp-modsecurity-crs.git /usr/share/modsecurity-crs

sudo mv /usr/share/modsecurity-crs/crs-setup.conf.example /usr/share/modsecurity-crs/crs-setup.conf

sudo vi /etc/apache2/mods-enabled/security2.conf

<IfModule security2_module>
  SecDataDir      /var/cache/modsecurity
  IncludeOptional /etc/modsecurity/*.conf
  IncludeOptional "/usr/share/modsecurity-crs/*.conf"
  IncludeOptional "/usr/share/modsecurity-crs/rules/*.conf
</IfModule>

sudo /etc/init.d/apache2 restart

Be sure to keep an eye on the apache and mod_security logs but running:

sudo tail -f /var/log/apache2/*.log /var/log/mod_evasive/*

Automatic updates:

Last but not least, automatic updates. I know I said, this blog post is to secure apache, but one thing I am an advocate of, is secure your server. If you are a one man show, use unattended-upgrades.

Unattended-upgrades purpose is to keep the server current, up to date with the latest security (and other) updates automatically. This alone will allow you too sleep safe and tight at night. :)

sudo apt-get install unattended-upgrades

For extra security look at mod_evasive, and dont forget fail2ban.

Puppetserver refuses to start with /tmp mounted noexec.

Im in the process of investigating and moving our code base to Puppet 4.

I came across something very interesting today, in that Puppet 4 does not like /tmp to be mounted noexec.

Now, we mount /tmp with nosuid and noexec as per the recommendation in the Securing Debian Manual.

After much hours of trouble shooting I came across "Puppet Server run issue when /tmp/ directory mounted noexec" (And I dont know why I didnt read this in the first place.)

As per the recommendation, you basically need to add to /etc/default/puppetserver.


JAVA_ARGS="-Xms2g -Xmx2g -Djava.io.tmpdir=/var/tmp"

HTH

Brent

Puppetdb dies at random / silently

If you ever have it that the connection to puppet-db is refused or you find the daemon died.

Even if trawling through the puppetdb logs reveal nothing.

It might be that PuppetDB is running out of heap? Check
/var/log/puppetdb for the presence of the  file 'puppetdb-oom.hprof' for an indication that this is happening.

You may find the following to help.
https://docs.puppet.com/puppetdb/2.2/configure.html#configuring-the-java-heap-size

HTH
Brent

Rate limiting with nginx

I just love Nginx, it's amazing at how little memory it consumes and how well it performs, much of this is owed to nginx’s use of the event driven mode.

One of my teams repeated problems that come up is the simple fact that some users and bots are a little excessive of the servers.

 http {

limit_conn_zone  $binary_remote_addr zone=conn_limit_per_ip:50m;   limit_req_zone   $binary_remote_addr zone=req_limit_per_ip:50m rate=1r/s;

server {

location / {

limit_conn conn_limit_per_ip 10;             limit_req zone=req_limit_per_ip burst=10 nodelay;

}

}

}

The directives that count here: limit_conn_zone, limit_req_zone, limit_conn and limit_req.

We first use limit_req_zone to set up at least a rate limit zone, which will then be enabled by placing them inside specific nginx location directives.

We start by setting up our first zone named ‘default’, give it 50 megabytes of memory to track our sessions, and set a rate at 1 request per second.

We then implement it in the ‘/’ location, and give it a ‘burst’ of 10.

Every time that a bot exceed the rate of 1 request per second, they have to pay a token.  Once they’ve spent all of their tokens, they are given an HTTP 503 error message. 

503 means  the server is currently unavailable (because it is overloaded or down for maintenance). Generally, this is a temporary state.

As you experience an excessive bot you will see the following in your log

2016/09/01 10:06:29 [error] 109154#109154: *42450 limiting requests, excess:
10.195 by zone "req_limit_per_ip", client: ip.of.attacher, server: default,
request: "GET

*42450 means:

This is a connection number, also available as $connection.

109154#109154 means:

This is nginx worker PID (also available as $pid) and thread identifier.

10.195 means:

This is number of requests accumulated in the bucket.  If this 
number is more than burst defined (10 in our case), further 
request will be rejected.

Number of requests in the bucket is reduced according to the rate 
defined and current time, and may not be integer.  The ".195" 
means that an additional request will be allowed in about 195 
milliseconds assuming rate 1r/s.

You can get more information at the following location(s) :
http://www.checkupdown.com/status/E503.html
http://nginx.org/en/docs/http/ngx_http_limit_conn_module.html
http://nginx.org/en/docs/http/ngx_http_limit_req_module.html

HTH
Brent

Another way to calculate PHP5-FPM max_children

This is a slight update from my previous post, but I stumbled across the following command  / method to help calculate the value of "pm.max_children" for PHP5-FPM.

echo "pm.max_children = $(( $(awk '/MemTotal:/ { printf "%d\n", ($2*0.66) }' /proc/meminfo) / $(ps --no-headers -o "rss,cmd" -C php5-fpm | awk '{ sum+=$1 } END { printf ("%d\n", sum/NR) }') ))"

Now whats interesting is that the suggested value return, is a ridiculously high value, but I think what comes out of this, is that it illustrates how much more RAM you can give  PHP5-FPM.

But (un)fortunately real world exists, and there are other existing services in play (e.g. MySQL could be installed).

So if you use the above command, I highly suggest you adjust the PHP5-FPM value sparingly, and thereafter recheck your Munin / Cacti etc

HTH
Brent

Adjusting PHP5-FPM child processes (Apache)

Every now and then, on a clients server, we get the following dreaded message:


WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 4 idle, and 48 total children

So we will start off, by determining the non-swapped physical memory usage by each PHP5-FPM processes (notice its in kilo Bytes)

ps -ylC php5-fpm --sort:rss
ps --no-headers -o "rss,cmd" -C php5-fpm | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024,"M") }'

Thereafter I ran the following command to give me the total memory, of my current PHP5-FPM configuration.

ps -ylC php5-fpm --sort:rss | awk '!/RSS/ { s+=$8 } END { printf "%s\n", "Total memory used by PHP-FPM child processes: "; printf "%dM\n", s/1024 }'

In my case I still had plenty of RAM left.

Remember the appropriate value for pm.max_children can be calculated as:

pm.max_children = Total RAM of your server / Max child process size - in my case it was 67MB

pm.max_children = 70
pm.start_servers = 20
pm.min_spare_servers = 20
pm.max_spare_servers = 35
pm.max_requests = 500

Debian upgrade Wheezy to Jessie

Good day Guys

Here is my tip / howto to upgrade Debian Wheezy to Jessie.

I highly recommend first ensuring that your server is update.

So run:

# apt-get update

# apt-get upgrade

# apt-get dist-upgrade 

# apt-get -f install

Fix any outstanding issues. 

If all is good. You can start by changing you sources.list to reflect Jessie

sed -i 's/wheezy/jessie/g' /etc/apt/sources.list

sed -i 's/wheezy/jessie/g' /etc/apt/sources.list.d/*

The second sed is to ensure all other third pary repos reflect Jessie too.

There after you are good to go.

Run:

# rm -rf /etc/apt/preferences.d/*

# apt-get update

# apt-get upgrade

# apt-get dist-upgrade

# apt-get -f install 

This can take awhile.

After the upgrade it is recommended to reboot your system: 

# reboot

After your server has come up, for safety sake, run another 

# apt-get update

# apt-get upgrade
# apt-get -f install

You should have a new and shiny new OS, and you can verify, by using either 
hostnamectl or 
lsb_release -a

HTH
Brent

Insecure VPN?

Today I was reading that 90% of All SSL VPNs are Insecure.

• 77% of the SSL VPNs still use the insecure SSLv3 or the even less secure SSLv2
• 76% use an untrusted SSL certificate
• 74% have an insecure SHA-1 signature
• 41% use an insecure key length of 1024 for RSA certificates
• 10% rely on versions of SSL that are still vulnerable to the  Heartbleed attack

If you want to test if your VPN supports SS3, just run the following:

 openssl s_client -connect <server>:<port> -ssl3

If the connection succeeds, sslv3 is enabled. If it fails, it is disabled.

HTH
Brent

A tip to speed up Mozilla Firefox

Sometime some domains Firefox needs to resolve each domain name, one for an IPv4 address and once for an IPv6 address.

This results in lots of DNS requests, slowing down your web access.

If you are like 99.999% of the population without IPv6 access, translating domain names in IPv6 addresses is useless.

Try disabling this functionality, by typing about:config into the address bar.

Type ipv6 into the search bar and toggle network.dns.disableIPv6 to true.

HTH
Brent

Running a local Debian mirror (i386 and amd64 only)

For a while now, I have been running a Debian mirror. When you manage as many hosts like I am part of a team do, you tend to think about how it all negatively affect the upstream providers, and really just the community in general.

So with out further ado, here is my take and steps on setting up a mirror, using Raphael Geissert's tool ftpsync.

First you need to create a user and ensure the switch to that user.

sudo adduser  \
--system \
--shell /bin/bash \
--gecos 'User for FTPsync' \
--group \
--disabled-password \
ftpsync

sudo  -i -u ftpsync

git clone https://github.com/rgeissert/ftpsync /home/ftpsync/

Next on the list if to create the configs. Generally people just copy sample config, but there is quite a few options, but you only need a hand few, so therefore I am just going to share what I have.

cat ~/etc/ftpsync.conf
MIRRORNAME=`hostname -f`
TO="/var/www/mirror/debian"
RSYNC_PATH="debian"
RSYNC_HOST=ftp.nl.debian.org
LOGDIR="${BASEDIR}/log"
ARCH_EXCLUDE="alpha arm arm64 armel armhf hppa hurd-i386 ia64 kfreebsd-amd64 kfreebsd-i386 m68k mipsel mips powerpc s390 s390x sh sparc source ppc64el"
MAILTO="your@addressgoes.here"

Next on the list is to have a cron entry to kick off you new shiny mirroring tool. And for that you can just crontab -e , as user ftpsync.

And then add:

5 * * * * /home/ftpsync/bin/ftpsync sync:all >/dev/null 2>&1

And that’s that. You now just need to sit and watch the location on where you defined your $TO variable grow.

One thing I would like too add, at the time of writing, mirroring just i386 and amd64 is using on my VM 373Gigs (Hence the reason why I only mirror these two architectures). Show maybe just make sure you enough space allocated.

Two last parts
You then need to install and configure nginx or apache to offer and share the docroute of '/var/www/mirror/'.
Next you need to edit  (or create)

echo 'deb http://your.mirror.co.za/debian/ wheezy main contrib non-free' > /etc/apt/sources.list.d/debian.list

HTH
Brent

P.s. Remember there too are other repos. i.e.

backports
security
volatile etc

Random sleep duration in bash

I needed to insert random data into a test database, but I it need to be a 1-10 second random sleep/pause between in each insert.

I came across this.

sleep $[ ( $RANDOM % 10 )  + 1 ]s
(Thanks  http://blog.buberel.org/2010/07/howto-random-sleep-duration-in-bash.html)

If you want to see  the command I was using:

while true ; do mysql bctest -e "INSERT INTO random_lookup(lookup_value) SELECT LPAD( '', 100, MD5( CAST( RAND() AS CHAR ) ) ) FROM random_lookup LIMIT 100000;"; sleep $[ ( $RANDOM % 10 )  + 1 ]s; done

HTH

Brent

Sysctl Linux server for performance enhancement.

Here are my  tips and suggestions, that I apply to a server for performance enhancement.
I find these values to be safe to run any and everywhere.

sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.core.rmem_default=16777216
sysctl -w net.core.wmem_default=16777216
sysctl -w net.core.optmem_max=40960
sysctl -w net.ipv4.tcp_rmem='4096 87380 16777216'
sysctl -w net.ipv4.tcp_wmem='4096 65536 16777216'
sysctl -w net.ipv4.tcp_window_scaling=1
sysctl -w net.ipv4.tcp_fin_timeout=30
sysctl -w net.ipv4.tcp_keepalive_time=1800
sysctl -w net.ipv4.ip_local_port_range='1024 65000'

For those that use puppet, for your hiera configs:

 sysctl:
  net.core.rmem_max:
    value: "16777216"
  net.core.wmem_max:
    value: "16777216"
  net.core.rmem_default:
    value: "16777216"
  net.core.wmem_default:
    value: "16777216"
  net.core.optmem_max:
    value: "40960"
  net.ipv4.tcp_rmem:
    value: "4096 87380 16777216"
  net.ipv4.tcp_wmem:
    value: "4096 87380 16777216"
  net.ipv4.tcp_window_scaling:
    value: '1'
  net.ipv4.tcp_fin_timeout:
    value: '30'
  net.ipv4.tcp_keepalive_time:
    value: '1800'
 net.ipv4.ip_local_port_range:
     value: '1024 65000'

HTH
Brent

Issues importing with myloader on galera cluster.

I had a very interesting situation and problem whereby,  I was loading data into a Galera cluster using `myloader` (http://www.mydumper.org/).

Our 3 node cluster was correctly replicate writes when we insert
rows from the console. When I loaded our SQL dump, the table
definitions (all InnoDB) got created on all the nodes.

However, the rows from the import only showed up on the single node, I was loading onto, and did not make their way to the other nodes.

Whats was interesting, nothing was showing up in the error logs for the other nodes during the import. I even manually inserted a row into one of the new tables after the big import is done and only that single row showed up on the other nodes.

I eventually found the option ' -enable-binlog', via the man page (default is off). This specifically means that events will not be replicated.

Long story short.

On a galera cluster you need to add the option (--enable-binlog)

time myloader --database=$DATABASE --directory=/RESTORE/PATH --queries-per-transaction=50000 --threads=6  --verbose=3 -o --enable-binlog

HTH
Brent

Ansible 2 On Debian Wheezy.

Today was an interesting day.

I needed ansible ( >= 1.9.2) for a quick research project that resided on Debian Wheezy.

The current version on backports is 1.7.2.

Here is my steps to compiling a Debian package from git.


export DEBFULLNAME="Brent Clark"
export DEBEMAIL="brentgclark@gmail.com"
sudo apt-get install cdbs debhelper dpkg-dev git-core reprepro  python-setuptools devscripts build-essential asciidoc -y
cd /tmp
git clone git://github.com/ansible/ansible.git
cd /tmp/ansible 
git submodule update --init --recursive
make deb

Ansible will too need these packages.

sudo apt-get install python-crypto python-httplib2 python-jinja2 python-markupsafe python-paramiko python-six python-yaml sshpass -y

And then, to install.
dpkg -i /tmp/ansible/deb-build/unstable/ansible_2.0.0-0.git201508210336.9bb95b5.devel~unstable_all.deb

And then to ensure all is working, run:

 ansible localhost -m setup

HTH

Brent

Command line set timezone

In my environment I use vagrant, and I find the timezone that comes with the boxes is not my current location.

Therefore a quick and dirty hack to ensure the timezone reflects your location is:

cp /usr/share/zoneinfo/Africa/Johannesburg /etc/localtime

HTH
Brent
:)