Thursday, November 19, 2015

Running a local Debian mirror (i386 and amd64 only)

For a while now, I have been running a Debian mirror. When you manage as many hosts like I am part of a team do, you tend to think about how it all negatively affect the upstream providers, and really just the community in general.

So with out further ado, here is my take and steps on setting up a mirror, using Raphael Geissert's tool ftpsync.

First you need to create a user and ensure the switch to that user.

sudo adduser  \
--system \
--shell /bin/bash \
--gecos 'User for FTPsync' \
--group \
--disabled-password \
ftpsync

sudo  -i -u ftpsync

git clone https://github.com/rgeissert/ftpsync /home/ftpsync/

Next on the list if to create the configs. Generally people just copy sample config, but there is quite a few options, but you only need a hand few, so therefore I am just going to share what I have.

cat ~/etc/ftpsync.conf
MIRRORNAME=`hostname -f`
TO="/var/www/mirror/debian"
RSYNC_PATH="debian"
RSYNC_HOST=ftp.nl.debian.org
LOGDIR="${BASEDIR}/log"
ARCH_EXCLUDE="alpha arm arm64 armel armhf hppa hurd-i386 ia64 kfreebsd-amd64 kfreebsd-i386 m68k mipsel mips powerpc s390 s390x sh sparc source ppc64el"
MAILTO="your@addressgoes.here"


Next on the list is to have a cron entry to kick off you new shiny mirroring tool. And for that you can just crontab -e , as user ftpsync.

And then add:

5 * * * * /home/ftpsync/bin/ftpsync sync:all >/dev/null 2>&1

And that’s that. You now just need to sit and watch the location on where you defined your $TO variable grow.

One thing I would like too add, at the time of writing, mirroring just i386 and amd64 is using on my VM 373Gigs (Hence the reason why I only mirror these two architectures). Show maybe just make sure you enough space allocated.

Two last parts
You then need to install and configure nginx or apache to offer and share the docroute of '/var/www/mirror/'.
Next you need to edit  (or create)

echo 'deb http://your.mirror.co.za/debian/ wheezy main contrib non-free' > /etc/apt/sources.list.d/debian.list

HTH
Brent

P.s. Remember there too are other repos. i.e.

backports
security
volatile etc


Wednesday, October 7, 2015

Random sleep duration in bash

I needed to insert random data into a test database, but I it need to be a 1-10 second random sleep/pause between in each insert.

I came across this.

sleep $[ ( $RANDOM % 10 )  + 1 ]s
(Thanks  http://blog.buberel.org/2010/07/howto-random-sleep-duration-in-bash.html)

If you want to see  the command I was using:

while true ; do mysql bctest -e "INSERT INTO random_lookup(lookup_value) SELECT LPAD( '', 100, MD5( CAST( RAND() AS CHAR ) ) ) FROM random_lookup LIMIT 100000;"; sleep $[ ( $RANDOM % 10 )  + 1 ]s; done

HTH

Brent

Tuesday, September 15, 2015

Sysctl Linux server for performance enhancement.

Here are my  tips and suggestions, that I apply to a server for performance enhancement.
I find these values to be safe to run any and everywhere.

sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.core.rmem_default=16777216
sysctl -w net.core.wmem_default=16777216
sysctl -w net.core.optmem_max=40960
sysctl -w net.ipv4.tcp_rmem='4096 87380 16777216'
sysctl -w net.ipv4.tcp_wmem='4096 65536 16777216'
sysctl -w net.ipv4.tcp_window_scaling=1
sysctl -w net.ipv4.tcp_fin_timeout=30
sysctl -w net.ipv4.tcp_keepalive_time=1800
sysctl -w net.ipv4.ip_local_port_range='1024 65000'

For those that use puppet, for your hiera configs:

 sysctl:
  net.core.rmem_max:
    value: "16777216"
  net.core.wmem_max:
    value: "16777216"
  net.core.rmem_default:
    value: "16777216"
  net.core.wmem_default:
    value: "16777216"
  net.core.optmem_max:
    value: "40960"
  net.ipv4.tcp_rmem:
    value: "4096 87380 16777216"
  net.ipv4.tcp_wmem:
    value: "4096 87380 16777216"
  net.ipv4.tcp_window_scaling:
    value: '1'
  net.ipv4.tcp_fin_timeout:
    value: '30'
  net.ipv4.tcp_keepalive_time:
    value: '1800'
 net.ipv4.ip_local_port_range:
     value: '1024 65000'

HTH
Brent

Wednesday, September 9, 2015

Issues importing with myloader on galera cluster.


I had a very interesting situation and problem whereby,  I was loading data into a Galera cluster using `myloader` (http://www.mydumper.org/).

Our 3 node cluster was correctly replicate writes when we insert
rows from the console. When I loaded our SQL dump, the table
definitions (all InnoDB) got created on all the nodes.

However, the rows from the import only showed up on the single node, I was loading onto, and did not make their way to the other nodes.

Whats was interesting, nothing was showing up in the error logs for the other nodes during the import. I even manually inserted a row into one of the new tables after the big import is done and only that single row showed up on the other nodes.

I eventually found the option ' -enable-binlog', via the man page (default is off). This specifically means that events will not be replicated.

Long story short.

On a galera cluster you need to add the option (--enable-binlog)

time myloader --database=$DATABASE --directory=/RESTORE/PATH --queries-per-transaction=50000 --threads=6  --verbose=3 -o --enable-binlog

HTH
Brent

Friday, August 21, 2015

Ansible 2 On Debian Wheezy.

Today was an interesting day.

I needed ansible ( >= 1.9.2) for a quick research project that resided on Debian Wheezy.

The current version on backports is 1.7.2.

Here is my steps to compiling a Debian package from git.


export DEBFULLNAME="Brent Clark"
export DEBEMAIL="brentgclark@gmail.com"
sudo apt-get install cdbs debhelper dpkg-dev git-core reprepro  python-setuptools devscripts build-essential asciidoc -y
cd /tmp
git clone git://github.com/ansible/ansible.git
cd /tmp/ansible 
git submodule update --init --recursive
make deb

Ansible will too need these packages.

sudo apt-get install python-crypto python-httplib2 python-jinja2 python-markupsafe python-paramiko python-six python-yaml sshpass -y

And then, to install.
dpkg -i /tmp/ansible/deb-build/unstable/ansible_2.0.0-0.git201508210336.9bb95b5.devel~unstable_all.deb

And then to ensure all is working, run:

 ansible localhost -m setup

HTH

Brent

Wednesday, August 19, 2015

Command line set timezone

In my environment I use vagrant, and I find the timezone that comes with the boxes is not my current location.

Therefore a quick and dirty hack to ensure the timezone reflects your location is:

cp /usr/share/zoneinfo/Africa/Johannesburg /etc/localtime

HTH
Brent
:)