Ivan Vari

A minimalist Sysop/Devops Craftsman

Solving Poor Network Performance on RHEL and CentOS 7

We are building the next generation online marketplace and part of it is a real-time Java application. This application is heavily optimised for its use case, handles zillions of short-lived tcp requests fast. Most of our operations complete quickly (<100ms) and some even more quickly (<500us).

Our old application pool is based on CentOS 6 nodes and we are doing considerably well on them. However recently, we deployed our new CentOS 7 based server farm and for some reason, we have been unable to meet the expectations set by the old pool.

Conditional SNAT With iRule on F5

Quick and dirty guide about how to create conditional SNAT with iRule on F5 and rewrite (NAT) IP addresses based on specific conditions.

We have 2 public IP netblocks for our production network, one is geographically registered in LA, California, the other is Amsterdam, Netherlands. It is very common that services such as Google, Amazon, Akamai, etc serve requests based on their source but occasionally they get it wrong so I needed a way to control what netblock my request is addressed out of.

Solving OpenVPN Poor Throughput and Packet Loss

This not about optimising OpenVPN, it is about solving OpenVPN poor throughput and packet loss issue, where the server receives traffic faster than it actually process.

We are currently in the process of moving data centers. This requires our Couchbase data to be in sync between Gütersloh (DE) and AMS-IX (NL) which does mean that XDCR needs to pump few hundred Gigs across every day and fast. After about 20 minutes or so, everything started to slow down for an unknown reason.

Merging Pillars in SaltStack

Merging or joining Pillars in SaltStack is supported but somewhat limited. It took me some time to work out a clean solution to support a specific manipulation so to make this easier, I am going to share my real life example.

WD MyCloud 2T NAS Review

I purchased a Seagate Central 2T NAS 5 months ago, for a low cost home media center solution. It worked reasonably well considering the low ~US130 cost although, I had ongoing issues with firmware updates, occasional drive performance, etc. Unfortunately, it failed last week and while I was looking for alternatives, I learnt that I was not the only one having problems with that device so I simply lost trust in Seagate forever.

I returned the drive, and the store offered me the WD MyCloud 2T as a replacement alternative without extra cost what I happily accepted.

Load Balancing and Sticky Sessions by URL Parameter

To be able to mimic our production workload in testing, we had to come with a low cost solution to load balance HTTP traffic between few application servers. In addition to that, for the first (initial request) we required even distribution amongst the backend nodes but, subsequent requests needed to be handled by the same backend server.

This task was relatively easy with NGINX, our preferred HTTP server however lately, I had to come up with a solution for apache 2.2 which was not as straight forward.

Dashlane vs 1Password

I am a sysop / devops engineer, love open source and security so I tend to ignore commercial software. For password valet, I have been using KeePass for years and happy with it except a couple of things:

  • written in .NET so cross platform integration has its challenges
  • browser integration

Although the browser integration is reasonably good now on Windows, it’s not as refined as its commercial competitors such as Dashlane or 1Password. So I decided to investigate these utilities to see if they can convince me to switch.

Running Pylint in PyCharm

I really liked the Pylint integration in Eclipse/Pydev but I have switched to PyCharm since JetBrains released CE edition. Pycharm supports PEP8 auditing “out of the box”, but I found out lately, that it is a little “loose” on style compared to pylint. Running pylint in pycharm didn’t seem to be supported in any ways so I became curious about how I could add this functionality to my favourite IDE.

After some searching, I realised that there is not much out there about this topic. I could not accept it and went after the challenge…

Solving Camel ActiveMQ Clients in TCP TIME_WAIT

We are an agile software development company and agile is great for “moving target”. We plan, work and implement changes in small batches and ongoing re-factoring is just the nature of what we do.

We recently added some functionality as well as increased traffic for one of our Java products utilising Apache Camel and ActiveMQ. The product has been in production for years now, functioning with very much zero defect rate. Not soon after deploying the new code, our monitoring system triggered alerts about unusually high TCP TIME_WAIT connection states on the server where the new code was running. We began the troubleshooting process and found they were all ActiveMQ connections to our broker. Our developers immediately confirmed that

there was no change on the ActiveMQ connection manager side.

Well, it turned out that it was exactly the problem.

Script to Clone SaltStack Formulas From GitHub

I am heavily into Salt infrastructure management at the moment, and wish to leverage all available (community written) formulas. Luckily, the SaltStack group maintains a collection of excellent formulas on their github page, and they are great source for states, ideas, best practices, etc. So I started cloning them, first the ones that I really needed. Then I realized later on, that some I may need in the near future so why not clone all of them and ensure I have a local copy of them for my development.

The pages have been updated fairly regularly, more and more people contributing now to the project, which is great however it started to become tedious to find new formulas and I needed an automated solution to keep up to date with the changes.