Apparently this book also does not suck. Yay!

I’ll be giving a live dramatic reading from the book — er, that is, teaching a course on DNSSEC — at BSDCan Wednesday morning this week.

And before you reach for a dictionary, “idempotent” means that you can run the same script against your servers and have the end result be the same. You can run an Ansible script (or playbook) against a group of servers, take note of those that fail, modify the script, and run it again against the same group of servers, and Ansible will verify that the servers need the playbook run before running it. Only the servers that need the change will get it.

Why would this ever happen? Maybe a datacenter is cut off by a network issue, or a LDAP server chokes, or gremlins invade a server, or a script fails because an intruder has hacked the server and this is your early warning. Your management tools need to deal with all of these.

For example, I have an Ansible playbook that uploads a new PF configuration file and reloads the PF rules. Ansible compares the existing PF configuration to one being distributed, and if the file hasn’t changed, doesn’t reload the rules. This isn’t a huge deal for PF, but for some applications it’s vital.

Another nice feature about Ansible is that it uses only SSH and Python. Most Unixes acquire Python as an application dependency somewhere along the way, and it’s small enough that I have no real objection to installing on servers without it. And both Puppet and CFEngine have dedicated agent software, so some kind of agent is going to wind up on the managed machine anyway.

The biggest problem I had with Ansible was with playbooks. There’s a whole bunch of playbook documentation, and Ansible ships with sample playbooks, but they’re written somewhat like man pages, for people who already have some clue about the topic. So here are a couple really rudimentary Ansible playbooks, with explanations.

---
- hosts: pf
  user: ansible
  sudo: yes
  tasks:
  - name: copy pf.mgmt.conf to servers
    action: copy src=/home/ansible/freebsd/etc/pf.mgmt.conf 
      dest=/etc/pf.mgmt.conf owner=root group=wheel mode=0644
    notify:
      - reload pf

  handlers:
    - name: reload pf
      action: shell /sbin/pfctl -f /etc/pf.conf

Ansible playbooks are written in YAML, Yet Another Markup Language. I concede that my first thought on hearing the words “yes another markup language” is “that statement needs some obscenities between the second and third word.” But XML would be way overkill for Ansible. (And the YAML folks have changed their name to a different acronym, Yaml Ain’t Markup Language, trying to escape the stigma of being yet another bleeping bleepety-bleep markup language.)

All YAML files start with a triple dash. They are space-sensitive — don’t use tabs, only spaces.

At the top level (no indents), we have the triple dash and a dash

---
- hosts: pf

The leading hyphen basically means “new thing here,” as far as I can tell.

At the second level of configuration, indented two spaces, we have five sections: hosts, user, sudo, tasks, and handlers.

The hosts statement gives the name of a group of hosts. Ansible has an easily understood hosts file. This playbook applies to a group of hosts called “pf.”

The user definition tells ansible which user to use. Ansible should SSH into the target servers as the user “ansible.”

The sudo statement tells ansible to use sudo to perform this command. My ansible user on each host has sudo privileges, but needs a password. We’ll get an opportunity to enter the password when we run the playbook. (Ansible could log into the target server as root and avoid the need for sudo, or let the ansible user have sudo without a password, but the thought of doing either makes my spleen threaten to leap up my gullet and block my windpipe, so I don’t.)

The tasks section is where things get interesting. We actually do stuff here. I define a third level of indentation (four spaces) after the tasks statement, and start them with a dash.

Our first task has a name, “copy pf.mgmt.conf to servers.”

The action that follows uses the Ansible copy module. I define a source file, a destination file, and set the owner and permissions.

The notify statement tells the task to activate the handler named “reload pf” If the action changes a target system, the action triggers the handler. If the action doesn’t change anything, the handler is not triggered.

We then have the handler section. It’s at the same indent level as tasks, sudo, user, and hosts, so it’s a major section. There’s one handler, “reload pf.” It performs one action, fires up a shell and runs a command.

Taken as a whole, this playbook copies a file to all the servers in the pf group and reloads the file. The file pf.mgmt.conf contains the IP addresses of my management hosts, as I discussed elsewhere.

Now let’s look at a slightly more complex playbook that does the same thing.

---
- hosts: linux-internal
  user: ansible
  sudo: yes
  tasks:
  - name: copy iptables.mgmt.conf to servers
    action: copy src=/home/ansible/linux/etc/iptables.mgmt.conf 
      dest=/etc/iptables.mgmt.conf owner=root group=root mode=0644
    notify:
      - reload ipset mgmt
  - name: copy iptables.rules to servers
    action: copy src=/home/ansible/linux/etc/solus.iptables.rules 
      dest=/etc/iptables.rules owner=root group=root mode=0644
    notify:
      - reload iptables

  handlers:
    - name: reload ipset mgmt
      action: shell /usr/sbin/ipset restore -! < /etc/iptables.mgmt.conf
    - name: reload iptables
      action: shell /sbin/iptables-restore -! < /etc/iptables.rules

This playbook updates the firewall rules on my Linux hosts. These CentOS hosts are a little simpler in that they all share a common function (virtualization). They can have a common iptables ruleset as well as a common list of management addresses. I talk about how I use ipsets, and why the rules are set up this way, elsewhere. But the important thing is:

ext_if="em0"
include "/etc/pf.mgmt.conf"
...
pass in on $ext_if proto icmp from any to any
#mgmt networks can talk to this host on any service
pass in on $ext_if from to any
...

The file pf.mgmt.conf looks like this:

table const { 192.0.2.0/24, 198.51.100.128/25 }

When I add new management addresses I copy pf.mgmt.conf to each machine, run pfctl -f /etc/pf.conf, and the new addresses can connect.

But surely there’s some similar function on a Linux box?

To complicate matters further, our environment includes both Ubuntu and CentOS machines. (Why? Because we don’t run operating systems, we run applications, and applications get picky about what they run on.) Each version has its own way of saving and restoring iptables rules. I want to use the same method for both operating systems. What we’ve used is a single rules file, /etc/iptables.rules, read by iptables-restore at boot. We specifically don’t want to trust a copy of the packet filter rules saved by the local machine, as problems can persist across reboots. The current iptables.rules looks something like this:

*filter
#mgmt addrs
-A INPUT -s 192.0.2.0/24 -i eth0 -j ACCEPT
-A INPUT -s 198.51.100.128/25 -i eth0 -j ACCEPT
#keep state
-A INPUT -p tcp -m state --state ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -p udp -m state --state ESTABLISHED -j ACCEPT
-A OUTPUT -p udp -m state --state NEW,ESTABLISHED -j ACCEPT
#local stuff here
...
#permit ICMP
-A INPUT -p icmp -j ACCEPT
-A OUTPUT -p icmp -j ACCEPT
-A INPUT -i eth0 -j DROP
COMMIT

I don’t want to change /etc/iptables.rules for each machine at this point. They all vary slightly. (One day the machines will be classified by roles, but we’re in an intermediate stage right now.) Instead, I want to have the list of management addresses in a separate file. I want to copy the new file to the server, run a command, and have the new list of management addresses be live.

ipsets seems to be the way to do this. Let’s find out.

On my crashbox, I’ll create an ipset. I’m using an ipset of type nethash, because it takes CIDR blocks rather than individual IP addresses. The ipset is called mgmt, just like the management addresses on my BSD machines.

# ipset create mgmt nethash

It returns silently. Did it create the ipset?

# ipset list
Name: mgmt
Type: hash:net
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 16760
References: 0
Members:

OK, it’s in memory. Now add some addresses.

# ipset add mgmt 192.0.2.0/24
# ipset add mgmt 198.51.100.128/25

Are those addresses really in the set? Let’s ask again.

# ipset list mgmt
Name: mgmt
...
Members:
192.0.2.0/24
198.51.100.128/25

Now, export this to a file.

# ipset save mgmt > iptables.mgmt.conf

I use the file iptables.mgmt.conf to mirror pf.mgmt.conf. That file should contain something like this:

create mgmt hash:net family inet hashsize 1024 maxelem 65536
add mgmt 192.0.2.0/24
add mgmt 198.22.63.128/25
add mgmt 198.51.100.128/25

Can I restore the ipset from the file? Destroy the set.

# ipset destroy mgmt
# ipset list

It’s gone. Now to restore it from memory.

# ipset restore < iptables.mgmt.conf
# ipset list
...

All my rules are there.

Now, let’s teach iptables how to use an ipset. Rather than defining addresses, we use the -m set option.

# iptables -A INPUT -i eth0 -m set --match-set mgmt src -j ACCEPT

In the iptables.rule file, it would look like this.

*filter
#allow mgmt IPs
-A INPUT -i eth0 -m set --match-set mgmt src -j ACCEPT
...

When you have several management networks, this is certainly much shorter and easier to read.

When you update the iptables.mgmt.conf file, read it in with ipset restore. You must use the -! flag. This tells ipset to ignore that the ipset already exists, and restore the contents of the ipset from the file.

# ipset restore -! < iptables.mgmt.conf

I can now copy this file to my hosts, run a command, and the packet filter rules are updated, without touching my main rules file.

I don’t recall anyone using a symbol as a command-line flag like this before, but I actually kind of like this one. “I said DO IT, damn you!”