Coffee & Beer

Rantings and Ravings of the technical sort

Clean Puppet Up After a Rebuilt Automatically With Cobbler Triggers

Being a shop that is mostly hpc, our compute nodes are pretty disposable, so we rebuild them from time to time. We’re coming up on a push to normalize them a bit, and will be looking to rebuild a bunch in big batches. One of the headaches, that isn’t REALLY a headache, is cleaning up the puppet certs when a system is rebuilt. We autosign puppet certs, so the new ones will come in just fine, but you’ve got to remember to clean the old ones during/before the rebuild. Add storedconfigs to this, and salt minion keys, and there is a good bit of cleanup to get done during a rebuild.

So, first, I wrapped the 3 things we want to clean up, in a script:

(puppet_rebuild.rb) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
#!/usr/bin/env ruby

def printusage(error_code)
  puts "Usage: #{$0} [ list of hostnames as stored in hosts table ]"
  exit(error_code)
end

printusage(1) unless ARGV.size > 0

ARGV.each { |hostname|
        system("puppet cert clean #{hostname}")
        system("puppetstoredconfigclean.rb #{hostname}")
        system("salt-key -d #{hostname}")
}

So, pretty obviously, that cleans the puppet cert, the storedconfigs db entry, and the salt key (puppet master = salt master)

Okay, so, one stop shopping there, but I want this automatic. Wel, we use Cobbler to build systems/define kickstarts, and one of the last things in all of our kickstarts is:

1
wget "http://cobbler/cblr/svc/op/trig/mode/post/system/SOME_HOSTNAME" -O /dev/null

Which lets cobbler know the build is done. This can optioanlly trigger scripts in /var/lib/cobbler/triggers/install/post, so, I added one:

(clean_puppet.sh) download
1
2
3
4
5
6
7
8
9
10
11
12
#!/bin/bash

#$1 = type
#$2 = system name (NOT DNS/FQDN)
#$3 = IP

name=$2
hostname=`curl -s -x "" http://localhost:3000/hosts?format=yaml  | grep $name | sed -e 's/  - //g'`

hostname_fixed=${hostname//[[:space:]]}

/usr/bin/puppet_rebuild_host $hostname_fixed

So, its passed 3 arguments: The object type (system), the system name, and the IP. I take the name, and query out forman api for the fqdn (we have a few domains so I can’t assume hostname.my.domain.com), and the call teh script above to clean out everything for that host!

So, when it comes to puppet/salt certs, we don’t care now. New system are automatically accepted, and if you rebuild, the old ones are removed and new ones accepted, just like that!

Comments