Rc_whatis, Finger for Systems

As I’ve mentioned, at work, we’ve got a lot of systems, physical and virtual. All sorts of different hardware, specs, ages, etc. Its hard to keep track of, and harder to quickly say “This is what that is” with confidence. Puppet and the ability to collect and query facts from systems has been a HUGE help with this, of course. We use Foreman to provide a nice shiny web interface to this, but, its not the fastest, and most of us live in the CLI dayin a day out. So, I wanted a way to quickly, from any system, find out about any other system in our infrastructure. So, rc_whatis was born.

Its really just a hacky bit of ruby. I’ve opened up the /facts REST endpoint on our puppet massters so that any of our systems can get the facts of any other system, ssl cert or no. We don’t have any secrets in this info.

1
2
3
4
5
6
[root@nichols2tst ~]# rc_whatis --help
Usage: whatis [options] <hostname>
-j, --json                       JSON output
-y, --yaml                       YAML output
-p, --pp                         Pretty Print output
-a, --all                        Use all facts

As you can see, its pretty staight forward to call. Even provides a few serialized forms of output so other scripts can call this (more on that in a few!). Output looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[root@nichols2tst ~]# rc_whatis nichols2tst
Hostname: nichols2tst
Born_on: 2012-08-24
Manufacturer: Red Hat
Productname: KVM
Serialnumber: Not Specified
Operatingsystem: CentOS
Operatingsystemrelease: 6.3
Architecture: x86_64
Processor0: QEMU Virtual CPU version (cpu64-rhel6)
Processorcount: 1
Memorytotal: 996.77 MB
Kernelrelease: 2.6.32-279.5.2.el6.centos.plus.x86_64
Ipaddress: 10.X.X.X
Macaddress: 00:16:3E:XX:XX:XX
Vlan: 375
Location_row: virtual
Location_rack: virtual
Location_ru: virtual
Uptime: 10 days
Virtual: kvm
Hypervisor: kvm03a

Thats the default output, of a select # of facts. -a, --all would get everything, of course. I’ve already mentioned born_on in another post. location_* comesfrom a hacky little interfacts (yaml) to racktables we have (this systems is virtual, so, no physical location). And hypervisor is a conditional query based on a fact we populate on production hypervisors that marks them as such, as well as a list of vms running on a given hypervisor.

The coolest bit is it now exists in roots $PATH on ALL of our systems, so this info, for any host, is now a few keystrokes away all the time!

Even better, our nagios alerts now call this when crafting their emails to send us, so when a system drops, there is no question as to what it is/where/etc. Its all right in the email, along with a like to its full Foreman page and entry in Nagios, of course along witht he normal alert info!

Source? Heres your source!

(rc_whatis.rb) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
#!/usr/bin/ruby

require 'optparse'
require 'yaml'
require 'puppet'
require 'puppet/node'
require 'puppet/node/facts'
require 'pp'
require 'json'
require 'yaml'


#Setup
#
#List of values we want, unelss we call w/ --all
values = ["hostname","born_on","manufacturer","productname","serialnumber","operatingsystem","operatingsystemrelease","architecture","processor0","processorcount","memorytotal","kernelrelease","ipaddress","macaddress","vlan","location_row","location_rack","location_ru","uptime","virtual"]

#Puppet Server Info:
puppet_server="puppet"
puppet_port="8140"
puppet_url="https://" + puppet_server + ":" + puppet_port

options={}
OptionParser.new do |opts|
        opts.banner = "Usage: whatis [options] <hostname>"

        opts.on("-j","--json","JSON output") do |j|
                options[:json] = j
        end
        opts.on("-y","--yaml","YAML output") do |y|
                options[:yaml] = y
        end
        opts.on("-p","--pp","Pretty Print output") do |p|
                options[:pp] = p
        end
        opts.on("-a","--all","Use all facts") do |a|
                options[:all] = a
        end
end.parse!

if ARGV.length != 1
        puts "Please pass a hostname, see --help"
        exit
else
        host = ARGV[0]
end

if host.match(".edu")
        fqdn = host.to_s
else
        fqdn = host.to_s + ".domain.edu"
end

fact_url = puppet_url + "/production/facts/" + fqdn
fact_cmd = "curl -s -k -H \"Accept: yaml\" " + fact_url
rawfacts = `#{fact_cmd}`
if rawfacts.match("Could")
        puts rawfacts
        exit
end
rawfacts = rawfacts.sub("!ruby/object:Puppet::Node::Facts","")
rawfacts = YAML::parse(rawfacts)
rawfacts = rawfacts.transform

#We can now access things like:
# rawfacts["values"]["virtual"]

facts = Hash.new
rawfacts["values"].each_pair do |a,b|
        facts[a] = b
end

#Okay, we have a hash or all facts.
#Make second hash of specific facts


facts2 = Hash.new
if options[:all] == true
  facts2 = facts
else
values.each do |val|
  facts2[val] = facts[val]
end
end

#Lets see if it is virtual so we can add a fact about where it is running...
if facts2["virtual"] == "kvm"
        hypervisor_url = puppet_url + "/production/facts_search/search?facts.kvm_production=true"
        hypervisor_cmd = "curl -s -k -H 'Accept: YAML' " +  hypervisor_url
        hypervisor_yaml = `#{hypervisor_cmd}`
        hypervisors = YAML::load(hypervisor_yaml)
        hypervisors.each do |hyp|
                hyp_facts_url = puppet_url + "/production/facts/" + hyp
                hyp_facts_cmd = "curl -s -k -H \"Accept: yaml\" " + hyp_facts_url
                hyp_facts = `#{hyp_facts_cmd}`
                hyp_facts = hyp_facts.sub("!ruby/object:Puppet::Node::Facts","")
                hyp_facts = YAML::parse(hyp_facts)
                hyp_facts = hyp_facts.transform
                vms = hyp_facts["values"]["kvm_vms"].to_a
                vms.each do |vm|
                        if vm.match(facts2["hostname"])
                                facts2["hypervisor"] = "#{hyp}"
                        end
                end
        end
        #Add "hypervisor" to the list of values we care about
        values.push("hypervisor")
end

#output time
if options[:json] == true
        puts facts2.to_json
elsif options[:yaml] == true
        puts facts2.to_yaml
elsif options[:pp] == true
        pp facts2
else
  if options[:all] == true
        pp facts2
  else
        values.each do |val|
                puts "#{val.capitalize}: #{facts2[val]}"
        end
  end
end

could it be written better? yep. But its quick and its a start!

Born on Dates for Systems

Wrote this fact a while ago but though it was worth throwing up here.

We’ve got a lot of systems. Our inventory is slightly lacking, and many were build a long long time ago. Many time we’ve found ourselves asking “When the hell was X system built?” or maybe “rebuilt”. Thus, for RHEL/CentOS systems at least, we can get a fact for that:

(born_on.rb) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#!/usr/bin/ruby
require 'facter'

begin
          Facter.operatingsystem
rescue
          Facter.loadfacts()
end
os = Facter.value("operatingsystem")
if os.match(/CentOS|RedHat/) then
  unless  `rpm -q --last basesystem`.empty?
      Facter.add("born_on") do
          setcode do
              date = `rpm -q --qf '%{INSTALLTIME}' basesystem`
              born_on = `date --date=@#{date} +%F`.chomp
              born_on
          end
      end
  end
end

Giving us:

1
born_on => 2011-11-03

Updated: Puppet Facts for Puppet Classes

Just the other day on Google+ I got a comment form someone who had found my old “Puppet facts about puppet classes” post and had used it. Sadly, I had gone through a few revisoins after that post and never followed up. There as a bit of a memory leak, and I decided I wanted things done a little different. Instead of creating a fact per-class (and having n fact if the clas wasn’t used), I’d rather have a list of the classes, as one fact, I can regex/etc on. Our group has recently started creating facts like this as json arrays so we can prase the data easy later, and its a bit more readable even if not.

(puppet_classes_2.rb) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#!/usr/bin/ruby
#Get puppet classes, from /var/lib/puppet/classes.txt
#
require 'facter'
require 'json'
begin
        Facter.hostname
rescue
        Facter.loadfacts()
end
hostname = Facter.value('hostname')

classes_txt = "/var/lib/puppet/classes.txt"

if File.exists?(classes_txt) then
        f = File.new(classes_txt)
        classes = Array.new()
        f.readlines.each do |line|
                line = line.chomp.to_s
                line = line.sub(" ","_")
                classes.push(line)
        end
        classes.delete("settings")
        classes.delete("#{hostname}")
        Facter.add("puppet_classes") do
                setcode do
                        classes.to_json
                end
        end
end

thus, a node would have a fact like:

1
puppet_classes => ["base","salt::minion","ssh::service"]

It Helps Not to Break Your Blog

Whenyou update the CNAME file.

Getting Racktables Location Info Into Puppet

At work we have had Racktables ((http://racktables.org/)) for a while for tracking where things are. Its…..okay. Its not the best,but, eh, it works. We need to do a better job with clena data etc, but, it works.

One thing we don’t like, however, is its current lack of an API. We can query the db directly, but thats kinda clunky. So, the other ngiht I had an idea. A few hours later, it was basically done. A YAML api (well, a cheap mans api) for racktables!

So, let me set this up. I want a yaml document for each host, with location info. This way, facter, or, anything else, can pull down that location info. When someone changes something in racktables, the ymal document should be updated. I don’t need real time, but liets say 30 minutes. Sounds like a cron job…

So, a script, running from cron, reading the racktables database and spitting out YAML documents of the data on a per host basis. Okay. What data? Well, I want Row (for us this is datacenter+row), Rack, RU, Height of the system (how many RU’s does it take up?). Since racktables does have some asset tags, might as well pull that so we can compaire to puppet/foreman while we’re at it.

So, a yaml document like:

(rackfact_example.yaml) download
1
2
3
4
5
6
---
ru: "16"
row: "DataCenterB Row 6"
rack: "1"
height: "1"
asset: "326859"

And I want things at a url like

1
http://server/rackfacts/systems/HOSTNAME

Also, at the request of a co-worker, just the endpoint /systems will return ALL of the systems.

So, after a BUNCH of digging into the racktables DB and dusting off my SQL, I came up with:

(rack2yaml.rb) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
#!/usr/bin/env ruby
require 'yaml'
require 'mysql'
path="/var/www/rackfacts/"

my = Mysql::new("racktables","rackuser","rackpass","racktables")

rackobjs=my.query("select distinct RackObject.name,RackSpace.unit_no,Rack.name,RackRow.name,RackObject.asset_no,RackObject.id from RackObject,RackSpace,Rack,RackRow where RackObject.id = RackSpace.object_id AND Rack.id = RackSpace.rack_id AND RackRow.id = Rack.row_id AND RackObject.objtype_id = 4;")

objects=Array.new
rackobjs.each do |row|
        obj=Hash.new
        obj["name"] = row[0].to_s.downcase.strip.delete(' ').delete('#').delete('/').delete('"')
        obj["ru"] = row[1].to_s.strip.delete('"')
        obj["rack"] = row[2].to_s.strip
        obj["row"] = row[3].to_s.strip
  obj["asset"] = row[4].to_s.strip
  obj["id"] = row[5]
        objects.push(obj)
end

#Need to get the height of a given system...
objects.each do |obj|
  height=my.query("SELECT COUNT(distinct unit_no) FROM `RackSpace` WHERE object_id #{obj['id']};")
  obj["height"] = height
end
#Writing Systems, so lets do this in /systems/

path = path + "systems/"

#Lets clean the existing ones, so stale thigns are removed.
clean = "rm -rf #{path} && mkdir #{path}"
%x[ #{clean} ]
objects.each do |thing|
        fpath = path+thing["name"]
        yobj=Hash.new
        yobj["ru"]=thing["ru"]
        yobj["rack"]=thing["rack"]
        yobj["row"]=thing["row"]
  yobj["asset"]=thing["asset"]
  yobj["height"]=thing["height"]
        f=File.open(fpath,'w')
        f.write(yobj.to_yaml)
end

allpath=path + "index.html"
all=File.open(allpath,'w')
all.write(objects.to_yaml)

So, this ruby script:

  • Sets up some stuff
  • connects to mysql
  • runs a query to get most (not height) of the system info.
  • Height is a second query, as racktables doesn’t know about height, but rather has a single object us multiple RU’s in a rack…
  • So, we query for that for each system, counting the times a given object is in a Rack, distinct on unit_no’s as racktables also has a front,back,middle format (so a 4U system that goes front to back might have 12 entries!)
  • We then merge all this data together in an array of hashes
  • Clean out our path
  • dump all the yaml documents
  • dumps out the whole array for the ALL systems bit

Ta Da!

Okay, so now we have those yaml documents, and every 30 minute sthis will get done so anythign we cleanup/remove will be available as well. Now what? Lets pull that in as some facts!

(rackfacts.rb) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
#!/usr/bin/env ruby

require 'facter'
require 'net/http'
require 'yaml'
require 'timeout'
begin
  Facter.hostname
rescue
  Facter.loadfacts()
end

if Facter.value('is_virtual') == "true"
  virtual = "virtual"
  Facter.add("location_ru") do
      setcode do
          virtual
      end
  end
  Facter.add("location_rack") do
      setcode do
          virtual
      end
  end
  Facter.add("location_row") do
      setcode do
          virtual
      end
  end
  Facter.add("location_height") do
      setcode do
          virtual
      end
  end
else   
  hostname = Facter.value('hostname')
  rackfact_host = "racktables"
  rackfact_dir  = "/rackfacts/systems/#{hostname}"
  unknown = "unknown"
  begin
      Timeout::timeout(2) {
          rescode=Net::HTTP.get_response rackfact_host,rackfact_dir
          if (rescode.code =~ /2|3\d{2}/ )
              rackfact = YAML::load(rescode.body)
              ru = rackfact["ru"]
              rack = rackfact["rack"]
              row = rackfact["row"]
              height = rackfact["height"]
              Facter.add("location_ru") do
                  setcode do
                      ru
                  end
              end
              Facter.add("location_rack") do
                  setcode do
                      rack
                  end
              end
              Facter.add("location_row") do
                  setcode do
                      row
                  end
              end
              if height != nil
                  Facter.add("location_height") do
                      setcode do
                          height
                      end
                  end
              end
          else
              Facter.add("location_ru") do
                  setcode do
                      unknown
                  end
              end
              Facter.add("location_rack") do
                  setcode do
                      unknown
                  end
              end
              Facter.add("location_row") do
                  setcode do
                      unknown
                  end
              end
              Facter.add("location_height") do
                  setcode do
                      unknown
                  end
              end
          end
      }
  rescue Timeout::Error
      Facter.add("location_ru") do
          setcode do
              unknown
          end
      end
      Facter.add("location_rack") do
          setcode do
              unknown
          end
      end
      Facter.add("location_row") do
          setcode do
              unknown
          end
      end
      Facter.add("location_height") do
          setcode do
              unknown
          end
      end
  end
end

Now, we can query the nice puppet/forman api’s for location data! Better yet, I can use these with storedconfigs to do things like add location info to Ganglia! Or have systems in 1 data center get specific configs (dns? puppet master? AD?)

Our config management system is now location aware!

Proper Kvm Xml Backups With Ruby

So, while kvm keeps the xml describing a running domain in /var/run/libvirt/qemu, it turns out this isn’t exactly the cleanest thing to backup. I HAD been doing this, with a simple rsync, but realizing this I decided we should do it propper, with libvirt runy bindings and all. So, here it is:

(dump_xmls.rb) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#!/usr/bin/env ruby

require 'libvirt'
hostname=`hostname --short`.chomp
conn = Libvirt::open('qemu:///system')
vms = Hash.new
conn.list_domains.each do |domid|
        dom = conn.lookup_domain_by_id(domid)
        vms[dom.name] = dom.xml_desc
end

destination = "/n/kvm_stor01/xml/#{hostname}/"
vms.each do |dom,xml|
  file_dest = "#{destination}#{dom}" + ".xml"
 puts "writting xml for #{dom} to #{file_dest}"
 f=File.open(file_dest, "w")
 f.write(xml)
 f.close
end

Pretty simple. Figure out the hostname (this is deployed to multiple kvm hosts), get the list of domains running here, itterate on each one and get the name and xml, and populate a hash with these. Then, itterate that hash dumping the xml into a file on our shared storage, which is in turn checkpointed and backed up. Ta da.

Clean Puppet Up After a Rebuilt Automatically With Cobbler Triggers

Being a shop that is mostly hpc, our compute nodes are pretty disposable, so we rebuild them from time to time. We’re coming up on a push to normalize them a bit, and will be looking to rebuild a bunch in big batches. One of the headaches, that isn’t REALLY a headache, is cleaning up the puppet certs when a system is rebuilt. We autosign puppet certs, so the new ones will come in just fine, but you’ve got to remember to clean the old ones during/before the rebuild. Add storedconfigs to this, and salt minion keys, and there is a good bit of cleanup to get done during a rebuild.

So, first, I wrapped the 3 things we want to clean up, in a script:

(puppet_rebuild.rb) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
#!/usr/bin/env ruby

def printusage(error_code)
  puts "Usage: #{$0} [ list of hostnames as stored in hosts table ]"
  exit(error_code)
end

printusage(1) unless ARGV.size > 0

ARGV.each { |hostname|
        system("puppet cert clean #{hostname}")
        system("puppetstoredconfigclean.rb #{hostname}")
        system("salt-key -d #{hostname}")
}

So, pretty obviously, that cleans the puppet cert, the storedconfigs db entry, and the salt key (puppet master = salt master)

Okay, so, one stop shopping there, but I want this automatic. Wel, we use Cobbler to build systems/define kickstarts, and one of the last things in all of our kickstarts is:

1
wget "http://cobbler/cblr/svc/op/trig/mode/post/system/SOME_HOSTNAME" -O /dev/null

Which lets cobbler know the build is done. This can optioanlly trigger scripts in /var/lib/cobbler/triggers/install/post, so, I added one:

(clean_puppet.sh) download
1
2
3
4
5
6
7
8
9
10
11
12
#!/bin/bash

#$1 = type
#$2 = system name (NOT DNS/FQDN)
#$3 = IP

name=$2
hostname=`curl -s -x "" http://localhost:3000/hosts?format=yaml  | grep $name | sed -e 's/  - //g'`

hostname_fixed=${hostname//[[:space:]]}

/usr/bin/puppet_rebuild_host $hostname_fixed

So, its passed 3 arguments: The object type (system), the system name, and the IP. I take the name, and query out forman api for the fqdn (we have a few domains so I can’t assume hostname.my.domain.com), and the call teh script above to clean out everything for that host!

So, when it comes to puppet/salt certs, we don’t care now. New system are automatically accepted, and if you rebuild, the old ones are removed and new ones accepted, just like that!

Vim Trick of the Day

So, we have lots and lots of puppet manifests and some of them are cleaner to read than others. I fnd myself re-indenting things all the time, and though, hmm, must be a better way. So, after a bit of searching, I added this to my ~/.vimrc

1
autocmd BufWrite,FileReadPre   *.pp    :normal gg=G

Now, whenever I open, or save a puppet manifest, the whole buffer is re-indented. So, if I’m editing some previously written file, or myself am sloppy, it all gets cleaned up as son as I open/save the file.

Win

Rsync vs Tar

So, I’ll spare the gory details (you can read about some of them at the bosses blog, here and here), but we at work had the need to shift a pretty sum of data (80TB or so) as fast as possible to recover from a hardware failure that impacted one particular group. Oh, and the 80TB is millions upon millions of tiny files.

Now, we were using rsnapshot, so there was no “restore” to be done to the files, we could access them straight away, and even exported them read-only while we came up with a plan so users at least had access.

Once the plan was formed, and destination storage quickly provisioned (hello smaller ext4 LVM volumes instead of monolithic xfs), came the task of copying the data over, and fast.

Rsync is always a friend and a bit of a good standby for moving data around, especially for the situations when you want to preserve everything about the files and have the ability top stop and restart transfers. Obviously we were not going to do this over ssh (ouch), so the target storage areas were mounted on the source systems, and rsyncs started running locally, a la:

1
rsync -av --progress /mnt/backups/section01/ /mnt/destination01/section01

Simple enough.

We had 3, 16 Core, 48GB source systems, and 5 8 core 24GB destination systems, so we spun up a few rsyncs per source system and let them roll overnight.

This morning, not nearly as much data had moved as we had hoped.

Why? Because rsync needs to do fstats on each file, on both ends as it rolls along. Big files flew over the wire, but, most of what we were moving were tiny, tiny files, and as I said, millions and millions of them.

So, we did:

1
2
cd /mnt/backups/
tar -cf - section01 | tar -xf - -C /mnt/destination01/ 

Background that, and crank up a few others on the other sections, and things started going MUCH faster. We wanted a little faster, so:

Source:

1
mount -o remount,noatime /mnt/backups

Destination:

1
mount -o remount,noatime /mnt/destination01

Boom. Far, far fewer fstats. Access times not being dealt with (we don’t care at this point), so fewer fstats again.

Moving Puppet From Subversion to GIT in 15 Minutes While Adding Dynamic Environments

For (almost) as long as we’ve had a pupet installation at work, we’ve had it in Subversion to track changes. This has changed/evolved a few times, but has always remained in subversion in one way or another. Recently we starting tracking other bits (documentation, scripts, etc) in git, and the idea of being on 2 different revision systems didn’t really sit well with me. Most of the team has taken up git very well, so the choice was made to move our puppet manifests etc to git. Once i started loking into it, also found the very cool “Dynamic Environments w/ git branches” trick, talked about here and here, and thought the move to git would be the perfect time to move to this.

I’m going to skip the testing/waiting I had to do (making sure our git server and puppet amster could talk ssh to each other, setting up the ssh keys for git to use to push and puppet to pull, etc), and jump right into the implementation and transition, which took about 15 minutes total yesterday.

So, first, To get the lions share of the data shifting done, I used svn2git to get the svnrepo sync’d to a git repo. our svn repo, while it used to have a few branches, prior to this had been compacted to just a trunk (which wasn’t called “trunk”) and no tags etc. So, I ran:

1
$> svn2git --username matt --nobranches --notags --rootistrunk -v https://puppet.server/svn/puppet/

And let that chug along for a while.

Once that was done I could add the remote git server as an origin:

1
2
$> git remote add origin git@git.server:puppet.git
$> git push

Now, the puppet svn repo is in git and on the git server. I had to sync a few times due to some quick changes going into the svn repo, so that was a simple

1
2
$> svn2git --rebase
$> git push

Okay, finally it was go time. Step 1, make the svn repo readonly. How? Thisis served with apache, so, a simple pre-commit hook did the job:

1
2
3
#!/bin/sh
echo "This svn repo is now read-only! No Commits accepted!"
exit 1

See what I did there? The pre-commit never DOES anything with the commit, so the commit is never acepted by the svnserver!

Okay, on to the transition. On our puppet master, everything lived in /etc/puppet which was an svn repo. Lets stop the puppet master for a moment (clients will go about their merry way) move that aside, and setup the new location:

1
2
3
4
5
6
$>service  httpd stop (we run puppet via passenger)
$>cd /etc
$>mv puppet puppet.svn.backup
$>mkdir -p puppet/environments
$>chown -R puppet:apache puppet
$>chmod -R g+w puppet

Okay. Now, lets get into the weeds for a second here. So, using the dynamic environments, our git server will, with a post-receive hook I’ll show in a second, ssh to our puppet master, and checkout the branch (master or otherwise) to /etc/puppet/environments/$BRANCH. Now, we tend to use “master” for most of our small edits (adding a node, etc), so “master” => “production”. I didn’t want a puppet environment called “master”, so a tiny bit extra logic was added to the post-receive to change the location of the branch checkout to “production” if the master branch was change, but otherwise use the branchname, such as “matts_new_feature”. Okay here is the post-receive:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
#!/bin/sh
read oldrev newrev refname

REPO="git@git.server/puppet.git"
BRANCH=`echo $refname | sed -n 's/^refs\/heads\///p'`
BRANCH_DIR="/etc/puppet/environments"
SSH_ARGS="-i /var/lib/puppet/.ssh/id_puppet_rsa"
SSH_DEST="puppet@puppet.server"

#If working on the master branch, its really production and should go in environments/production
if [ "$BRANCH" == "master" ] ; then
        BRANCH_REAL="production"
else
        #Otherwise its a non-master/production branch and the env can be created w/ the branch name
        BRANCH_REAL="$BRANCH"
fi

if [ "$newrev" -eq 0 ] 2> /dev/null ; then
  # branch is being deleted
  echo "Deleting remote branch $BRANCH_DIR/$BRANCH_REAL"
  ssh $SSH_ARGS $SSH_DEST /bin/sh <<-EOF
  cd $BRANCH_DIR && rm -rf $BRANCH_REAL
EOF
else
  # branch is being updated
  echo "Updating remote branch $BRANCH_DIR/$BRANCH_REAL"
  ssh  $SSH_ARGS $SSH_DEST /bin/sh <<-EOF
  { cd $BRANCH_DIR/$BRANCH_REAL && git pull origin $BRANCH ; } \
  || { mkdir -p $BRANCH_DIR && cd $BRANCH_DIR \
  && git clone $REPO $BRANCH_REAL && cd $BRANCH_REAL \
  && git checkout -b $BRANCH origin/$BRANCH ; 
  EOF
fi 

This, paired with a nice pre-receive server side syntax check, and we’re looking pretty good and automatic (I’ll share that in another post) Okay, so thats in place. Now, svn2git works great, btu i want to use a clean, clean git-only checkout of the new puppet repo to finish this off with, so, on my local system:

1
2
3
4
$> mv puppet puppet.svn-git
$> git clone git@git.server:puppet.git puppet.git
$> cd puppet.git
$> vi puppet.conf

Now, in puppet.conf, I setup the dynamic environments:

1
2
3
environment = production
manifest = $confdir/environments/$environment/manifests/site.pp
modulepath = $confdir/environments/$environment/modules

Now lets commit/push:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$> gits
# On branch master
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
# modified:   puppet.conf
#
no changes added to commit (use "git add" and/or "git commit -a")

$> git commit -a
[master d846620] change to puppet.conf for the environments change
  1 files changed, 7 insertions(+), 7 deletions(-)

$> git push
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 383 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Updating remote branch /etc/puppet/environments/production
remote: Cloning into production...
To git@git.server:puppet.git
      73f5405..d846620  master -> master

Ta Da! Now, lets look on the puppet master:

1
2
3
4
$>pwd
/etc/puppet/
$>ls environments
production

So, its there, lets start puppet back up:

1
$>service httpd start

Get on a client of two, run puppet agent -t, all is well, and its great success! This litterally (of course, with testing, setting up keys, etc out o the way) took about 15 minutes yesterday. Nothing like pulling the rug out from under 1800+ systems/14000 cores and putting it back without them noticing!

Now, the really cool part is:

1
2
3
4
$>git checkout -b matts_test
$>vim somefiles.pp
$>git commit -a
$>git push

Will make a new branch/environment in /etc/puppet/environments/matts_test, which clients can use like puppet agent -t --environment=matts_test! Want it to go away after changes have merged? Simple!

1
2
3
4
5
  $> git checkout master
  $> git merge matts_test
  $> git push origin :matts_test
  $> git branch -d matts_test
  $> git push

Ta da! The branch/environment is no longer valid, and as been removed form /etc/puppet/environments/!