Managing the puppet manifest using a vcs is a best practice and there is a lot of material on the web. The easier way to do it, is to use git directly in the directory /etc/puppet and use a simple synchronization strategy with an external repo, either to publish your work, or simply to keep a backup somewhere.

Things are a bit more complicated when you would like to co-administer the machine with multiple people. Setting up user accounts, permission and everything can be a pain in the neck. Moreover working from your desktop is always more comfortable then logging in as root on a remote system and make changes there ...

The solution I've chosen to make my life a bit easier is to use gitolite, that is a simple git gateway that works using ssh public keys for authentication and does not require the creating of local users on the server machine. Gitolite is available in debian and installing it is super easy : apt-get install gitolite .

If you use puppet already you might be tempted to use puppet to manage your gitolite installation. This is all good, but I don't advice you to use modules like this one http://forge.puppetlabs.com/gwmngilfen/gitolite/1.0.0 as it's going to install gitolite from source and on debian it's not necessary ... For my purposes, I didn't find necessary to manage gitolite with puppet as all the default config options where good enough for me.

Once your debian package is installed, in order to initialize your repo, you just need to pass to gitolite the admin public key, that is your .ssh/id_rsa.pub key and then run this command:

This will create the admin and testing repo in /var/lib/gitolite/repositories and setup few other things. At this point you are ready to test you gitolite installation by cloning the admin repo :

Gitolite is engineered to use only the gitolite use to manage all your repositories. To add more repositories and users you should have a look at the documentation and then editing the file conf/gitolite.conf to add your new puppet repository.

At this point, you can go two ways. If you use git to manage your puppet directory, you can just make a copy of it somewhere and then add gitolite as a remote

If you didn't use git before, you can just copy the manifest in your new git repository, make a first commit and push it on the server.

Every authorized users can now use git to clone your puppet repository, hack, commit, push ...

One last step is to add a small post-receive hook on the server to synchronize your gitolite repository with the puppet directory in /etc : This will sync your main puppet directory and trigger changes on the nodes for the next puppetd run. First I created a small shell script in /usr/local/bin/puppet-post-receive-hook.sh :

This script presuppose that your git repo in /etc/puppet has the gitolite repo as origin ... Then I added a simple hook in the gitolite git repo that calls this script using sudo :

And since you are at it you should also add a pre-commit hook to check the manifest syntax. This will save you a lot of useless commits.

If you use a more complicated puppet setup using environments (I'm not there yet, and I don't think my setup will evolve in that direction in the near future), you can use puppet-sync that seems a neat script to do the job.

For the moment this setup works pretty well. I'm tempted to explore mcollective to trigger puppet runs on my nodes, but I'm there yet...

Developing software while working at university always and invariably put you in a uncomfortable position. On the one side academia is one of the driving forces behind good software development practices. We study, analyze, test, defend and sometimes attack different development methodology and more importantly we teach students what they should do once outside academia. On the other hand, just because our primary job is to do all of the above, sometimes is difficult while developing software ourselves, to follow these best practices. Sometimes is more a matter of mind setting, sometimes is a matter of resources and time.

Today I invested a bit of time to configure and install virtualbox and run a jenkins instance on it. I prefer not to litter my laptop with jenkins as I know I won't run it all the time and I don't want to leave around hundreds of MBs of unused dependencies.

Installing virtualbox is pretty easy. It's in the debian repos, and it's just one apt-get way. Once installed, you need to create a virtual machine. For this purpose I simply downloaded on the the netinstall CD and I use it in the VB GUI as installation CD. Everything went smoothly and my host was up and running in no time.

By default VB set up a NAT network on the first adapter (eth0). This is nice and easy if you want a machine that does not need to talk to the outside world. On the other hand, if you want to to connect to this machine you need to do a bit more of work. To this end I added a host-only network between the guest VM and the host. The catch is that you first need to create a host adapter on the host machine. Simply go File -> Preferences -> Network and create a new interface. This is the interface that will appear on you host. On the guest side, configure the second adapter (eth1) as host-only network and select the interface that you just created before.

The first time you run the VM, the NAT connection should work straightaway, while the second interface will not. To fix this problem you need to edit the file /etc/network/interfaces and set eth1 to auto-configure using dhcp.

Once this is all done, we need to install jenkins. This is pretty easy as well.

the jenkins wiki gives all the explanations you need :

https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu

once this is done, on your host got to :8080 and voila ! you can start playing with jenkins !!!

Third post about ganeti.

Ganet-debootstrap-instance contains a nice set of scripts to create a debian (or derivatives) image using debootstrap. Images can be configured and customized by writing simple hooks script to modify various aspects of the default installation. However writing these script is not really fun and pushing it too far can lead to long messy scripts, loosing the overall benefit of automatic configuration.

Puppet is my configuration management tool of choice, but installing puppet on a new machine requires few magic incantations that the user should perform manually, or in a semi automatic mode (autosign=true) to make it work. My goal is to install puppet automatically on the newly created instance so it will run and configure the new instance at the first boot. From that moment on I'll forget about ganeti and configure all remaining services of my new VM using puppet.

In order to do so, we need to install puppet (and apt-get update/upgrade...), create the ssl certificates for the client and enabling the puppet daemon on the client. We add another hook in /etc/ganeti/instance-debootstrap/hooks/ :

This script uses puppetca to create on the puppet (and ganeti) server the client key, sign it, and then copy it to the target machine. Notice that we create the certificate for a fqnd name $INSTANCE_NAME.$DOMAIN or otherwise puppet will complain loudly. This is not strictly needed, but if you want to do otherwise, you'll need to fiddle with the puppet configuration a bit more. The procedure to create a puppet certificate server-side is well documented on the puppet website, so if you are curious about the details duck-duck-it .

I'll start here a small series of posts about ganeti, xen and puppet. For my work I run few servers sitting on xen and it has always been a bit of a pain to create a new instance and keep it up to date. Up to now I've used the excellent xen-create-image tool to create my VMs, but I wanted to try something new and more sexy... Last week I finally found some time to learn (and a spare box to run my experiments) how to use ganeti. Ganeti is the only tool I tried out, but it seems to fit the bill for my use and it seems polished and mature project to me... Moreover I've seen a presentation about it in every FLOSS conference I've attended in the last few years and I thought it was time to give it a try.

Installing and configuring ganeti is fairly easy, there is a lot of documentation available and this post is not going to be about installing it, but rather how to create a new bare instance with ganeti-deboostrap-instance. There is also a way to create a new instance from an image, but I didn't go that way yet.

This first post is about the first problem I've encountered, that is, how to automatically assign a network address and a name to each new instance created by gnt-instance add. Since all my instances should be able to communicate together on the same subnet, I've decided to configure xen to create a NATted private network and add each new instance to this network.

The first step is to create an interface in /etc/network/interfaces .

This is the standard debian way but since xen uses a different naming convention (here I'm using ganeti naming convention xenbr0 vs xen-br0), I need to convince tell xen what I intend to do by adding these lines in /etc/xen/xend.config :

Next I have to connect my real network interface to the private network using few iptables rules in /etc/rc.local (probably there is a better place to do this...):

The xen setup is complete and every new image should have a vif connected to the subbet 10.0.0.0. The xen setup corresponds to the physical wiring of the network. The next step is to configure each instance so to allow them to communicate on this subnet. Since I build my VMs using ganeti-debootstrap-instance, and by default debootstrap does not configure the network, we need to add a new hook in the directory /etc/ganeti/instance-debootstrap/hooks.

This hook will do two things. First it will configure the interfaces of the new instance to get configured using dhcp, second, it will add an entry to the dnsmasq configuration file to make this instance known to the world. This basically boils down to add a file in /etc/dnsmasq.d/ with the mac address of the new instance and its designated name. Dnsmasq will then provide an ip address for this instance and add it to the dns.

dhcp-host=aa:00:00:24:6c:8a,node1

Configuring dnsmasq is pretty easy as well. First I want it to answer dhcp queries only on the internal network, second I want to configure my clients passing 10.0.0.1 as nameserver and gataway. You can just add the following lines in /etc/dnsmasq.d/general to get it going.

To create your new instance you can just run the following command :

If you are running your dom0 on debian squeeze before running this command you should configure ganeti to pass the right xen parameters to the newly created instance :

I use --no-ip-check and --no-name-check to skip ip and dns check performed by ganeti and to avoid a sort of chicken-egg problem, where the name and address of this new instance is yet unknown to dnsmasq and that node1 is the name that will be used by the hook to add an entry in the dnsmasq configuration. debootstrap+unstable is a variant of the default configuration and you need to add it to the list of variants used by ganeti-deboostrap-instance.

That should be it. The new instance should come up with a dynamically assigned ip address, able to talk to the outside world and automatically known by all the other machine on the subnet via dns.

Next post will be about how to add a swap hook for ganete-debootstrap-istance.