Tracking, auditing and managing your server configuration with Subversion in 10 minutes

Hello, again! For this article in the Server management series, we'll explore an uncommon use of Subversion — instead of using it for source code management, you'll be using it as a tool to make server administration painless and error-proof.

Most servers are configured once, then left to run for as long as they can (which, in practical terms, means 'until hardware or software fails'). More often than not, the cause for server failure is rooted in configuration mistakes. That's why it's wise to keep a log of all configuration changes, and take detailed notes every time a setting is changed. But, as the collection of servers you need to manage grows, bookkeeping tasks turn into a humongous monster.

No more. Enter Subversion.

What Subversion is

Subversion is, primarily, a source code control system. You feed your files into a Subversion repository, and from that point on, Subversion oversees them. As you or other developers change files, you periodically commit your changes to the Subversion repository. Subversion tracks those modifications, and allows you to replicate them, take snapshots of an entire source tree, and other nifty things. Even file moves, copies and deletes are (nearly) transparently tracked by Subversion — and it has the memory of an elephant: you can delete files and, later on, recall them by revision number.

Subversion also excels with text files; fortunately for us, UNIX configuration files are, mostly, text.

Subversion is, as you probably already know, a better CVS than CVS. Indeed, it was designed as a replacement for CVS, so if you're familiar with CVS, you'll probably feel right at home with Subversion.

Let's understand three fundamental concepts in Subversion.

• There's a central repository. Only the repository administrator should touch that. You'll be playing the role of repository administrator only once.
• Users don't touch the central repository. They check out private "working copies" and work on those copies. Once they've hacked their working copy into a desired state, they commit those changes into the repository. You'll be playing the role of user most of the time.
• On any given working copy, there are two 'axes' users can travel (whether to check out a few files, or to make a diff between two states):
• in time: once you're using Subversion, you can go back and forth "in time", by switching to older and newer revisions (the newest of which is customarily called HEAD)
• between branches and tags: at any point in time, you can instruct Subversion to make a copy of a folder into another folder. You can use this functionality to make periodic snapshots of a certain state, regardless of revision number.

How we'll use Subversion

First, let's review the tasks we'll perform:

• We'll create a Subversion repository to store the server configuration history
• We'll prepare the configuration directory
• We'll check the configuration files in
• We'll investigate how to perform followup tasks

I'm assuming that you have Subversion installed; in other words, you should have the svn and svnadmin commands and they should work properly. I'm also assuming that you'll be performing the following tasks as root.

The ideal situation to begin applying this tutorial is right after your server has been freshly installed. However, for practical purposes, any server that's configured and running will do.

Okay. That's enough of the lists and introductions. Time for some action.

Creating the Subversion repository

If you're familiar with UNIX, you'll know /var is the customary directory for files that pertain to the whole system and are changed. So, following tradition, we'll create a /var/preserve/config repository. Type the following command at your console:

[rudd-o@amauta2 ~]# svnadmin create \
/var/preserve/config

(note the backslashes are being used to add whitespace)

That should create a /var/preserve/config directory, with a couple of files in it. Those files are not meant to be editing, and they'll be opaque to us for the rest of the tutorial. As usual, I'd advise you to secure that directory so only root can read and write files to it.

Now, you'll create two directories directly into the repository. You'll use these directories to travel back and forth between known configuration states.

• trunk/ will contain the current configuration
• tags/ will contain snapshots (we'll learn more about them later)

To perform this task, just type:

[rudd-o@amauta2 ~]# svn mkdir \
file:///var/preserve/config/trunk/ \
file:///var/preserve/config/tags/ \
-m 'Creating trunk and tags directories'

The -m argument specifies a message to attach to the operation. You can consult these messages afterwards through the svn log command.

Preparing the configuration directory

In true UNIX tradition, /etc is the place to go for system-wide configuration. For the rest of the tutorial, I'll assume those are the files you want to keep in check.

To track files in /etc, you need to both:

• place its contents into the Subversion repository
• make it into a working copy

That's easily accomplished via the following command:

[rudd-o@amauta2 ~]# svn checkout \
file:///var/preserve/config/trunk/ /etc

Once you've done that, /etc will be a working copy. Time to add existing files into Subversion.

Checking existing configuration files into the repository

[rudd-o@amauta2 ~]# cd /etc
[rudd-o@amauta2 /etc]# svn status

You should see a long listing of files, like this:

?      4Suite
?      acpi
?      adjtime

The question marks at the beginning of each line mean that Subversion has no idea what those files are doing there. So, you'll add them to the repository:

[rudd-o@amauta2 /etc]# svn add *

You'll see svn working intensely to add those files. Note that the files are not being added to the repository yet — they're only being queued for addition. To commit these files into the repository:

[rudd-o@amauta2 /etc]# svn commit \
-m 'Initial addition of files'

And svn should start doing its magic. Once it's done, it'll tell you the revision number.

Followup maintenance

Okay, let's review a few things you need to keep in mind from now on.

When configuration files are added to /etc

Check for added files with svn status /etc. You should see them listed with a question mark.

You should use svn add to add them to the working copy, and then svn commit the added files into the repository. Many people make the mistake of configuring freshly installed files. Do not do that. Instead, commit new files first, then edit. That way, you'll have a way to track modifications right back to the pristine configuration files.

When configuration files are deleted from /etc

Check for deleted files with svn status /etc. You should see them listed with an exclamation sign.

After doing the check, svn delete them. Don't forget to commit at the end.

Before moving files or directories within /etc

Disregard the standard UNIX coreutils for file management, because they bypass Subversion tracking. Use svn add, svn copy, svn move and so on. Don't forget to commit at the end.

What to write on log messages

Use your imagination! At the very least, you should be typing who did the change, the reasons behind the change and any other related ideas that might be forgotten but needed later on.

How to view the log and audit changes

Use svn log /etc. Adding a --verbose option will show you the paths changed.

You can also obtain a patch (diff output) of changes between revisions. Use svn diff -r first:last /etc, replacing first and last with the first and last revision numbers you want to use as first and last reference points, respectively. Remember that HEAD works great when used in place of the last revision. A plain svn diff /etc will show you the differences in files that haven't been committed yet, in diff format.

How to snapshot the configuration and travel between snapshots

To snapshot the configuration, first ensure no files have changes pending to be committed in /etc (using svn diff /etc and svn commit /etc), then use the following command:

[rudd-o@amauta2 ~]# svn copy \
file:///var/preserve/config/trunk \
file:///var/preserve/config/tags/a-snapshot

replacing a-snapshot with the name of the snapshot you want to take. Subversion will take a snapshot of the current configuration state.

Seeing what's changed between two tags or the trunk and a tag is easy:

[rudd-o@amauta2 ~]# svn diff \
file:///var/preserve/config/tags/a-snapshot \
file:///var/preserve/config/tags/a-newer-snapshot

[rudd-o@amauta2 ~]# svn diff \
file:///var/preserve/config/tags/old-snap \
file:///var/preserve/config/tags/trunk

And traveling to a tag, making the whole /etc and its contents go to a known state, is easy as well.

[rudd-o@amauta2 ~]# cd /etc; \
svn switch file:///var/preserve/config/tags/known-state

I don't recommend making any changes to configuration files and then committing those changes if your /etc directory has been switched to a tag, because you'd overwrite the snapshot with those changes. Better to create a top-level branches/ directory in your repository, copy the tag to branches/a-branch-name, switch to the branch a-branch-name and then make changes there.

Conclusions

Indeed, adding Subversion to the server management mix introduces extra complexity. But the gains fairly outweigh the drawbacks. Clever readers should have noticed, just by skimming this piece, the potential that Subversion has. Subversion makes auditing, known configuration state restoration, and configuration backups much more deterministic and predictable.

Moreover, common snapshotting tasks can be automated via Vixie cron jobs. If you know how to serve Subversion repositories through the network, you'll see that designing, staging and distributing configuration sets and managing them remotely from a central location is incredibly easy. And there's so much more Subversion can do to stretch your limited available time. If you want to know more, there's always the Subversion book (aptly named Version control with Subversion).

And that's my cue to continue my day job. Happy hacking!