What's in Your Runbook?

At least once a year, the time comes to re-address the documentation around the IT department regarding disaster recovery. One of the things I've been working on improving over the last two years is our network runbook. We keep a copy of this binder in two places - in our document management system (which can be exported to a CD) and in hard copy, because when systems are down the last thing you want to be unable to access is the documentation about how to make things work again. 

Here's a rundown of what I have in mine so far, it's in 10 sections:

1. Runbook Summary - A list of all servers with their IP address, main purpose, a list of notable applications running on each and which are virtual or not. I also include a list of which servers are running which operating system, a list of key databases on servers and finally copies of some of our important passwords.
2. Enterprise AD - A listing of all corporate domains and which servers perform what roles. I include all IP information for each server, the partitions and volumes on each and where the AD database is stored. Functional levels for the domain and forest are also documented.
3. Primary Servers and Functions - This is similar to the Enterprise AD section, but it's for all non-domain controllers. I list out server information for file services, database servers and their applications and backup servers. I document shares, partition and volume information (including the size), important services that should be running and where to find copies of installation media.
4. ImageRight - Our document management system deserves it's own section. In addition to the items similar to the servers in the previous section, I also include some basic recovery steps, dependencies and the boot sequence of the servers and services. Any other information for regular maintenance or activities on this system are also included here.
5. Email / Exchange - This is another key system that deserves it's own section in my office. I include all server details (like above) and also completely list out every configuration setting in Exchange 2003. This will be less of an issue with Exchange 2007 or 2010 where more of the configuration information is stored in Active Directory. However, it makes me feel better to have it written down. I also include documentation related to our third-party spam firewall and other servers related to email support.
6. Backup Details - A listing of each backup server, what jobs it manages and what data each of those jobs capture.
7. Telecommunications - Details about the servers and key services. I also include information regarding our auto attendants, menu trees and software keys.
8. Networking - Maps and diagrams for VLANs, static IP address assignments, external IP addresses
9. Contacts & Support - Internal and external support numbers. Also include circuit numbers and other important identifying information.
10. Disaster Recovery - Information about the location of our disaster recovery kit, hot line and website. A list of the contents of our disaster kit and knowledge base articles related to some of our DR tasks and hard copies of all our disaster recovery steps.

This binder is always in flux - I'm always adding and changing information and making notes, as well as trying to keep up with changes that other team members are making to the systems they work with most.  It will never be "done" but I'm hoping that whenever I have to reach for it, that it will always be good enough.