I’ve Got Nothing: The DR Checklist

So what do you have to lose?  If you’ve been reading along with the blog series, I hope you’ve been thinking a bit about ways you can bring your disaster recovery plans to the next level. My first post in the series on what to consider might have gotten you started on some of the items in this list. If you need some ideas of where to go next, or if you happen to be just starting out, here is a even longer list of things you might need.

Disclaimer: I love technology, I think that cloud computing and virtualization are paramount to increasing the speed you can get your data and services back online. But when disaster strikes, you can bet I’m reaching for something on paper to lead the way.  You do not want your recovery plans to hinge on finding the power cable for that dusty laptop that is acting as the offline repository for your documentation. It’s old school, but it works. If you have a better suggestion than multiple copies of printed documentation, please let me know. Until then, finding a ring binder is my Item #0 on the list.  (Okay, Hyper-V Recovery Manager is a pretty cool replacement for paper if you have two locations, but I'd probably still have something printed to check off...)

The Checklist

1. Backups - I always start at the backups. When your data center is reduced to a pile of rubble the only thing you may have to start with is your backups, everything else supports turning those backups into usable services again. Document out your backup schedule, what servers and data are backed up to what tapes or sets, how often those backups are tested and rotated. Take note if you are backing up whole servers as VMs, or just the data, or both. (If you haven’t yet, read Brian’s post on the value of virtual machines when it comes to disaster recovery.)
2. Facilities - Where are you and your backups going to come together to work this recovery magic? Your CEO’s garage? A secondary location that’s been predetermined? The Cloud?  List out anything you know about facilities. If you have a hot site or cold site, include the address, phone numbers and access information. (Look at Keith’s blog about using Azure for a recovery location.)
3. People - Your DR plan should include a list of people who are part of the recovery process. First and foremost, note who has the right to declare a disaster in the first place. You need to know who can and can’t kick off a process that will start with having an entire set of backups delivered to an alternate location.  Also include the contact information for the people you need to successfully complete a recovery - key IT, facilities and department heads might be needed.  Don’t forget to include their backup person.
4. Support Services - Do you need to order equipment?  Will you need support from a vendor? Include names and numbers of all these services and if possible, include alternatives outside of your immediate area. Your local vendor might not be available if the disaster is widespread like an earthquake or weather incident.
5. Employee Notification System - How do you plan on sharing information with employees about the status of the company and what services will be available to use?  Your company might already have something in place - maybe a phone hotline or externally hosted emergency website. Make sure you are aware of it and know how you can get updates made to the information.
6. Diagrams, Configurations and Summaries - Include copies of any diagrams you have for networking and other interconnected systems. You'll be glad you have them for reference even if you don't build your recovery network the same way.
7. Hardware - Do you have appropriate hardware to recover to? Do you have the networking gear, cables and power to connect everything together and keep it running? You should list out the specifications of the hardware you are using now and what the minimum acceptable replacements would be. Include contact information for where to order hardware from and details about how to pay for equipment. Depending on the type of disaster you are recovering from, your hardware vendor might not be keen on accepting a purchase order or billing you later. If you are looking at Azure as a recovery location, make sure to note what size of compute power would match up.
8. Step-By-Step Guides - If you’ve started testing your system restores, you should have some guides formed.  If your plans include building servers from the ground up, your guides should include references to the software versions and licensing keys required. When you are running your practice restores, anything that makes you step away from the guide should be noted. In my last disaster recovery book, I broke out the binder in sections, in order of recovery with the step-by-steps and supporting information in each area. (Extra credit if you have PowerShell ready to automate parts of this.)
9. Software - If a step in your process includes loading software, it needs to be available on physical media. You do not want to have to rely on having a working, high-speed Internet connect to download gigs of software.
10. Clients - Finally, don’t forget your end users. Your plan should include details about how they will be connecting, what equipment they would be expected to use if the office is not available and how you will initially communicate with them.  Part of your testing should include having a pilot group of users attempt to access your test DR setup so you can improve the instructions they will be provided. Chances are, you’ll be too busy to make individual house calls. (For more, check out Matt’s post on using VDI as a way to protect client data.)

Once you have a first pass gathering of all your disaster recovery items and information, put it all in a container that you can send out to your off-site storage vendor or alternate location. Then when you practice, start with just the box - if you can’t kick off a recovery test with only the contents (no Internet connection and no touching your production systems) improve them and try again.  Granted, if you are using the cloud as part of your plan, make sure you know which parts require Internet access, have a procedure for alternative connectivity and know what parts of your plans would stall while securing that connection.  You won't be able to plan for every contingency, but knowing where parts of the plan can break down makes it easier to justify where to spend money for improvement, or not.

No matter the result of your testing, it will be better than the last time. Go forth and be prepared.

Oh, one more thing, if you live in a geographic area where weather or other "earthly" disasters are probable, please take some time to do some DR planning for your home as well.  I don't care who you work for, if your home and family aren't secure after a disaster you certainly won't be effective at work. Visit www.ready.gov or www.redcross.org/prepare/disaster-safety-library for more information.

This is post part of a 15 part series on Disaster Recovery and Business Continuity planning by the US based Microsoft IT Evangelists. For the full list of articles in this series see the intro post located here: http://mythoughtsonit.com/2014/02/intro-to-series-disaster-recovery-planning-for-i-t-pros/

Help Shape Microsoft’s Cloud Technologies: Join the IT Pro User Panel

IT Professionals: Have you ever been deep in the guts of a gnarly infrastructure deployment, automation, configuration, trouble-shooting or similar task, and thought to yourself something like this: “Why didn’t that darn product team at Microsoft make this tool work better, more like what I need it to do?” If you see yourself in this story, we have an opportunity for you!

The Windows Server System Center design and development team is looking for IT Pros with knowledge & experience in all aspects of infrastructure and services management. We need to know how to make Microsoft technologies supporting these scenarios work better for you.

IT Pros like you with these specialized knowledge and skills are hard to find, so we’ll make it worth your while. Here’s what you get from participating in the Panel:

·       Opportunities to influence WSSC design and development in areas such as

o   Prioritizing tool capabilities (what do you need MOST in these tools and technologie?)

o   User interaction (how would you expect this to work?)

o   User interface design (does this wording or picture, etc. make sense to you? Can you suggest something better?)

·       A thank you gift! After participating in a study, you’ll have the option of selecting from a list of Microsoft software, hardware, games, and more. Since IT Pros are the hardest folks to find, you get the best gifts!

If you’d like to be considered for the IT Pro User Panel, please complete this brief survey.

If you want to know more about Microsoft User Research overall, see the Microsoft User Research page.

Note: Microsoft full or part-time employees, vendors, or contingent employees are not eligible.

Azure IT Camps - Coming to a City Near You!

Do you want to learn more about Windows Azure and System Center 2012 R2?  Now is your chance!



Join us at this FREE full-day hands-on event in a city near you to experience the power of Hybrid Cloud. Our field-experienced Technical Evangelists will guide you through the process of jumpstarting your knowledge on Windows Azure Storage, Virtual Machines and Virtual Networking for key IT Pro scenarios. Complete all of the hands-on labs and you’ll walk away with a fully functional Windows Server 2012 R2 or Linux cloud-based test lab running Windows Azure!

Session Requirements
Be sure to bring a modern laptop that is capable of running the following prerequisites. For more detailed system specs, click on the city nearest you.

• Modern operating system, including Windows 7, Windows 8, Linux or Mac OS X
• Modern web browser supporting HTML5 and Javascript, including IE 9 or later, Chrome, Firefox and Safari
• A remote desktop (RDP) client – included with Windows platforms. Mac and Linux RDP clients can be downloaded for free

All participants registering for the event should have an active Windows Azure subscription. If you have not already done so, sign up for a FREE trial of the Windows Azure platform and services, including access to Virtual Machines and Virtual Networks.



Hope to see you there!

Question: Is there value in testing your Disaster Recovery Plan?

Answer: Only if you want a shot at it actually working when you need it.

There are a few reasons you need to regularly test your recovery plans… I’ve got my top three.

1. Backups only work if they are good.
2. Your documentation is only useful if you can follow it.
3. You are soft and easily crushed.

Backups
Everyone knows the mantra of “backup, backup, backup” but you also have to test those backups for accuracy and functionality. I’m not going to beat this one endlessly, but please read an old post of mine - “Epic Fail #1” to see how backups can fail in spectacular, unplanned ways.

Documentation
Simply put, you need good documentation. You need easy to locate lists of vendors, support numbers, configuration details of machines and applications, notes on how “this” interacts with “that”, what services have dependencies on others and step by step instructions for processes you don’t do often and even those you DO do everyday.

When under pressure to troubleshoot an issue that is causing downtime, it’s likely you’ll loose track of where to find information you need to successfully recover.  Having clean documentation will keep you calm and focused at a time you really need to have your head in the game.

Realistically, your documentation will be out of date when you use it.  You won’t mean for it to be, but even if you have a great DR plan in place, I’ll bet you upgraded a system, changed vendors, or altered a process almost immediately after your update cycle. Regular review of your documents is a valuable part of testing, even if you don’t touch your lab.

My personal method is to keep a binder with hard copies of all my DR documentation handy.  Whenever I change a system, I make a note on the hard-copy. Quarterly, I update the electronic version and reprint it.  With the binder, I always have information handy in case the electronic version is not accessible and the version with the handwritten notes is often more up to date with the added margin notes. Even something declaring a section “THIS IS ENTIRELY WRONG NOW” can save someone hours of heading down the wrong path.

You
No one wants to contemplate their mortality, I completely understand. (Or maybe you just want to go on vacation without getting a call half way through. Shocker, right?) But if you happen to hold the only knowledge of how something works in your data center, then you are a walking liability for your company. You aren't securing your job by being the only person with the password to the schema admin account, for example. It only takes one run in with a cross-town bus to create a business continuity issue for your company that didn't even touch the data center.

This extends to your documentation. Those step-by-step instructions for recovery need to include information and tips that someone else on your team (or an outside consultant) can follow without having prior intimate knowledge of that system.  Sometimes the first step is “Call Support, the number is 800-555-1212” and that’s okay.

The only way to find out what others don’t know is to test.  Test with tabletop exercises, test with those backup tapes and test with that documentation.  Pick a server or application and have someone who knows it best write the first draft and then hand it to someone else to try to follow. Fill in the blanks. Repeat. Repeat again.

A lot of this process requires only your time. Time you certainly won’t have when your CEO is breathing down you neck about recovering his email.

Additional Resources
This is post part of a 15 part series on Disaster Recovery and Business Continuity planning by the US based Microsoft IT Evangelists. For the full list of articles in this series see the intro post located here: http://mythoughtsonit.com/2014/02/intro-to-series-disaster-recovery-planning-for-i-t-pros/

If you are ready to take things further, check out Automated Disaster Recovery Testing with Hyper-V Replica and PowerShell - http://blogs.technet.com/b/keithmayer/archive/2012/10/05/automate-disaster-recovery-plan-with-windows-server-2012-hyper-v-replica-and-powershell-3-0.aspx

Happy Valentine's Day From TechNet Radio!

Its time to break out the “We ♥ PowerShell” candy and cuddle up with your favorite PowerShell cmdlets as Dr. Love…er…we mean Dr. Scripto, joins us for this special Valentine’s Day edition of TechNet Radio. Tune in as Blain Barton and Ed Wilson dish out the PowerShell love advice and give us their Top 14 things they love about Windows PowerShell.

• [1:17] I have a lot of workloads out there, is Windows PowerShell everywhere I need it to be?
• [2:12] Is Windows PowerShell really readable code?
• [3:10] Is it easy to get started with Windows PowerShell?
• [5:54] Is Windows PowerShell well documented via the community, and via books and blogs? 
• [7:32] I think I’m addicted to Windows PowerShell.  Should I see a doctor?
• [9:12] We are on version 4.0 now- does PowerShell get better with age?
• [10:16] Is Windows PowerShell easy to use? 
• [11:08] I’ve been using PowerShell forever, but have some newbies that want to learn, is it easy to get help
• [12:57] Is Windows PowerShell Self Discoverable? 
• [14:06] Is Windows PowerShell Self Documenting?
• [15:39] Does Windows PowerShell really save IT time
• [17:38] Does Windows PowerShell really save IT money?
• [20:12] Is knowing Windows PowerShell good for job security? 
• [20:56] I don’t always script, but when I do, should I use PowerShell and why? 
• [23:08] Are PowerShell and Azure meant for each other?
• [24:54] A PowerShell Love Poem: “How do I love thee PowerShell….let me count the ways…”

Here is a direct link to the episode:

http://channel9.msdn.com/Shows/TechNet+Radio/TechNet-Radio-IT-Time-Valentine-s-Day-Special-The-14-Things-we-LOVE-about-PowerShell

From my Friends over at Microsoft Virtual Academy

9/23/2014 Update - This event is available on demand, when you need it.  If you are just starting to take a closer look at Windows Server and Hyper-V, this is a great way to get a jumpstart.

----------------------------------------------------------------------------------------

Free Online Event: Virtualizing Your Data Center with Hyper-V and System Center

Free online event with live Q&A: http://aka.ms/virtDC

Wednesday, February 19^th from 9am – 5pm PST

If you're new to virtualization, or if you have some experience and want to see the latest R2 features of Windows Server 2012 Hyper-V or Virtual Machine Manager, join us for a day of free online training with live Q&A to get all your questions answered. Learn how to build your infrastructure from the ground up on the Microsoft stack, using System Center to provide powerful management capabilities. Microsoft virtualization experts Symon Perriman and Matt McSpirit (who are also VMware Certified Professionals) demonstrate how you can help your business consolidate workloads and improve server utilization, while reducing costs. Learn the differences between the platforms, and explore how System Center can be used to manage a multi-hypervisor environment, looking at VMware vSphere 5.5 management, monitoring, automation, and migration. Even if you cannot attend the live event, register today anyway and you will get an email once we release the videos for on-demand replay! 

Topics include:

• Introduction to Microsoft Virtualization
• Host Configuration
• Virtual Machine Clustering and Resiliency
• Virtual Machine Configuration
• Virtual Machine Mobility
• Virtual Machine Replication and Protection
• Network Virtualization
• Virtual Machine and Service Templates
• Private Clouds and User Roles
• System Center 2012 R2 Data Center
• Virtualization with the Hybrid Cloud
• VMware Management, Integration, and Migration

 Register here: http://aka.ms/virtDC 

Also check out the www.MicrosoftVirtualAcademy.com for other free training and live events.

Disaster Recovery for IT Pros: How to Plan, What are the Considerations?

I've done a little disaster recovery planning in my day. As an IT Professional, it's really easy to get caught up in the day-to-day. We have users that need assistance, servers that need love (updates), applications that need upgrading... whatever today's problem is, it needed solving yesterday. Disaster Recovery is often the elephant in the room, the insurance you don't have time to buy. Everyone knows it's needed, no one ever wants to use it and often, there is no clear way to begin.

I've always thought that being an IT Pro is one of the most powerful, powerless jobs in existence. We have our fingers on the pulse of what makes our businesses run, we have access to ALL THE DATA and we have the power to control access and availability to the resources. But we are often slaves to the business - we are responsible for providing the best up times, the best solutions and the best support we can. Facing budgets we can't always control while trying to explain technology to people who don't have time to understand it.

So where do you begin when tasked with updating or creating your disaster recovery plan? The good news is you don't need money or lots of extra hardware to start good disaster recovery planning - grab the note-taking tools of your choice and start asking questions.

Here are my three main questions to get started:

1. What is the most important application or services in each business unit or for the business overall?
2. How much downtime is acceptable?
3. How much data loss is acceptable?

These are your considerations. Period. I didn't mention money, but I know you want to argue that you can't recover without it. And that is true.  But until you know what your goal is, you have no idea how much it may or may not cost.

This post is one of many in disaster recovery series being penned by the IT Pro Evangelists at Microsoft. As the series progresses, you'll find the complete list on Brian Lewis's blog post, "Blog Series: DR Planning for IT Pros." We will cover tools and applications you can consider in you planning and get you started with using them. They have various costs, but until you know your goal, you won't know what tools will help and can't argue the budget.

So let's put the pencil to the paper and start answering those three questions.

Start at the top:  Go to upper management, have your CTO or CIO to pull together a leadership meeting and rank what systems the business units use and what they think is needed first. Get them to look at the business overall and determine how much downtime is too much, how quickly do they want services recovered and how much data they are willing to lose.

When it comes to determining your internal SLA you do need to know what scenario you are planning for. Preparing for a riot that blocks access to your office is different than an earthquake that renders your data center a steaming pile of rubble. Ultimately, you want different plans for different scenarios, but if you must start somewhere, go with the worst case so you can cover all your bases.

But what if you can't get leadership to sit down for this, or they want you to come to the table first with draft plan. Just GUESS.

Seriously, you have your hand on the data center, you know the primary goals of your business. If it was your company, what do you think you need to recover first? Use your gut to get you started. Look at your data center and pick out some of the key services that likely need to be recovered first to support the business needs. Domain controllers, encryption key management systems, infrastructure services like DNS and DHCP, communication tools and connectivity to the Internet might float to the top.

Sort the List: People want email right away? Great, that also needs an Internet connection and access to your authentication system, like Active Directory. People want the document management system or CRM or some in-house app with a database back-end? Fabulous, you need your SQL Servers and maybe the web front-end or the server that supports the client application.

Gather Your Tools: Look at your list of loosely ranked servers, devices and appliances and start building a shopping list of things you need to even start recovery. I always start with the "steaming pile of rubble" scenario, so my list starts like this:

1. Contact information for hardware and software vendors
2. Contact information and locations where my data center can function temporarily
3. List of existing hardware and specifications that would need to be met or exceeded if ordering new equipment for recovery
4. List of operating systems and other software, with version details and support documentation
5. Names of the people in the company that would be crucial to the successful recovery of the data center

Type this all up. If any of the things listed above involve looking at a server or visiting a web page, remember that in the "pile-of-rubble" scenario you will likely not have access to those resources. Save it wherever you save this type of documentation. Then print out a copy and put it in a binder on your desk. Print out another copy, seal it in an envelope and take it home.

Congratulations! You are closer to a usable DR plan than you were before you started and we've just scratched the surface. Disaster Recovery planning is often pushed off until tomorrow. Whatever you have today, be it an outdated document from your company leadership, server documentation that is a year old, or NOTHING, you can take time each day to improve it. How you plan is going to depend on the needs of your organization and you won't be able to complete the process in a silo, but you can get started.

I really enjoy disaster recovery planning. It's challenging, it's ever changing and I haven't even mentioned how things like virtualization, Hyper-V Replica and Azure can be some of the tools you use. Stay tuned for more in the series about how some of those things can come into play. Sometimes the hardest part about disaster recovery planning is just getting started.

***

This is post part of a 15 part series on Disaster Recovery and Business Continuity planning by the US based Microsoft IT Evangelists. For the full list of articles in this series see the intro post located here: http://mythoughtsonit.com/2014/02/intro-to-series-disaster-recovery-planning-for-i-t-pros/

Where Have You Been All My Life?

My subject line might be a bit dramatic, but the last two weeks has been a busy whirlwind of travel for technical training and team building.  Microsoft has announced the new CEO, I've got a growing list of meet-ups and community groups I want to check out in 2014 and my days have just been flying by.   My head is filling with ideas for things to bring to you on Techbunny, but I haven't had the time to write them down.

But don't fret, you'll have plenty to read about starting on Monday. I promise. Particularly if disaster recovery planning is on your list!

Check back here or follow me on Twitter @jkc137.  And did you know I'm on Facebook? Yep. That too. 

Have a great weekend!