Goodbye Windows Server 2003!

Time flies and today marks end of support for Windows Server 2003.  In case, you missed some of the available information to help you migrate onto a more modern copy of Windows Server, here are some links!

Listen to @RicksterCDN's letter to Windows Server - The End of a Affair... 

On-Demand Events

• Windows Server 2003 Migration: Hardware Modernization: http://aka.ms/WS03blog
• Windows Server 2003: Most Common Application Migration Concerns: http://aka.ms/commonblog
• Windows Server 2003: Security Risk and Remediation: http://aka.ms/remeblog
• Windows Server 2003 End of Support Migration Overview
• How to Assess Your Windows Server 2003 Enviroment
• Modernizing Your Datacenter Jump Start

Windows Server 2003 Still Around? Check out some useful webcasts!

Ah, Windows Server 2003, that sturdy workhorse that just keeps going and going.  If you still have a server or two chugging along, you might find some of these upcoming webcasts of interest.

• 1/22/15 - Windows Server 2003 Migration: Hardware Modernization: http://aka.ms/WS03blog
• 2/5/15 - Windows Server 2003: Most Common Application Migration Concerns: http://aka.ms/commonblog
• 2/18/15 - Windows Server 2003: Security Risk and Remediation: http://aka.ms/remeblog

For additional information, you can also visit Microsoft Virtual Academy for some on-demand courses.

• Windows Server 2003 End of Support Migration Overview
• How to Assess Your Windows Server 2003 Enviroment
• Modernizing Your Datacenter Jump Start

Microsoft Virtual Academy - Modernizing Your Data Center JumpStart is Coming!

Worried about Windows Server 2003 end of support?

Fear no more. You can get ahead of the game with a look at modernization and data center transformation options in Windows Server 2012 R2 and Microsoft Azure. Want to decide what works best in your environment? Check out “Modernizing Your Data Center,” on October 28, and learn from the experts exactly what you need to update your data center to match your workloads.

Join me and Matt Hester for a look at administration tools, storage improvements, Hyper-V, and best practices for virtualizing domain controllers, plus how to simplify day-to-day server management with PowerShell and Desired State Configuration. Explore Microsoft Azure and how to make hybrid cloud a reality. And review the four major steps for planning a migration project. 

Take this opportunity to get your modernization questions answered!

Course Outline

·       Windows Server 2012 R2 and Automation

·       Azure Infrastructure as a Service

·       Migration Processes, Roles, and Tools

Register now!

Modernizing Your Data Center Jump Start

Date:  October 28, 2014
Time: 9am‒2pm PDT
Where: Live, online virtual classroom
Cost: Free!

Throwback Thursday: Sessions from TechEd Houston

Today is my final installment of highlights from TechEd Houston! Below are some of my session picks from the last day of the conference.

• TWC: Hacker's Perspective on Your Windows Infrastructure: Mandatory Check List (DCIM-B366)
• Windows 8 Security Internals (WIN-B350)
• Real-World Windows 8.1 Deployment Note from the Field (WIN-B358)
• Providing SaaS Single Sign-on with Microsoft Azure Active Directory (PCIT-B326)
• Delivering Disaster Recovery Solutions Using Windows Server 2012 R2, Microsoft System Center 2012 R2 and Microsoft Azure (DCIM-B421)
• How IPv6 Impacts Private Cloud Deployments (DCIM-B373)
• Windows Server 2003 End of Life Migration Planning for Your Workloads (DCIM-B376)
• Upgrading Active Directory the Safe Way: Using Virtualization Technologies (PCIT-B341)

For my lists of sessions from the other days, you can find them here: Monday, Tuesday and Wednesday.

Microsoft End of Life Dates - Mark Your Calendars!

Where is 2012 going?  It seems like just yesterday I filed away my planner for 2011 and crack open that fresh page to January 2012. Now that we are racing towards Spring, you might want to highlight a few of these special dates for the future.

Here are some future "end of life" dates for some Microsoft products you might still have floating around on your network.  Some will be supported for several more years, but it never hurts to keep your eye on the horizon.

These dates are the end of support life for the product as a whole (no more extended support), so start thinking about your budget cycles and internal support needs for the next few years.

Windows XP - 4/8/2014
Server 2003 - 7/14/2015
Windows Vista - 4/11/2017

Exchange Server 2007 - 4/11/2017
SQL Server 2000 - 4/9/2013
SQL Server 2005 - 4/12/2016

Office 2003 - 4/8/2014
Office 2007 - 10/10/2017

These dates are for specific service packs for these products, so be sure to install the latest available service pack, if you haven't already.

SQL Server 2005 SP 3 -1/10/2012
Exchange 2010 SP 1 - 1/8/2013
Office 2007 SP 2 - 1/8/2013

For more information about other Microsoft Server products, check out the Lifecycle Info for Server Products list. - http://support.microsoft.com/gp/lifeSelectServ

** 11/21/14 Update **

For some current end of life dates - visit this post.  Interested in learning more about getting away from on-prem Exchange and Office?  Check out these courses from the Microsoft Virtual Academy -

• Managing Exchange Online Using the Console 
• Managing Exchange Online Using PowerShell 
• Managing Office 365 Identities and Services
• Office 365 ProPlus Deployment for IT Pros

Replication Warnings? - It could be just one Attribute.

Active Directory can be a funny beast.  This week, I noticed a reoccuring replication error that didn't seem to be sorting itself within a reasonable time frame.  I was seeing NTDS Replication Warning 1083, referencing a specific user account: 

Event Type: Warning
Event Source: NTDS Replication
Event Category: Replication
Event ID: 1083
Date:  10/3/2011
Time:  11:45:00 AM
User:  NT AUTHORITY\ANONYMOUS LOGON
Computer: DC1
Description:
Active Directory could not update the following object with changes received from the domain controller at the following network address because Active Directory was busy processing information.

Object:
CN=Joe Smith,OU=Accounts,DC=mydomain,DC=org
Network address:
a5b5b72d-c74b-486a-9dfa-f6516f37b38b._msdcs.caclo.org

Following it was the informational event 1955 about a write conflict:

Event Type: Information
Event Source: NTDS Replication
Event Category: Replication
Event ID: 1955
Date:  10/3/2011
Time:  11:45:00 AM
User:  NT AUTHORITY\ANONYMOUS LOGON
Computer: DC1
Description:
Active Directory encountered a write conflict when applying replicated changes to the following object.

Object:
CN=Joe Smith,OU=Accounts,DC=mydomain,DC=org
Time in seconds: 0 

After some research I tried the following troubleshooting steps:

1) Moved the offending user to a different OU temporarily to see if the problem resolved.  This essentially "tickles" AD into replicating that particular user. I recieved the same messages, but the user's CN had been updated to the new OU.
2) Used the LDP tool to see if there was duplicate entries for this user somehow, but only one instance was found.
3) Used repadmin to look at the time stamps of various attributes on the account, particular one with a time stamp close to the time that the replication warnings started appearing in the event log.

Repadmin was where I had the most luck.  You'll want to run the following command for Windows 2003 SP2 DCs:

repadmin /showobjmeta DC1 "CN=Joe Smith,OU=Accounts,DC=mydomain,DC=org"

This will return a list of attributes with timestamps.  In my case it was the attribute related to the last password change, which was the only one that had a timestamp of the same date when the errors began.  I reset the password on the account to "tickle" that particular attribute and the replication completed without any complaint.

Some anticodotal stories on the Internet indicate that this attribute can cause trouble if replication occurs while an account happens to be locked out.  In this case, the account was for a consultant who didn't log in very often, so the locked account went unnoticed for some time, causing the replication issue.

Tackling Windows 2003 Server Space Issues

Got a Windows 2003 server with a small hard drive that keeps filling up? Make sure to check out these potential space hogs:

1. The Framework.log file in the %systemroot%\system32\wbem\logs folder. This file has the potential to grow out of control, but that problem can be easily remedied with a quick permissions change. Check out KB836605 for details.
2. Some auditing and logging applications might be making backups of your Event Logs, which often end up in your %systemroot%\system32\config folder. Check for .EVT files you no longer need so you can move or delete them.

Finally, not sure what taking up the most space? Check out the free tool called WinDirStat for a quick visual mapping of what's taking up the most space.

The Post-Mortem of a Domain Death

The past few days have been busy as we've been performing the tasks to remove our failed domain controller and domain from our Windows 2003 Active Directory forest.  Now that everything is working normally and I can check off that long-standing IT project of "remove child domain" from my task list, I'd like to share a few things we've learned.

• NTDSUTIL will prompt you several times when it comes to removing the last DC in a domain using the steps in KB 216498. It will even hint that since you are removing the DC in the domain, that you are also removing the domain itself.  But you are not.  You must take additional steps in NTDSUTIL to remove the orphaned domain, see KB 230306 to finish up.
• How do you know you have an orphaned domain? Check AD Domains and Trusts.  If you still see a domain in your tree that you can't view the properties of, you aren't done yet.  Also, if your workstations still show the domain as a logon option in the GINA, get back to work.
• You might remember to clean up your DNS, but don't forget to also clean up WINS.  WINS resolution can haunt you and keep your workstations and applications busy looking for something that isn't there anymore.
• Watch your Group Policy links.  If you've cross-linked policies from the child domain to your forest root or other domains, workstations will indicate USERENV errors referencing the missing domain.  Policies from other domains won't show up in your "Group Policy Objects" container the GPMC.  You'll need to expand all your other OUs in the GPMC to find any policy links that report an error. 
• If you are using a version of Exchange that has the infamous Recipient Update Service, remove the service entry that handles the missing domain.  You'll see repeated MSExchangeAL Events 8213, 8250, 8260 and 8026 on your mail server otherwise.

I've used NTDSUTIL in the lab and in production several times to remove failed domain controllers, but removing an orphaned domain happens far less frequently.  While the majority of our Microsoft applications handled the existence of references to the orphaned domain with grace until we completed the clean up, one of our third party applications, ImageRight, was far more sensitive about it. 

We found that a combination of the WINS resolution and the orphaned trust relationship distracted the application enough that it was slow to operate, sometimes refused to load at all, and hung on particular actions.  If you happen to be an ImageRight customer who uses the Active Directory integration features, keep in mind that it likes all the AD ducks to be in a row.

While we had a little a bit of pain getting to this point, I'm really happy that our AD forest is neater and cleaner because of it.  It'll make it much easier to tackle other upgrade projects on the horizon for Active Directory and Exchange.

Windows Server July 2010 Support Changes

On July 13, 2010, serveral Windows Server products will hit new points in their support lifecycle. Windows 2000 Server will move out of Extended Support and will no longer be publicly supported. Windows Server 2003 and Server 2003 R2 will be moving from Mainstream Support to Extended Support. Extended Support will last another 5 years.

This forces a new deadline on the some of the improvements that need to be planned at my office. Our phone system and our main file server are still operating on 2000 Server. I have been planning to upgrade the phone system for a long time now, but it continually gets pushed back due to other more pressing projects. Our file server is an aging, but sturdy, HP StorageWorks NAS b3000 - "Windows-powered" with specialized version of 2000 Server. Both deserve more attention than they've been getting lately, so now there is a reason to move those items higher up on the list.

For more information about these support changes, check out "Support Changes Coming July 2010" at the Windows Server Division Weblog.

MS Security Advisory

Keep an eye out for this one since there isn't a fix yet, outside of a workaround disabling some COM objects in the registry for Windows XP and Windows Server 2003.

Microsoft Security Advisory (972890) - Vulnerability in Microsoft Video ActiveX Control Could Allow Remote Code Execution

I suspect we'll see a patch very soon. Vista and Windows Server 2008 are not affected, so those running the Windows 7 RC are likely safe too!

NTDS Error 2103

This week one of my domain controllers developed a curious problem. I don't like curious problems, especially ones that rear their heads after the server reboots.


The error was an NTDS General event 2103, which indicates that the AD database "was restored using an unsupported procedure and Net Logon service has been paused". Research and KB Article 875495 lists event 2103 and 3 other events related to a condition known as USN Rollback.

This DC is running Windows 2003 SP2, so based on the article, I should be seeing at least the more serious NTDS Replication 2095 event as well, due to a hotfix in SP1 that made the error logging somewhat more verbose. But I'm not. This makes it more curious. Am I in a rollback state or not?

KB 8759495 also lists some possible causes of this state, some of which are possible in a virtual environment - the case for this DC. It points me to another KB Article 888794 which lists out a bunch of considerations for hosting DCs as VMs. However our environment met all the requirements, including one related to write caching on disks, as our host machine has battery backed disk caching. So I rule out that we actively caused a potential rollback.

Repadmin has a switch (/showutdvec) that can be used to determine USN status by displaying the up-to-dateness vector USN for all DCs that replicate a common naming context. If the direct replication partners have a higher USN for the DC in question than that DC has for itself, that's considered evidence of a USN rollback. My DC did not have this problem, as it had a USN higher than it's partners. So at this point I couldn't confirm or deny a true USN rollback issue, however it seemed the the DC "thought" it was having this problem. Maybe I could figure out why the DC was in this limbo.

So I returned to the original article to look for specific causes. One line reads, "Starting an AD domain controller whose AD database file was restored (copied) into place by using an imaging program such as Norton Ghost."

Thinking back, the conversion of this DC from physical to virtual did not go as smoothly as I would have hoped. I remembered I had to resolve some issue where I was getting an error in the logs related to the directory database file not being where the OS expected it, even though the path on the server hadn't changed during the conversion. It was odd at the time, but the posted fix seemed to clear the issue and I'd moved on.

I'm guessing that perhaps that was the start of my issues - maybe the P2V process made the OS think the database was different copy even though it wasn't. The result was that the server thought it was rolled back, but the USNs never reflected a problem. So I decided it was better to be safe than sorry and assume this "limbo" condition was not how I wanted to leave things.

The resolution for USN rollback is a forced removal of the domain controller from AD. Since this is a DC in a child domain that's being phased out, very few changes happen to that domain so I wasn't concerned about possibly loosing changes that may have been made on that DC. It was only the FSMO holder for one role which was easily seized by the other DC.

My decision now is to decided between bringing up a replacement DC for this domain next week or just run one DC for the time being and try to speed up the remaining tasks that need to be done before we can removed the child domain all together.

But that's for another day!

Immediately = 15 Minutes

Yesterday: One of my office domain controllers, ROOTDC01, failed. Not so much that things stopped working when it failed, but it left us open to serious downtime if it's partner, ROOTDC02, failed before we had replaced the first one. I decided that it didn't make sense to bring a replacement Windows 2000 domain controller in, only to proceed with our planned domain controller upgrade project in about 4 weeks. It only made extra work. This was the (sort of) perfect opportunity to bring in a shiny new already Windows 2003 DC into the organization. And it would also force me to finally "walk the walk" after quite a few months of "talk" (and testing!).

Earlier Today: This evening, a co-worker and I started on upgrading the schema in our organization to support this shiny new DC. This process, which happens on ROOTDC02 (the remaining DC), is relatively simple on paper and successful 99.9% of the time. But it could do major damage the other .01%. And since I didn't have an 2nd DC to act as a backup, a screw-up could leave me doing a lot of disaster recovery. For many many hours.

All I really had to do was follow the step-by-step directions that I prepared for myself during the testing phases. And then, of course, second guess my directions. Wring my hands, close my eyes tightly and pace around while things were happening. And in some cases, when Microsoft documentation says "immediately" they really mean "give it 15 minutes to stew a bit." This is when most of the pacing happens. And rapid refreshing of my replication monitor application.

Now: Everything seems to have gone nicely. No system errors that weren't expected. No crashes, no blips. It's only the year 2006 and I've finally gotten around to getting our systems up to 2003.

Monday: Bring in ROOTDC03, the new partner for ROOTDC02. We are still in a touchy spot over the weekend - but I think we'll be fine. Once that new server is running, I can start upgrading the other DC to Windows 2003... might even finish the whole project before my deadline.