The Imperfect Lab: Syncing AD to Azure AD

Today I decided to ease myself into my next steps and build out a member server to sync AD to.  I reused some previous PowerShell to deploy a member server and join it to my domain.  It is possible to run the sync services on an existing domain controller, but as a best practice I don't like to install one-off applications on my domain controllers.  I like to keep them identical, thus the need for different member server to perform the sync role.

I had previously uploaded the Microsoft Azure AD Sync Services (aka AADSync) application to my Azure file share, but you can find it at http://aka.ms/azureadsync.  You will want to install and run the Microsoft Azure AD Connection Tool.  Please note that Microsoft Azure AD Sync Services is DIFFERENT from Windows Azure Active Directory Sync (aka DirSync)

Once the Sync Server is built, you will want to kick off the installation of the application, but not before you'd made some adjustments to your Azure Directory.  In the Portal, I went to my directory and created a new user account to be my Azure AD Administrator (newuser@imperfectlab.com) and made it a Global Administrator.  You will also need to go through the sign-in process to set a non-temporary password.

Once you have this account, you simply need to throw the switch under "Directory Integration -> Directory Sync" from Inactive to Active.  Once the setting is saved, the "Last Sync" field will say "never synced".  Now go over to your sync server and run that connection tool.

You'll need the account and credentials you created for the new Azure AD Admin and some information about your domain.  For the addition of the forest, you'll need your domain name and the username and password of a enterprise domain admin from your local domain.  This will be different than the account your created directly in Azure AD.

Leave the User Matching page at the defaults but select "Password Synchronization" from the Optional Features. Finally, review your configuration screen and verify that "Synchronize Now" is checked and click finish.  At this point, your users should sync into Azure AD and after a few minutes you'll see a list of them in the portal.

If you want to make any changes to the settings of your AD Sync, like adding in a feature, simply rerun the tool after disabling the Azure AD Sync Task in Task Scheduler.  The task will be re-enabled automatically when you finish the wizard again.

If you want to force a sync for Azure AD Sync Services for any reason, the default location of the command line tool is:

c:\program files\microsoft azure ad sync\bin\directorysyncclientcmd [initial|delta]

Happy Syncing!

The Imperfect Lab: Adding A Custom Domain

This will be a super short post, because this task is super easy!

My lab in Azure wouldn't be complete without its own custom domain. Honestly, this is one of those "just pop over to the Portal" tasks because it only takes a few clicks, particularly if you are only doing it once.  But you won't be able to complete in a hurry, because your registrar will update the public DNS entries on their own sweet time and that update is needed to complete the process.

By the way, if you really want to do this without the Portal, you can find information on installing the right PowerShell modules and the commands here. (http://msdn.microsoft.com/en-us/library/azure/jj151815.aspx)  If you are going to managing multiple tenants over time, PowerShell will likely be the best way to go.

Anyway, when you are in the Portal, click "Active Directory" in the navigation.  Select the domain directory you want to add a custom domain to.  In this case, I wanted to create a new Azure Directory for the Imperfect Lab, so I clicked "New" and then went to APP SERVICES -> ACTIVE DIRECTORY -> DIRECTORY -> CUSTOM CREATE.

I named my directory "ImperfectLab" and picked my region.  The domain name for the directory is now "Imperfectlab.onmicrosoft.com".  Since I don't want to be using the "onmicrosoft.com" moniker for very long, I need to add my recently purchased domain.  You actually have to a own (or at least control) the domain you want to add because it's requirement to add a TXT or MX record to your public DNS.

Click into the directory you want to use and go to the "Domains" section. On the bottom action bar, click "Add". Then type in the FQDN for your "real" domain, in my case "imperfectlab.com".  You be given the information to create either at TXT or MX record that needs to be added to your DNS records managed by your registrar.

My registrar doesn't accept the @ symbol for the parent zone, but leaving that field blank worked fine.  You have to add the record, wait for the external DNS to update and then return to the portal to verify it.

Once verified, you can create (or sync) users into your Azure Active Directory using either your "user@domain.onmicrosoft.com" UPN or your "user@domain.com" UPN.

The Imperfect Lab: Fleshing Out Active Directory

Having a domain with no users isn't any fun.  So my next task for the Imperfect Lab was to create a few accounts to act as my users for provisioning access and eventually syncing with Azure Active Directory.

You can do a lot with some basic PowerShell to create OUs and User Accounts.  Here are a few basic lines that would create something in my lab domain:

New-ADOrganizationalUnit –Name “DOGS” –Path “DC=imperfectlab, DC=Com”

New-ADUser -Name "Lizbeth Tiburon" -Path "OU=DOGS,dc=imperfectlab,dc=com" -AccountPassword $newPassword -Department "Career Changed" -SamAccountName "LTibu" -Surname "Tiburon" -GivenName "Lizbeth" -DisplayName "Lizbeth Tiburon"

Those lines would create a OU and then a user account in the new OU.  But what if you wanted to create more users at once?  I could simply duplicate the 2nd line, but figured there had to a relatively easy way to get data straight from a CSV file.

I did some looking around online and since no good Internet search goes unpunished, I found this: https://gallery.technet.microsoft.com/scriptcenter/PowerShell-Create-Active-7e6a3978#content by @mwashamtx.  Honestly, this a great script that I couldn't have written by myself at this point, but I was able to tweak it enough to do my bidding. 

I changed the paths (to reflect the drive letter and file location I set up using Azure Files), removed a lot of the fields the script used to populate account attributes and edited the CSV file to match.  I uploaded my CSV file to my Azure file share. I left the script writer's five character SAM account name creation as is and ran it remotely via PS-Session on my domain controller.  The DC tapped the CSV file in my Azure File share and wrote the log to that same location.  The script does some great error handling, which was really helpful for troubleshooting.  Mission accomplished!

And for those of you who are curious about the user created in that line above, Lizbeth is a dog who didn't complete the training to become a guide dog. 

The Imperfect Lab: More DCs and Static IPs

When I was last working in my Imperfect Lab, I added another server to the existing cloud service and decided to make it a domain controller.  When you set up domain controllers (cloud or on-premises) a few things become really important - IP Addresses and DNS.

By default, Azure will provide DNS services from the fabric if you don't specify your own DNS.  You would think there is some PowerShell to do that directly, but surprisingly there isn't.  You can set the DNS for each network using the Management Portal or by exporting the network configuration file and updating it.  I just used the portal and made sure that my ImperfectNet listed the IP address for both servers that would act as domain controllers.

If you don't set a domain controller as the DNS server, all the VMs that come up inside your virtual network will look to an Azure fabric DNS server and won't be able to authenticate to your domain.  Since this is a crucial to AD function, I also wanted to make sure that the VMs that were acting as domain controllers had static internal IP addresses. 

Now, these addresses aren't really "static" on the OS. They are more like DHCP reservations handed out from the fabric manager.  But the end result is the same - VMs that have the correct IP address, regardless of the order they are started.

To do this with PowerShell, you first need to have the VMs in the Stopped (Deallocated) state. This way the addresses are free to assign.  If the VM is already running, the address is allocated already, thus can't be assigned.  You can double check that an address is free with:

Test-AzureStaticVNetIP –VNetName ImperfectNet –IPAddress 192.168.1.5

To set the static address, I used:

Get-AzureVM -ServiceName ImperfectCore -Name DC-Cloud1 | Set-AzureStaticVNetIP -IPAddress "192.168.1.4" | Update-AzureVM

Take note of the use of quotes around the IP address in that last line. It matters. I don't know why.  Just trust that I wasted a lot of time on your behalf for that knowledge.

Then to finally kick off the addition of my second domain controller in this domain, I used:

Install-ADDSDomainController -Credential (Get-Credential) -DatabasePath 'C:\Windows\NTDS' -DomainName 'imperfectlab.com' -InstallDns:$true -LogPath 'C:\Windows\NTDS' -NoGlobalCatalog:$false -SiteName 'ImperfectNet' -SysvolPath 'C:\Windows\SYSVOL' -NoRebootOnCompletion:$true -Force:$true -Verbose

One note about the paths used for the logs and SYSVOL… I've left them on C:\ for convenience, but for production, you will want to set up your DCs in Azure with an additional disk where you direct those files to go.  Read more about the reason behind that best practice here.

Also, if this Domain Controller happens to connect back to an on-premises domain. Be sure to make the proper changes to you AD Sites and Services to ensure proper site topology.

Update (12/26/14): For easy access to code snippets, you can find them here.

Throwback Thursday: Sessions from TechEd Houston

Today is my final installment of highlights from TechEd Houston! Below are some of my session picks from the last day of the conference.

• TWC: Hacker's Perspective on Your Windows Infrastructure: Mandatory Check List (DCIM-B366)
• Windows 8 Security Internals (WIN-B350)
• Real-World Windows 8.1 Deployment Note from the Field (WIN-B358)
• Providing SaaS Single Sign-on with Microsoft Azure Active Directory (PCIT-B326)
• Delivering Disaster Recovery Solutions Using Windows Server 2012 R2, Microsoft System Center 2012 R2 and Microsoft Azure (DCIM-B421)
• How IPv6 Impacts Private Cloud Deployments (DCIM-B373)
• Windows Server 2003 End of Life Migration Planning for Your Workloads (DCIM-B376)
• Upgrading Active Directory the Safe Way: Using Virtualization Technologies (PCIT-B341)

For my lists of sessions from the other days, you can find them here: Monday, Tuesday and Wednesday.

Replication Warnings? - It could be just one Attribute.

Active Directory can be a funny beast.  This week, I noticed a reoccuring replication error that didn't seem to be sorting itself within a reasonable time frame.  I was seeing NTDS Replication Warning 1083, referencing a specific user account: 

Event Type: Warning
Event Source: NTDS Replication
Event Category: Replication
Event ID: 1083
Date:  10/3/2011
Time:  11:45:00 AM
User:  NT AUTHORITY\ANONYMOUS LOGON
Computer: DC1
Description:
Active Directory could not update the following object with changes received from the domain controller at the following network address because Active Directory was busy processing information.

Object:
CN=Joe Smith,OU=Accounts,DC=mydomain,DC=org
Network address:
a5b5b72d-c74b-486a-9dfa-f6516f37b38b._msdcs.caclo.org

Following it was the informational event 1955 about a write conflict:

Event Type: Information
Event Source: NTDS Replication
Event Category: Replication
Event ID: 1955
Date:  10/3/2011
Time:  11:45:00 AM
User:  NT AUTHORITY\ANONYMOUS LOGON
Computer: DC1
Description:
Active Directory encountered a write conflict when applying replicated changes to the following object.

Object:
CN=Joe Smith,OU=Accounts,DC=mydomain,DC=org
Time in seconds: 0 

After some research I tried the following troubleshooting steps:

1) Moved the offending user to a different OU temporarily to see if the problem resolved.  This essentially "tickles" AD into replicating that particular user. I recieved the same messages, but the user's CN had been updated to the new OU.
2) Used the LDP tool to see if there was duplicate entries for this user somehow, but only one instance was found.
3) Used repadmin to look at the time stamps of various attributes on the account, particular one with a time stamp close to the time that the replication warnings started appearing in the event log.

Repadmin was where I had the most luck.  You'll want to run the following command for Windows 2003 SP2 DCs:

repadmin /showobjmeta DC1 "CN=Joe Smith,OU=Accounts,DC=mydomain,DC=org"

This will return a list of attributes with timestamps.  In my case it was the attribute related to the last password change, which was the only one that had a timestamp of the same date when the errors began.  I reset the password on the account to "tickle" that particular attribute and the replication completed without any complaint.

Some anticodotal stories on the Internet indicate that this attribute can cause trouble if replication occurs while an account happens to be locked out.  In this case, the account was for a consultant who didn't log in very often, so the locked account went unnoticed for some time, causing the replication issue.

Inside MDOP: AGPM 4.0

In case you missed the PacITPros meeting on December 7th, you missed out on some interesting vendor and technical presentations.  In addition to a presentation from BlueCat Networks and Hurricane Electric, I did a short demo of one of the MDOP tools - the Advanced Group Policy Manager 4.0.

This tool hooks right into the existing Group Policy Manager snap-in you know and love in your MMC and with the use of a designated archive server, extends the functionality to include better search features and change management.  No matter the size of your organization or the number of IT staff you share group policy tasks with, you can benefit from this tool.  Even if you are the only person who does anything with group policies, this tool will make your life easier.

First, the change control features take away much of the pain of keeping track of what was changed when and potentially by who.  Policies that are controlled by the system must be checked in and out for adjustments, which automatically creates a history record capturing the state of a policy at any given time.  These records can easily be reviewed for corporate compliance and policies can even be rolled back to previous states.

With new roles created within the tool, non-admininstrators (even regular domain users) can be granted the ability to review or edit policies... leaving the actual deployment and linking of the GPOs to system administrators.

The abililty to search and filter your view of policies is much improved.  You can search by name, state (checked in, checked out), even by variables such being updated "last month" or "last week". 

Finally, you can easily import and export policies, even across forests.  No more manual recreation of the perfect policy just because you want to use it in your test lab environment or in another forest.

Finally, keep in mind that APGM 4.0 adds support for Windows Server 2008 R2 and Windows 7, as well as runs on Windows Server 2008 and Vista.  If you are supporting an environment with older versions of Windows Server, consider version 2.5 or 3.0 of the tool.  Not of all of the features are included, but if you are looking specifically for the change management aspects, those older versions may work for you until you upgrade your servers.

Out of the six tools in the Microsoft Desktop Optimization Pack, APGM isn't one I'd overlook. 

The Post-Mortem of a Domain Death

The past few days have been busy as we've been performing the tasks to remove our failed domain controller and domain from our Windows 2003 Active Directory forest.  Now that everything is working normally and I can check off that long-standing IT project of "remove child domain" from my task list, I'd like to share a few things we've learned.

• NTDSUTIL will prompt you several times when it comes to removing the last DC in a domain using the steps in KB 216498. It will even hint that since you are removing the DC in the domain, that you are also removing the domain itself.  But you are not.  You must take additional steps in NTDSUTIL to remove the orphaned domain, see KB 230306 to finish up.
• How do you know you have an orphaned domain? Check AD Domains and Trusts.  If you still see a domain in your tree that you can't view the properties of, you aren't done yet.  Also, if your workstations still show the domain as a logon option in the GINA, get back to work.
• You might remember to clean up your DNS, but don't forget to also clean up WINS.  WINS resolution can haunt you and keep your workstations and applications busy looking for something that isn't there anymore.
• Watch your Group Policy links.  If you've cross-linked policies from the child domain to your forest root or other domains, workstations will indicate USERENV errors referencing the missing domain.  Policies from other domains won't show up in your "Group Policy Objects" container the GPMC.  You'll need to expand all your other OUs in the GPMC to find any policy links that report an error. 
• If you are using a version of Exchange that has the infamous Recipient Update Service, remove the service entry that handles the missing domain.  You'll see repeated MSExchangeAL Events 8213, 8250, 8260 and 8026 on your mail server otherwise.

I've used NTDSUTIL in the lab and in production several times to remove failed domain controllers, but removing an orphaned domain happens far less frequently.  While the majority of our Microsoft applications handled the existence of references to the orphaned domain with grace until we completed the clean up, one of our third party applications, ImageRight, was far more sensitive about it. 

We found that a combination of the WINS resolution and the orphaned trust relationship distracted the application enough that it was slow to operate, sometimes refused to load at all, and hung on particular actions.  If you happen to be an ImageRight customer who uses the Active Directory integration features, keep in mind that it likes all the AD ducks to be in a row.

While we had a little a bit of pain getting to this point, I'm really happy that our AD forest is neater and cleaner because of it.  It'll make it much easier to tackle other upgrade projects on the horizon for Active Directory and Exchange.

Virtualized Domain Controllers? I’ll pass, thanks.

There are a lot of good arguments for virtualizing DCs. You should have several of them for redundancy, but depending on the number of employees and general work load, DCs tend to be underutilized and it can be hard to warrant having a whole physical server for each one. But after loosing a second domain controller after doing essentially some basic VM maintenance, I’m not sold.

You may remember a previous post of mine from the summer of 2009 about NTDS Error 2103, when the DCs in a small child domain were virtualized.  I had agreed to virtualize both DCs from that domain as the domain was not supporting any user accounts and had less than a half dozen servers as members.  One did not convert well and we decided to just leave the remaining DC as the sole one standing for that domain after vetting out the risks.  There are several “rules” to follow when virtualizing DCs, particularly not restoring snapshots of them and not putting yourself in the situation where your VM host machine need to authenticate to DCs that can’t start up until your host authenticates.

Fast forward about 16 months, to now.  Our system administrator who handles the majority of our ESX management was migrating many of our VMs to our newly installed SAN.  He reported that he shut down the DC normally, moved the VM and then started it back up a few hours later after all the server files had been copied over. The few servers that use that DC were working properly and everything looked good. 

But alas, a few weeks later, the server reported a USN rollback condition. Replication and netlogon services stopped.  I checked the logs to see if I could figure out the cause, but only saw things that added to the confusion.  The DC was mysteriously missing logs from between the time of the VM relocation and the time of the NTDS error.  And the forest domain controllers had logs indicating it had been silent for nearly 2 weeks. At this point, I can only speculate what went bad.

We slapped a bandage on the server by restarting netlogon so those few servers could authenticate, but without replication happening properly, the server will simply choke up again. And after the tombstone lifetime passes, the forest domain will consider it a lost cause.  It’s essentially a zombie.

So begins our finally steps to decommission that child domain. I have no interest in restoring that domain from backup, since removing that domain has been an operations project that has been bumped for a long time. Now our hand has been forced and the plan is simple.  Change a couple service accounts, move 2 servers to join the forest root domain and then NTDSUTIL that DC into nothingness.

As for our two forest root domain controllers?  I’ll throw my body in front of their metal cases for a long time to come.

Round Up: September Events and Windows Resources

As the summer draws to a close, here are a few thing that have caught my eye.  Check them out and see for yourself.

First up, a comprehensive article on Redmondmag.com by Greg Shields on "Avoiding Active Directory Disasters", particularly with Windows 2000 and 2003 versions of AD.  It might just be time to do a little end of summer health check for an important component of your network.

PacITPros will be holding their monthly meeting tonight starting at 6pm.  It's not to late to RSVP and get in on the action.  Compellent will be presenting on their data storage solutions and there will also be a presentation on the latest and greatest with Microsoft's OCS.

On 9/30, don't miss out on the Springboard Series Virtual Roundtable on migrating from IE6 to IE8 on Windows 7, starting at 9am.  This virtual, interactive discussion will cover migration strategies, standards, and support for organizations moving from Internet Explorer 6 to Internet Explorer 8.

Finally, check in with the folks at TechNet on 9/22 when they host an in-person event on Deploying Windows 7 at the Microsoft downtown office in San Francisco.  Learn more or register at TechNet Events Presents: Deploying Windows 7. This event will also be held in Irvine, CA on 9/20 and in Los Angeles on 9/21.

Happy 10th Anniversary to Active Directory

Time sure flies when you are busy keeping up with Active Directory, which has been around since it's release on February 17, 2000 with Windows Server 2000.

I remember the first time I was part of an upgrade from NT 4.0 to Windows 2000 Active Directory. I was the sole IT person in the branch office and was working on a project to upgrading my branch office to be a child domain in the headquarters' "new" Windows 2000 Active Directory forest.

The NT 4.0 PDC in my office had a DNS suffix defined in the network settings, and unknown to us at the time, caused my domain to end up with a disjointed namespace. Once we realized we had an issue, I got to be part of my first upgrade and my first rollback - all in the same evening.

Because I had taken my backup domain controller offline, it was pretty easy for me to bring NT 4.0 back to life. It was far more work for my colleagues at headquarters, who had to call support services for details on using NTDSUTIL to remove the remnants of the child domain controller out of AD forest so we could perform the upgrade again.

Several years, and several domain controller demotions later, I'm quite comfortable with the process I first saw happen back in that little closet of a server room. Active Directory, it's certainly been fun 10 years!

Don't Overlook the Metadata Cleanup

It seems inevitable that while restoring Active Directory in a disaster recovery scenario, one is going to feel rushed. Even with this being a test environment, I felt like getting AD back was something that needed to be quick so we could move onto the more user-facing applications, like Exchange.

My network has two active directory domains, a parent and a child domain in a single forest. The design is no longer appropriate for how things are organized for our company and we've been slowly working to migrate servers and services to the root domain. Right now, we are down to the remaining 3 servers in our child domain and one remaining service account. The end is in sight, but I digress.

The scope of our disaster recovery test does not involve restoring that child domain. This is becoming an interesting exercise, because it will force us to address how to get those few services that reside in that domain working in the DR lab. This will also help us when we plan the process for moving those services in production.

Bringing back a domain controller for my root domain went by the book. I could explain away all of the random error messages, as they all were related to this domain controller being unable to replicate to other DCs, as they hadn't been restored. I had recovered the DC that held the majority of the FSMO roles and sized the others. I started moving onto other tasks, but I couldn't get past the errors about this domain controller being unable to find a global catalog. All the domain controllers in our infrastructure are global catalogs, including this one, as I hadn't made a change to the NTDS settings once it was restored.

So I took the "tickle it" approach and unchecked/rechecked the Global Catalog option. The newly restored DC successfully relinquished its GC role and then refused to complete the process to regain the role again. It was determined to verify this status with the other domain controllers it knew about, but couldn't contact.

I knew for this exercise, I wasn't bringing back any other domain controllers. And in reality, even if I was going to need additional DCs, it was far easier (and less error-prone) to just promote new machines than to bother restoring every DC in our infrastructure from tape. (However I still back up all my domain controllers, just to be prepared.)

To solve the issue, I turned to metadata cleanup. Using NTDSUTIL, I removed the references to the other DC for root domain, the DC for the child domain and finally, the lingering and now orphaned child domain itself. I also had to go into "AD Domains and Trusts" to delete the trust to the child domain, which wasn't removed when the metadata was deleted. Once all these references were removed, the domain controller successfully was able to assume the global catalog role and I could comfortably move on to restoring our Exchange server.

And I've learned that just because I can explain an error, doesn't mean I can ignore it.

AD Recycle Bin - New in Server 2008 R2

This week I continued with disaster recovery testing in our lab, the first machine restored from tape being one of our domain controllers. While checking over the health of the restored Windows 2003 active directory, I remembered that we are using a third-party tool in production to aid in the recovery of deleted items - Quest's Active Directory Recovery Manager. To be honest, we haven't had a reason to use the software since we installed it, which I suppose is a good thing. But it is a stress reliever to know that it's there for us.

Restoring this product in our test lab isn't part of the scope of this project, but it does have me looking forward to planning our active directory migration to Server 2008 R2, which includes a new, native "recycle bin" feature for deleted active directory objects. You can find more details about how this feature works in Ned Pyle's post on the Ask the Directory Services Team blog, The AD Recycle Bin: Understanding, Implementing, Best Practices, and Troubleshooting.

While the native feature doesn't have the ease of a GUI and requires your entire forest to be at the 2008 R2 functional level, it's certainly worth becoming familiar with. Once I'm done with all this disaster testing, you can be sure this feature will on the top of my list to test out when I'm planning that upgrade.

Check Out TechNet Events

Today I enjoyed a morning at the Microsoft office in SF attending an event in the current series of TechNet Events. Through the months of September and October, the TechNet Events team is traveling around the US providing tips, solutions and discussion about using Windows 7 and Server 2008 R2.

Today's presentation was given by Chris Henley, who led some lively and informative discussions on three topics - Tools for migration from Windows XP to Windows 7, Securing Windows 7 in a Server 2008 R2 Environment (with Bitlocker, NAP and Direct Access) and new features in Directory Services.

I was excited to see specific information on Active Directory. If you missed the blogs about Active Directory Administrative Center back in January like I did, you'll like some of the new features in this 2008 R2 tool, including the ability to connect to multiple domains and improved navigation views.

If there isn't an event near you this time around, check back after the holidays when they'll head out again for another series.

Stumbling Over AD Intergration in ImageRight

Friday night, I was responsible for a maintenance upgrade to our document imaging system, ImageRight. This upgrade was required to repair a potentially serious data corruption issue that was discovered by the vendor. We weren't affected by the corruption at the time it was discovered but some functionality had been disabled as a work-around, so we had to schedule time to perform the fix.

First off, let me say that I really like the vendor and I like how the product works overall. However, we always seem to be the client who has issues that the vendor never seems to encounter before. It was almost refreshing when they called about the corruption issue and it wasn't something we'd found first.

Friday morning, I had exchanged a few emails with the vendor support tech who was going to do the upgrade to firm up the planned roll-back procedures (for our change management documents) and to clarify any last minute items. He mentioned a known bug related to environments with ImageRight users that spanned multiple Active Directory domains, fondly referred to as the "AD dual domain bug" and how the upgrade shouldn't be performed if we had an environment with those characteristics.

Yes, we have two domains. But no, we don't have accounts that are used by ImageRight from the second domain. We confirmed those details and I mentioned in one of my reply emails that the AD bug had me a bit worried anyway. I was told my environment wasn't going to be an issue based on their testing. (Okie-dookie then.)

So away we went with the upgrade. That was the easy part. Then came the testing.

Exactly half the program worked - literally half. The program launches two windows when it starts - one window acts as the file manager, for searching and loading image files and the other is the image viewer. I could see and use the file manager portion, but the viewer never loaded, only returning an error that was cryptic overall but referenced active directory about 15 times.

Um, yeah.

So they uninstalled and reinstalled just to make sure some random DLL wasn't left behind or something. But that didn't solve the problem. I chilled out on hold for a while. It was already after 7pm here on the west coast, so I felt a little bad for the tech on the east coast. "Are we sure this isn't a different manifestation of the AD bug?", I asked.

I chilled out on hold for a while longer while the support tech consults some developers on his cell phone. No, it shouldn't be the AD bug, we are getting an error "too soon" in the loading of the program, but there is a hotfix for that bug that should be released on Monday. A developer was working on getting a copy for us now if we wanted to try that.

Why not? It's already broken, might as well toss one more thing at it before we roll everything back. Sure enough the hotfix did the trick, avoiding a roll-back and saving me another late night at the office. I wasn't surprised that it was yet another instance of something no other client they have has ever experienced.

I'm not sure how I feel about always being the "one-off" case, but it always seems to work out fine in the end. Though I'm thinking of framing my "I'm a bit worried about the AD bug" email that I sent out before we even began.

NTDS Error 2103

This week one of my domain controllers developed a curious problem. I don't like curious problems, especially ones that rear their heads after the server reboots.


The error was an NTDS General event 2103, which indicates that the AD database "was restored using an unsupported procedure and Net Logon service has been paused". Research and KB Article 875495 lists event 2103 and 3 other events related to a condition known as USN Rollback.

This DC is running Windows 2003 SP2, so based on the article, I should be seeing at least the more serious NTDS Replication 2095 event as well, due to a hotfix in SP1 that made the error logging somewhat more verbose. But I'm not. This makes it more curious. Am I in a rollback state or not?

KB 8759495 also lists some possible causes of this state, some of which are possible in a virtual environment - the case for this DC. It points me to another KB Article 888794 which lists out a bunch of considerations for hosting DCs as VMs. However our environment met all the requirements, including one related to write caching on disks, as our host machine has battery backed disk caching. So I rule out that we actively caused a potential rollback.

Repadmin has a switch (/showutdvec) that can be used to determine USN status by displaying the up-to-dateness vector USN for all DCs that replicate a common naming context. If the direct replication partners have a higher USN for the DC in question than that DC has for itself, that's considered evidence of a USN rollback. My DC did not have this problem, as it had a USN higher than it's partners. So at this point I couldn't confirm or deny a true USN rollback issue, however it seemed the the DC "thought" it was having this problem. Maybe I could figure out why the DC was in this limbo.

So I returned to the original article to look for specific causes. One line reads, "Starting an AD domain controller whose AD database file was restored (copied) into place by using an imaging program such as Norton Ghost."

Thinking back, the conversion of this DC from physical to virtual did not go as smoothly as I would have hoped. I remembered I had to resolve some issue where I was getting an error in the logs related to the directory database file not being where the OS expected it, even though the path on the server hadn't changed during the conversion. It was odd at the time, but the posted fix seemed to clear the issue and I'd moved on.

I'm guessing that perhaps that was the start of my issues - maybe the P2V process made the OS think the database was different copy even though it wasn't. The result was that the server thought it was rolled back, but the USNs never reflected a problem. So I decided it was better to be safe than sorry and assume this "limbo" condition was not how I wanted to leave things.

The resolution for USN rollback is a forced removal of the domain controller from AD. Since this is a DC in a child domain that's being phased out, very few changes happen to that domain so I wasn't concerned about possibly loosing changes that may have been made on that DC. It was only the FSMO holder for one role which was easily seized by the other DC.

My decision now is to decided between bringing up a replacement DC for this domain next week or just run one DC for the time being and try to speed up the remaining tasks that need to be done before we can removed the child domain all together.

But that's for another day!