You Can't Send That From Here

Recently, an end user of mine was getting immediate NDR messages when sending to an outside party with the error message of "None of your e-mail accounts could send to this recipient."

Normally, I chalk these type of issues up to a temporary problem with the external server, but the fact that it was an immediate NDR indicated it was some type of internal problem. Also, the language of the error message seemed odd, as our users only have one email account and aren't configuring their office Outlook clients to connect to other POP mail services.

While my research didn't turn up the exact scenerio my end user was seeing, it came down to the fact that he replied to a recent email using the "mailto" address link from a previous message that had imbedded the "mailto:" in the email address and Outlook 2007 was mishandling it in some way.

The link had then updated his Outlook autocomplete file to include the "mailto", so every time he tried to send something else to same address, the error reoccured. We had to delete the autocomplete entry and retype the email address to make sure everything worked properly again.

Their is a hotfix (http://support.microsoft.com/kb/2475888) that address several issues, including this one.I didn't apply the provided fix at this time, since the issue has only come up once so far. But I'll be on the lookout for more of this in case a mass deployment of the fix is required for our desktops.

Exchange 2010 on the Horizon

I started this week and hopefully I'll get to spend more time next week working on my lab for migrating from Exchange 2003 to Exchange 2010.   Today I'm using the Exchange Server Deployment Assistant, which asks a few questions about your environment and then generates a checklist of things for you to check and do as you move through the installation process.

I'm also hoping to find some time to run the Exchange Pre-Deployment Analyzer in my production environment and see if that give me some good news.

Happy Friday Everyone!

Memory Leak cripples OWA

I have to admit the Exchange 2003 Outlook Web Access has me a bit spoiled. It always seems to be there - day in, day out. So when a report of OWA not loading came in, I was surprised. Where to begin?

I really don’t like rebooting Exchange. The usually ever-reliable attempt to restart the IIS service didn’t bring it back to life and nothing suspicious was in the event logs, so our resident webmaster took a look in the IIS logs and found several “connections refused” errors in the %WINDIR%\logfiles\httperr\httperr1.log.

This gave me something to start with and after some research I found that those type of errors in the HTTPERR log often point to a non-paged pooled memory leak. As per the Troubleshots MSDN blog:

While there are many possible causes for the “Page cannot be displayed” error, there is only root cause which causes the http.sys driver to begin refusing client connections--a depletion of non-paged pooled memory, an NPP leak. The HTTP.sys driver was new with Windows 2003, is a kernel mode driver, and, at the risk of splitting hairs, is technically not part of IIS 6.0. This distinction is important in troubleshooting. When http.sys refuses to hand connections to IIS a “Connection_refused” or “Connections_refused” will be logged in the httperr log (C:\WINDOWS\system32\Logfiles\HTTPERR) rather than the IIS logs.

At this point, I didn't want to just reboot the server to clear the memory leak. I wanted to know what was leaking. Using Task Manager, I added the columns for the Non-paged Pool and the process for “NPSrvHost” shot to the top of the list with almost 10x the average memory consumed compared to the other processes. NPSrvHost belongs the NetPro Compliance Agent. I stopped and restarted that service and memory usage returned to a normal range.

Finally, I performed and IISRESET and the OWA service came back to life.

Recovering Hard Deleted Items in Outlook

This isn't new information, but it's something that comes up from time to time - recovering hard deleted (SHIFT+DEL) items from Outlook. Hard deleted items skip over the "Deleted Items" bin, so they can't be recovered using the regular "recover deleted items" tool within the Outlook client.

Exchange 2003 OWA can be used to recover items that were hard deleted using the Outlook client. To get back those items, log into the OWA web page. Then edit the URL to be: "https://server_name/exchange/user_name/inbox/?cmd=showdeleted". The "dumpster" for the inbox will appear and you can recover your deleted email. If you want to recover items from other folders, just change the word "inbox" in the URL to the folder you need, like "calendar" or "drafts".

If you are using Outlook 2003 as your mail client there is a registry setting that you can add to turn the dumpster on for all the folders. Outlook 2007 has the registry setting already enabled by default. Of course, recovering any deleted items assumes that the deleted items retention settings have been configured on your Exchange server.

Exchange Server under the tree this Christmas?

I've been reading a lot about Exchange 2007 and have been thinking about what the next move for our Exchange server at the office should be. We haven't decided on Exchange 2007 vs. Exchange 2010 yet, but no matter... I want Santa to bring me a way to eliminate all the PST files being used around the office.

We don't have a large staff. With less than 70 people our Exchange server doesn't work that hard. However, with the desire to bring email services back up as quickly as possible after a failure we have a policy in place that limits the amount of mail stored on the server to 250MB per user. This leaves our data store at a little over 18GB. Our last test restoration of exchange required about 2 hours for loading the database.

Contrary to this is everyone's need to keep every scrap of every email message. This has lead to numerous PST files created as archives for all this mail. It's pretty safe for me to assume that almost every employee has at least one PST file and they are all stored on the network shares.(Yes, I know PST storage on the network is unsupported.) My quick search yielded about 30 GB of PST files and I know I didn't find them all.

So what exactly can Santa bring me?

First, I would be lying if I said I needed a server with more space. The current exchange server still has upwards of 180GB free, so it's likely I could support years of user email with our current setup just by throwing open the storage limits.

I would like to have a proper email archiving system that would automatically move mail from the active mailboxes to secondary storage, thus leaving my primary database small while allowing users to seamlessly access old messages. Personally, I don't keep much in the way of work email and I think that if my company wants me to keep mail for historical purposes, they should provide an easy way to do so. However, I haven't managed to convince the powers-that-be that this is something to embrace quite yet.

My next choice would be reconfiguring Exchange using 2007 or 2010 to take advantage of additional storage groups and "dial-tone" mail service. If I could virtualize the mail server with a SAN for storage, I could bring basic services up in a snap(shot). By breaking up users into multiple storage groups, it would be possible for us to restore mail service immediately and then backfill the databases in small chunks. While it would still take time to restore all the data, users would be able to send and receive mail while old mail would trickle in as the storage groups come back online.

I know "dial-tone" restores are possible with my current setup, but utilizing it in Exchange 2007 or later is much easier than Exchange 2003 due to the auto-discovery features. I also would like to have at least one storage group (with only one database) per department, nearly double of the four storage group limit with Exchange 2003. With the 50 storage group limit in Exchange 2007 I wouldn't have any problem meeting my goal. Also, Exchange 2010 has some good "starter" archiving features for mail management that might be worth a closer look.

Of course Exchange 2007 and 2010 require 64-bit hardware, so maybe Santa can bring me that new server after all.

Migrating to Exchange 2010 - Where to Begin?

One of the potential projects on my list for 2010 is migrating our Exchange 2003 server to either 2007 or 2010. I'll likely have to do some more detailed comparisons of features and requirements, as well as take a close look at some of the long term business goals so that we can make the most cost effective decision that will also give us some longevity. It seems like just yesterday that I migrated from Exchange 2000 to Exchange 2003.

Just to wet my feet a little bit, I found some great posts by Rand Morimoto regarding migrating to Exchange 2010.

• The Good, the Bad and the Ugly in Migrating to Exchange Server 2010


• Migrating from Exchange 2003 to Exchange 2010


• Migrating from Exchange 2007 to Exchange 2010

Also, be sure to check out the Exchange Server Supportability Matrix, which lists out which operating systems support installing the various flavors of Exchange, as well as which flavors of Active Directory are also supported with Exchange. For example, Exchange 2003 SP2 will run in a 2008 domain (but not 2008 R2) and don't even think about pushing your forest or domain functional levels past 2003 compatibility. On the flipside, Exchange 2007 and higher won't run on a 2000 Active Directory enviroment, so if you are still sporting that type of domain you know where to start.

At any rate, there are quite a few little ducks that need to be a row. I know I've got a bit more reading to do before I start writing up my migration plan.

Restoring Exchange 2003

After getting Active Directory up and running in my disaster recovery lab, my next task was to restore the Exchange server. The disaster we all think of most in San Francisco is an earthquake, which could either make transportation to the office ineffective or could render the office physically unsafe. Email is one of our most used applications and in a disaster we predict it will be the primary means of communication between users working from outside of the office. Assuming a functional Internet backbone is available to us, email is will likely be the fastest way to get our business communications flowing.

Restoring Exchange 2003 is a straight-forward process, you have all the previous configuration information available to you. In our DR kit, we have a copy of Exchange 2003 and the current Service Pack we are running. We also have a document that lists out the key configuration information. Before you restore, you'll want to know the drive and paths for the program files, the database and the log files. If the recovery box has the same drive letters available, the restoration is that much smoother when you can set the default installation paths to the same location and ensure that you'll have the necessary amount of space free.

It's important to remember to install Exchange and the Service Pack using the /DisasterRecovery switch. This will set up the installation to expect recovered databases instead of automatically creating new ones. I had manually mount the databases after the restoration, even though I had indicated in the Backup Exec job to set the databases to be remounted when the restore was complete. Microsoft KB 262456 details the error event message I was getting about the failed MAPI Call "OpenMsgStore", which was a confirmed problem in Exchange 2000/2003.