Aftershocks

Just thinking out loud…. I’ve just replaced the 8th server hard drive since Aug 2011. That’s more than I’ve replaced in the past 2 years combined. I suspect this is a result of the 2011 Louisa, Virginia earth quake.  Servers are not meant to dance across the floor. 

At the time, the only visible damage I found was a Wrightline Work Surface that had not been bolted together that has been shaken apart. I’m wondering if anyone else has seen an increase in drive failures.

System Administrator Password Reset Woes

So what do you do when a system administrator has left the company and no one knows the password for their system? You use the reset password link of course :) Unfortunalty, in our case the system admin had used their personal email account for the password reset feature.  While this was against our SOP and security policies the fact is it happens. Hopefully you’re applying message hygiene policies on your outbound mail.

I’m using Trend Micro’s InterScan Message Security Suite (IMSS). All applications sending mail must first go through my IMSS server. So I created a policy that looked for adminname(at)hotmail.com  and redirected it to the new administrator using his internal company account. This sent the password reset link to the new admin. Crisis averted.

Ich spreche kein Deutsch or why are my Outlook messages in the wrong language

Working at an international l company provides lots of benefits.  One of these is the ability to practice your language skills. While my accent is horribly southern, I can ask directions and say “please” and “thank you” in French, German and Spanish – Que pass ya’ll

I recently received a helpdesk ticket where the non-delivery reports, read receipts and delivery receipts were received in German.  Jumping into their mailbox with OWA I found their regional settings were in Swiss – German. Luckily I read German fairly well. So I selected Optionen, in the upper right corner. Language is Ländereinstellungen  on the menu on the left side of the screen. So I changed the language to Englisch (Vereinigte Staaten).and closed the ticket with a quick note to the user with the solution. 

Unfortunately this didn’t fix the problem.  Usually I’d be pointing to store cache as being the culprit.  Many mailbox settings do not take effect until after the store cache refresh which is at the 2 hour default setting. However, I changed the language settings on a Friday and Monday she was still getting read receipts in German.  An internet search reviled a lot of people having this issue and setting the language in OWA correcting the issue. Calling Microsoft we spent a couple hours digging around and doing the typical Q&A. 

After a while I pull up the mailbox in OWA to confirm that I did indeed change it and sent a quick test mail.  Low and behold the read receipt came back in English.  All of our previous testing had been done with Outlook using terminal server because the mailbox was fairly large. I didn’t change it back to repeat the process, but I was left with conclusion that sending the read receipt in OWA caused the Exchange servers to sense the new setting. 

Orpaned Public Folders

Public Folders folders can be contrary

We make extensive use of public folders.  Most offices have a dedicated public folder structure. Every distribution list we create has a corresponding public folder as a member.  At the end of a contract we typically supply these messages to the client by exporting them to a PST.  Recently I received a helpdesk ticket for one of our offices who could not get export to work. 

The error was “This folder cannot be opened because there is a configuration problem with the server.”

I completed the usual checks for public folders. The default permissions were correct. Anonymous permissions existed and were configured correctly.  The folder had a replica. In fact there were no issues copying mail from the folder into a PST.

I did find that the alias was “mail. Our default address policy applies the alias to the SMTP properties to create addresses.  The folder already had a unique email address from being mail enabled when it was hosted in Exchange 2003. Adding a unique alias simply cause the policy to add another SMTP address to the folder.

Using the exchange shell I gave myself the Owner role and started checking its properties. What I found was that the folder’s location was very strange.  Our folder paths typically are Location\Clients\Contract Code and a “mail” subfolder. This was appeared to be randomly generated numbers.

Checking another folder gave similar results so I started checking the folder path working my way back up the chain.  What I found was that I was unable to display the contents of the Clients folder.  Pulling up the folder properties in the Exchange Shell I found that the folder didn’t have any replicas. Once I added a replica and waited for the store cached to refresh I was able to see the correct folder location and complete an export.

Log Jam

It’s been a long while since I posted anything.  Frankly, that’s because things have been quiet and I’ve not run into anything unusual or noteworthy.  One of my resolutions this year is to start posting again, so I’ll be digging back though my notes. I’m also wrapping up some projects that I’ll post.

Today on of our Exchange 2007 servers hosting our public folders started consuming log file space at the rate of about 1 new log file per second.  This corresponded with unexpected database growth. I’ve seen this before.  There are quite a few blogs and KB articles for troubleshooting the problem.  One of the best is here: http://blogs.technet.com/b/mikelag/archive/2009/07/12/troubleshooting-store-log-database-growth-issues.aspx

Unfortunately none of the suggestions actually pointed to my problem. It starts by looking for looping mail using Exchange User Monitor (Exmon) to look for a looping mail. Exmon runs on servers that host a database.  It cannot be used on Hub/CAS servers unless they are also hosting a database. Not the output below shows that the local HUB/CAS box is the top CPU consumer. So that’s a start

 

Moving to the Hub/CAS server I can see in the screenshot below that I have quite a few emails in retry that are heading for my journaling server and I have a small message that seems to be stuck.  In reality it’s looping faster than the GUI can keep up. If I double click the message or try to right-click and delete it I get a message indicating that the system has processed it already.

The first time I saw this I spent time trying to delete it using the Exchange management shell and the remove-message commandlet:

Remove-message -filter {Subject -eq "Win Big"} -WithNDR false

Yeah….That didn’t work either. I punted and called Microsoft. By now the system, which is located in EU had eaten 250 gigs. The data drive and log drive are both on the SAN so we were able to expand them but it was consuming space at such a rate that we would soon need to expand the drive again.  To make matters worse space on the SAN was already constrained so I would not be able to expand the drives enough to continue to troubleshoot. It was using 5 GB every 20 minutes.  Of course we had to rerun all the troubleshooting  :)

While on hold, as he discussed the problem with the next level engineer, I continue to scour the net for a solution. I found a vague reference to the queues becoming corrupt. I’ve recreated the queues before when I had a huge mail backup which caused hub throttling to kick in.  The queue database does not shrink once it’s expanded.  The easiest way to reclaim that drive space is to create a new queue. So I gave that a try.

Fixing a corrupt queue on a hub transport server is fairly simple:

•    Stop the MS transport service on the Hub/CAS server
•    Rename  the queue folder

(Note that most instructions indicate that you should recreate a new "queues" folder.  My experience has been that you should let the transport server create this folder itself when you start it)

•    Restart the MS transport service
•    Test mail transport using a mailbox in that site sent to/from a mailbox in another site
•    Monitor the queues, database and log file drives for growth.

Fixed. No more mail loop. new logs dropped from 130 a second to about 1 a minute.