Connecting our Anti-spam to Exchange 2007

The next step in shutting down our old Exchange 2003 servers is to configure our anti-spam appliance.  If you’re looking for a great anti-spam vender check out Xecuritas. We’re in the healthcare industry which makes it tough to filter based on the body parts and drugs listed in your typical spam message. These guys do a great job and proved stellar service.

So the process involves setting up a NAT for LDAP authentication so folks can log in and check their quarantines.  This NAT points to a DC and provides a one-to-one connection from the anti-spam appliance.  It only accepts those 2 IP addresses and is only open for LDAP traffic.

I also have a NAT for recipient verification. This allows the appliance to check the recipient for every mail and if the recipient doesn’t exist we drop the mail.  Yes, I know the current RFC says we should bounce that mail, but hey, during a heavy spam flood we may see 2.7 MILLION messages a day.  I can’t really afford to double that kind of traffic with NDR’s This NAT is also a one-to-one connection from the appliance to my HUB server. It only allows SMTP traffic.

The Exchange 2007 native anti-spam features are installed using the install-AntispamAgents.ps1 script located in Program Files\Microsoft\Exchange Server\Scripts. All of the anti-spam options are enabled by default. To turn on recipient filtering Go in the properties of recipient filtering and enable the option "Block messages sent to recipients not in the global address list" on the Blocked Recipients list.  Click Apply/OK. Restart the Microsoft Exchange Transport Service again.

I know what you’re thinking. My boss asked the same thing, “What about all of those mail enabled public folders that you have hidden?”  They still work. They are still in the GAL.   Hiding them does not remove them from the GAL. They simply have a database flag set the hide them.

The final Exchange configuration piece is to disable tarpitting.  This feature is to help prevent directory harvest attacks.  It’s a delay in the rejection response.  The idea is that the spamming server will drop the connection before the rejection is generated. If your Exchange server is open to outside connections this is a good thing. I have two hygiene servers between my Exchange environment and the rest of the world so I don’t need this feature.  In fact I want that rejection to happen instantly so mail doesn’t back up on my inbound mail server.

You can check your tarpitting interval from the command prompt.

Get-ReceiveConnector | select name,tarpitinterval

It’s turned off by the command line too.

set-ReceiveConnector "Server1\Default Server1" -tarpitinterval 00:00:00

set-ReceiveConnector "Server1\Client Server1" -tarpitinterval 00:00:00

set-ReceiveConnector "Server2\Default Server2" -tarpitinterval 00:00:00

set-ReceiveConnector "Server2\Client Server2" -tarpitinterval 00:00:00

OAB Generation Issues

We make extensive use of public folders.  Every distribution list files into a mail enabled public folder.  I’ve tried to take full advantage of 2007’s distributed architecture.  I have three clustered mailbox servers in three separate datacenters. Since CCR is not supported while hosting public folders I also have three public folder servers in each datacenter. (They host-super-huge-ginormous-mongo public folder databases. I lose sleep over this, but we’re not going there right now.) Recently I’ve seen an uptick of instances where mail enabled public folders never generate an SMTP address. Cycling the system attendant service fixes it.  There’s no indication of a problem other than getting alerted that the proxy address is not available when going to the folder permissions.

I’m slowly edging my way toward shutting down the final Exchange 2003 system. So this week I moved the Offline Address Book generator to one of the 2007 servers hosting public folders. No big deal right?  Testing in the lab, it was just a process of right click and pick an exchange 2007 server. The uptick in SA problems coincide with the OAB move. I also found 9386 and 9399 warning in the event log.

The description of the error looks like this:

OALGen is configured to generate version 2 or version 3 OAB files for offline address book '/o=Organization/cn=addrlists/cn=oabs/cn=Default Offline Address List' but there is no public folder server available. OAB versions prior to version 4 require a public folder server and cannot be generated at this time. Please ensure that a public folder server with a replica of the Offline Address Book system folder is online and mounted, or disable all OAB versions other than version 4.

Thinking back to moving the public folders (there were ~17,000 of them. ~3.5 million messages for a total of 4T of data) I double checked to make sure my OAB system folders are replicated to the 2007 public folder server.  They were. Here’s a very important piece I’ve not told you.  My public folder servers do not host mailbox databases.  I did this so that mailboxes could not accidentally be created on a “public folder” server. Keep this in mind while you read “Troubleshooting MSExchangeSA 9386 Warnings and OAB Generation and Access Issues”

Back already? My System Attendent service did NOT have a HostMDB property. If the System Attendant service does not have a host mailbox database there are quite a few actions that it cannot perform – database cleanup tasks are one, and OAB generation are another. So I quickly created a mailbox database and, using ADSIedit, gave the System Attendant a host database property.

IF you’re following along because you had the same issue, you should now see Event ID 4002’s –

An unexpected failure has occurred. The problem will require administrator intervention. The service will retry in 56 seconds. Microsoft.Exchange.Data.Storage.MailboxUnavailableException: Cannot open mailbox

 If you have ever had to move the System Attendant mailbox before then you might recognize this one.  Restart the Microsoft System Attendant service and it will create a new mailbox.

Finally I get Event ID 9331’s –

OALGen encountered error 8004010f (internal ID 501080c) accessing the public folder store while generating the offline address list for address list '\Global Address List'.

There are followed by an Event ID 9358 –

OALGen could not update the timestamp of the previous version of offline address list '\Global Address List'.  Please check earlier event log messages for the specific error.

Updating the offline address lists manually from the Exchange Management Shell using the commands below fixed these errors.

Get-Addresslist | Update-Addresslist

Get-GlobalAddresslist | Update-GlobalAddressList

Get-offlineaddressbook | update-OfflineAddressbook

Update-FiledistrubutionServer

PF - DOES NOT - Admin

One of the very few things I’ve found in Exchange 2007 that cannot be done through the command line is check/ set the calendar permissions on everyone’s mailbox.  In 2010 this is done using the Add-MailboxFolderPermission command which unfortunately is not available in 2007. The next best option is to use PFDavAdmin.  Unfortunately it refused to work on one of my mailbox servers. I can connect. It provides a list of all the mailboxes on the server. Running the permission report creates a file with no data. When I try to expand the mailbox I get an error: Could not expand http://server.domain.doman/exadmin/domain.com/mbx/mailboxname.non_ipm_subtree/:The remote server returned an error (404) Not Found

I went through the standard troubleshooting of checking security settings, using a machine in the same physical subnet, using a different privileged account, etc.  Microsoft suggested using PFDavAdmin’s replacement ExFolders however that only works if you have a 2010 server.

3 calls and three engineers later I ended up checking the Dot Net versions and digging deeper. Finally I ended up replacing the OWA exadmin virtual directory. I found someone whoi had a similar issue here on the Exchange Server Tech Center. SO I read up on it between getting put on hold and actually fixing the issue.

First I exported the folder settings to a text file.

Get-OwaVirtualDirectory <Server>\exadmin* |fl >d:\foldersettings.txt

You need to know what format the folder is using which is done like so:

Get-OwaVirtualDirectory <Server>\exadmin*

Name                       Server                     OwaVersion

----                       ------                     ----------

Exadmin (Default Web Site) <Server>               Exchange2003or2000

I needed to know that it was using Exchange 2003or2000. Then I removed the virtual directory using this command:

Remove-OwaVirtualDirectory <Server>\exadmin* -verbose

The verbose switch gives this output:

VERBOSE: Remove-OwaVirtualDirectory : Beginning processing.

VERBOSE: Remove-OwaVirtualDirectory : Administrator Active Directory session

VERBOSE: Remove-OwaVirtualDirectory : Searching objects "<server>\exadmin*"

 of type "ADOwaVirtualDirectory" under the root "$null".

VERBOSE: Remove-OwaVirtualDirectory : Previous operation run on domain

controller 'dc.local'.

Confirm

Are you sure you want to perform this action?

The Remove-OwaVirtualDirectory cmdlet is removing Outlook Web Access virtual

directory "<server>\exadmin*".

[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help

(default is "Y"):y

VERBOSE: Remove-OwaVirtualDirectory : Deleting "<Server>\Exadmin (Default

Web Site)" of type "ADOwaVirtualDirectory".

VERBOSE: Remove-OwaVirtualDirectory : Previous operation run on domain

controller 'DC.local'.

VERBOSE: Remove-OwaVirtualDirectory : Ending processing.

The command below creates a new virtual directory, using the OWA version type displayed in the first Power Shell command.

New-OwaVirtualDirectory exadmin -owaversion Exchange2003or2000 -virtualdirectorytype exadmin –verbose

The verbose switch gives you this output.

VERBOSE: New-OwaVirtualDirectory : Beginning processing.

VERBOSE: New-OwaVirtualDirectory : Administrator Active Directory session

settings are:  View Entire Forest: 'False', Default Scope:

'Domain.local', Configuration Domain Controller:

'DC.l.local',

VERBOSE: New-OwaVirtualDirectory : Searching objects

"<Server>.Domain.local" of type "Server" under the root "$null".

VERBOSE: New-OwaVirtualDirectory : Previous operation run on domain controller

'DC.local'.

VERBOSE: New-OwaVirtualDirectory : Processing object "<Server>\exadmin".

VERBOSE: Creating Outlook Web Access virtual directory "Default Web Site" on

server "<Server>.Domain.local".

VERBOSE: New-OwaVirtualDirectory : The properties changed are: "{

OwaVersion='Exchange2003or2000', VirtualDirectoryType='Exadmin',

FolderPathname='\\.\BackOfficeStorage', Url={ 'http://<Server>:80/exadmin'

}, MetabasePath='IIS://<Server>.Domain.local/W3SVC/1/ROOT/exadmin',

AuthenticationMethodFlags='Basic, Ntlm, WindowsIntegrated',

AuthenticationMethodFlags='Fba', Id='<Server>\exadmin (Default Web Site)'

}".

VERBOSE: New-OwaVirtualDirectory : Saving object "<Server>\exadmin (Default

 Web Site)" of type "ADOwaVirtualDirectory" and state "New".

VERBOSE: New-OwaVirtualDirectory : Previous operation run on domain controller

'DC.local'.

VERBOSE: New-OwaVirtualDirectory : Searching objects "<Server>\exadmin

(Default Web Site)" of type "ADOwaVirtualDirectory" under the root "$null".

VERBOSE: New-OwaVirtualDirectory : Previous operation run on domain controller

'DC.local'.

Name                       Server                     OwaVersion

----                       ------                     ----------

exadmin (Default Web Site) <Server>               Exchange2003or2000

VERBOSE: New-OwaVirtualDirectory : Ending processing.

So why does this work?  According to this TechNet article the OWA virtual directories are used to “handle Exchange 2003 Outlook Web Access requests, Exchange 2000 Outlook Web Access requests, WebDAV requests, and some administrative functions.” Ok, I knew that but why ExAdmin? We the ExADmin virtual directory is used for Exchange Administrative (or like TechNet says here, to change administrative settings and properties.) access just like the name implies and PFDavAdmin must access it to make permissions changes etc.

Exchange 2007 – Where Nothing Happens Fast

One thing that I really like about exchange 2007 is The Exchange Management Shell (and Powershell). I try to use it for most of my daily tasks but sometimes it’s just quicker to right-click and pick new.

... until you make a mistake. I meant to create a mailbox database but I created a public folder database.  No big deal right? just delete it. There’s no data on it? Not so fast. Here’s what I get:

Remove-PublicFolderDatabase : Exchange is unable to check the public folder rep

licas. Verify the Microsoft Information Store service is running, and that the da

tabase is properly mounted

OK. FINE I’ll mount the database. GRRRR now I get this:

The public folder database contains folder replicas. Before deleting the public

folder database, remove the folders or move the replicas to another public

folder database. For detailed instructions about how to remove a public folder

database, see http://go.microsoft.com/fwlink/?linkid=81409.

I’m going to rant a little here. I’m wrapping up my 2003 to 2007 migration this month. I really liked that Exchange 2003 had pretty much done away with the little idiosyncrasies we’d come to love/hate about earlier versions. Gone were the days of “Exchange is just that way.” Well I’m starting to see that those days are back.  2007 puts a copy of the public folder hierarchy on all public folder databases.  This is a good thing. But what this means for me right now is that I have to wait for this database to replicate to all of my other public folder servers around the world. I also have to wait for the store cache to flush which may take 2 hours. Then I can delete this database.

Well I have things to do. I can’t wait around all day until Exchange decides it can find all the replicas for this database.  So firing up ADSIEDIT I find the Exchange Administrative Group for 2007, drill down to the Server CN.  Next I expand the container for the server with the wayward public folder and expand the information store container.  Expand the Storage group and delete the database. Now all that’s left is to manually delete the edb file and logs and I can fix my mistake and go home on time.

The first post - Shoulder Pad Build-up

For the guys out there – my wife introduced me to this term. It happens when every top a lady is wearing has shoulder pads so that they end up looking like an NFL linebacker. 

This week I was asked to fix a friend's computer.  The complaint was, “It’s not doing anything.” I found that it was in fact doing lots of things, all at the same time and not doing any of it very well.  It booted up quickly enough, but once I had logged in everything became agonizingly slow.  I found that disabling the virus protection “fixed” it but it took 25 mintues to get there.

The real problem was that Trend Micro’s Titanium (which I help load in a drive by, over the shoulder, can you help request), McAfee and Fix It Pro were loaded hence the “shoulder pad buildup” comment. In this case we had system protect buildup.  A quick trip to Trend’s  perfomance troubleshooting page to download the McAfee uninstaller and I saw a vast improvement.  Next I set scanning exceptions for the Fix It Pro software and we’re done. Everything’s back to normal.