IMAP transport logging

So if you’re building custom applications to connect with IMAP you may need to turn on logging for the IMAP transport layer.  Don’t confuse this with turning up logging on the IMAP4 Service to Expert.  That will just give you increased application logging.  I’m talking about monitoring transport and connection attempts.

To enable the logging for IMAP, you would do this:

1. Browse to C:\Program Files\Microsoft\Exchange Server\ClientAccess\PopImap
2. Open Microsoft.Exchange.Imap4.exe.config with text editor (e.g. Notepad) and scroll to the bottom
3. Modify ProtocolLog from false to true.
4. Modify LogPath as necessary
5. Restart the MsExchangeImap4 service

Another setting that you may find interesting is AgeQuotaInHours.  This prevents you from filling the log drive with old log files. The default is 24 hours so you’ll need to change this if you are having intermittent or ongoing issues.

You can do the same thing for POP, simply edit Microsoft.Exchange.Pop3.exe.config instead and restart the MsExcahngePop3 service.

Troubleshooting IMAP

Troubleshooting IMAP.

Unless you’re working at an ISP chances are you may never use IMAP if you’re a straight MS shop since outlook uses MAPI for its connections.  However, if you want to connect an application to your mail environment then you’ll probably be using IMAP.  This is because IMAP connections provide flexibility for connecting to subfolders, monitoring for new message arrival and downloading the messages that is not available in a POP connection. 

Enabling IMAP in Exchange 2007 is very straightforward.  One thing to keep in mind is that the Exchange IMAP service works just like the Exchange Transport Service Any changes to the configuration will require that the service be restarted.

My goal last week was to get our Siebel system connected to a mailbox on my Exchange 2007 server.  Sounds like a simple process.  I have another application using IMAP to import mail into an archive system. It works just fine.  So far Siebel does not play well with others. So I needed to convince myself and my consultants that the problems were specific to the application and not IMAP

Troubleshooting IMAP begins with a telnet session just like troubleshooting SMTP.  However, the similarities stop there.  If you have TLS enforced for IMAP then you’ll need to telnet to port 993. For non-encrypted connections port 143 is used.

This is really important - Just connecting is not enough.  You’re connecting to a mailbox, not relaying anonymous mail.  Some type of authentication is required so you must at least test that part of the process. Then, once you’ve authenticated, you’ll want to verify that you can see the folders in the mailbox. Often the application process will be able to connect but cannot authenticate or cannot list the items in the mailbox. Your job is to follow the same process the application is using to attempt to duplicate the problem.

You also need to know that IMAP supports multiple connections so each command must start with a unique connection tag.  It can be something as simple as a period or you can get fancy and use an alpha-numeric tag like A01. I won’t go into the details of the commands.  Rather I’ll give you a link to one of the best pages I’ve found explaining IMAP, here. He also covers POP. You’ll want to review it too since there’s a good chance the application will want to send mail using POP.

At a minimum I’d suggest the following:

telnet <server> 143

. login mailbox@domain.com password

. list "" "*"

. status INBOX (messages)

These commands will test the basic connectivity.  Again, I suggest that you duplicate the process your application is uses using the commands to fetch a message and read it.

If a MIME uses email does anyone read it?

The challenge a couple weeks back was to finish moving the remaining applications that are using my Exchange 2003 bridgehead servers for SMTP services to one of my message hygiene servers.

The last remaining application is in Oracle.  When pointing to the Exchange 2003 server it works.  When pointed to the Exchange 2007 server or to my Trend IMSS server it drops the first line of data,

Here’s the code they’re using.

Oracle code

l_mail_conn := utl_smtp.open_connection(l_mailhost, l_smtp_port);

-- SMTP on port 25

-- l_mailhost ex2003SRV

utl_smtp.Helo(l_mail_conn,l_mailhost);

utl_smtp.Mail(l_mail_conn,l_msg_from);

utl_smtp.Rcpt(l_mail_conn,p_msg_to);

utl_smtp.Open_Data(l_mail_conn);

utl_smtp.Write_Data(l_mail_conn, 'Date: '||to_char(sysdate,'Dy, DD Mon YYYY hh24:mi:ss')|| utl_tcp.crlf);

utl_smtp.Write_Data(l_mail_conn, 'From: Table Switching'||utl_tcp.crlf);

utl_smtp.Write_Data(l_mail_conn, 'To: '||p_msg_to||utl_tcp.crlf);

utl_smtp.Write_Data(l_mail_conn, 'Subject: ' ||p_msg_subject||utl_tcp.crlf);

utl_smtp.Write_Data(l_mail_conn, p_msg_text);

-- p_msg_text is the message

utl_smtp.Close_Data(l_mail_conn);

utl_smtp.Quit(l_mail_conn);

I was sure that this was a 7 to 8 bit MIME conversion issue.  In the olden days all mail consisted of straight ASCII characters. Email has evolved to handle attachments and rich text items such a font changes and even HTML code but SMTP still remains a 7 bit system. So email with this type of data is typically encoded in 8bitmime and then encapsulated in 7-bit packets for transmission.  The receiving server then converts it back to 7-bit format.  I first looked at this in depth when troubleshooting an issue with receiving this type of mail from a client. Trend IMSS does not aways complete the conversion process correctly with IMSS version 7.0.  Their workaround is to turn off the 8bitmime advertisement.  I’m not willing to do this because it adds about a 33% increase in overhead on the server and in the size of a mail.

Checking the IMSS server, the 2003 and 2007 servers I found that they all three advertise 8bitmime.  Since the 2003 server is advertising 8bitmime I doubt it's a cnversion issue. Looking at the code I also see that they are issuing a HELO command not an EHLO command so the ESMTP 8bitmime advertisement shouldn’t be negotiated. I need to look elsewhere.

The challenge is that I need to see the raw mail to understand why that line is dropped.  If this was an HTML formatted mail I could view the source, but this is text.  Outlook doesn’t provide a method for view a raw message, just the internet headers.

I need to grab the mail before Outlook does any conversion. I decided to try MFCMAPI.  This allows a direct MAPI connection to a mailbox without using Outlook.

 Raw Mail

Looking at the raw mail with MFCMAPI the body of the corrupt mail looks like this:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">

<HTML>

<HEAD>

<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">

<META NAME="Generator" CONTENT="MS Exchange Server version 08.01.0240.003">

<TITLE>Table Switching </TITLE>

</HEAD>

<BODY>

<!-- Converted from text/plain format -->

<BR>

<P><FONT SIZE=2>23-FEB-2011 04:26:04</FONT>

</P>

</BODY>

</HTML>

The body of the correctly displayed mail looks like this:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">

<HTML>

<HEAD>

<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">

<META NAME="Generator" CONTENT="MS Exchange Server version 08.01.0240.003">

<TITLE>Table Switching</TITLE>

</HEAD>

<BODY>        

<!-- Converted from text/plain format -->

<P><FONT SIZE=2>23-FEB-2011 04:40:43 - Database: PRACDSD.KCI</FONT>

<BR><FONT SIZE=2>SUCCESS - DL_ZPD48943DM9403.RPT_PATIENT </FONT>

</P>

<P><FONT SIZE=2>23-FEB-2011 04:40:47</FONT>

</P>

</BODY>

</HTML>

I’ve highlighted the problem in red.  MFCMAPI is a thin client so this is displayed in an HTML rendering. <BR> is an HTML equivalent of ASCII CHAR 10 13 (carriage return line feed).  According to RFC 2045 8bit email transfer supports "up to 998 octets per line with CR and LF (codes 13 and 10 respectively)" but this is only allowed to appear as part of a CRLF line ending.

What’s happening is the mail servers are interpreting the CHAR 10 13 as the end of the line and dropping the remaining characters. So why does it work when the mail is sent to the Exchange 2003 server?  It's been my experience that the Exchange team sometimes plays a little fast and loose with their compliance to the current RFC standards.  In this case we were able to get away with being a little sloppy with our coding until the compliance was tightened in a newer version of Exchange. Changing the placement of the CHAR 10 13 resolved the issue.

Here are some links to more detailed MIME conversion and ESMTP articles.

http://searchexchange.techtarget.com/tutorial/Part-3-How-Extended-SMTP-works-and-common-ESMTP-commands

http://technet.microsoft.com/en-us/library/dd535395%28EXCHG.80%29.aspx

http://download.oracle.com/docs/cd/E19636-01/819-2650/bgamw/index.html

http://technet.microsoft.com/en-us/library/bb232174.aspx

http://support.microsoft.com/?id=323489

http://support.microsoft.com/?id=836555

http://en.wikipedia.org/wiki/MIME

http://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol

Content conversation2010 –

http://technet.microsoft.com/en-us/library/bb232174.aspx

Connecting our Anti-spam to Exchange 2007

The next step in shutting down our old Exchange 2003 servers is to configure our anti-spam appliance.  If you’re looking for a great anti-spam vender check out Xecuritas. We’re in the healthcare industry which makes it tough to filter based on the body parts and drugs listed in your typical spam message. These guys do a great job and proved stellar service.

So the process involves setting up a NAT for LDAP authentication so folks can log in and check their quarantines.  This NAT points to a DC and provides a one-to-one connection from the anti-spam appliance.  It only accepts those 2 IP addresses and is only open for LDAP traffic.

I also have a NAT for recipient verification. This allows the appliance to check the recipient for every mail and if the recipient doesn’t exist we drop the mail.  Yes, I know the current RFC says we should bounce that mail, but hey, during a heavy spam flood we may see 2.7 MILLION messages a day.  I can’t really afford to double that kind of traffic with NDR’s This NAT is also a one-to-one connection from the appliance to my HUB server. It only allows SMTP traffic.

The Exchange 2007 native anti-spam features are installed using the install-AntispamAgents.ps1 script located in Program Files\Microsoft\Exchange Server\Scripts. All of the anti-spam options are enabled by default. To turn on recipient filtering Go in the properties of recipient filtering and enable the option "Block messages sent to recipients not in the global address list" on the Blocked Recipients list.  Click Apply/OK. Restart the Microsoft Exchange Transport Service again.

I know what you’re thinking. My boss asked the same thing, “What about all of those mail enabled public folders that you have hidden?”  They still work. They are still in the GAL.   Hiding them does not remove them from the GAL. They simply have a database flag set the hide them.

The final Exchange configuration piece is to disable tarpitting.  This feature is to help prevent directory harvest attacks.  It’s a delay in the rejection response.  The idea is that the spamming server will drop the connection before the rejection is generated. If your Exchange server is open to outside connections this is a good thing. I have two hygiene servers between my Exchange environment and the rest of the world so I don’t need this feature.  In fact I want that rejection to happen instantly so mail doesn’t back up on my inbound mail server.

You can check your tarpitting interval from the command prompt.

Get-ReceiveConnector | select name,tarpitinterval

It’s turned off by the command line too.

set-ReceiveConnector "Server1\Default Server1" -tarpitinterval 00:00:00

set-ReceiveConnector "Server1\Client Server1" -tarpitinterval 00:00:00

set-ReceiveConnector "Server2\Default Server2" -tarpitinterval 00:00:00

set-ReceiveConnector "Server2\Client Server2" -tarpitinterval 00:00:00

OAB Generation Issues

We make extensive use of public folders.  Every distribution list files into a mail enabled public folder.  I’ve tried to take full advantage of 2007’s distributed architecture.  I have three clustered mailbox servers in three separate datacenters. Since CCR is not supported while hosting public folders I also have three public folder servers in each datacenter. (They host-super-huge-ginormous-mongo public folder databases. I lose sleep over this, but we’re not going there right now.) Recently I’ve seen an uptick of instances where mail enabled public folders never generate an SMTP address. Cycling the system attendant service fixes it.  There’s no indication of a problem other than getting alerted that the proxy address is not available when going to the folder permissions.

I’m slowly edging my way toward shutting down the final Exchange 2003 system. So this week I moved the Offline Address Book generator to one of the 2007 servers hosting public folders. No big deal right?  Testing in the lab, it was just a process of right click and pick an exchange 2007 server. The uptick in SA problems coincide with the OAB move. I also found 9386 and 9399 warning in the event log.

The description of the error looks like this:

OALGen is configured to generate version 2 or version 3 OAB files for offline address book '/o=Organization/cn=addrlists/cn=oabs/cn=Default Offline Address List' but there is no public folder server available. OAB versions prior to version 4 require a public folder server and cannot be generated at this time. Please ensure that a public folder server with a replica of the Offline Address Book system folder is online and mounted, or disable all OAB versions other than version 4.

Thinking back to moving the public folders (there were ~17,000 of them. ~3.5 million messages for a total of 4T of data) I double checked to make sure my OAB system folders are replicated to the 2007 public folder server.  They were. Here’s a very important piece I’ve not told you.  My public folder servers do not host mailbox databases.  I did this so that mailboxes could not accidentally be created on a “public folder” server. Keep this in mind while you read “Troubleshooting MSExchangeSA 9386 Warnings and OAB Generation and Access Issues”

Back already? My System Attendent service did NOT have a HostMDB property. If the System Attendant service does not have a host mailbox database there are quite a few actions that it cannot perform – database cleanup tasks are one, and OAB generation are another. So I quickly created a mailbox database and, using ADSIedit, gave the System Attendant a host database property.

IF you’re following along because you had the same issue, you should now see Event ID 4002’s –

An unexpected failure has occurred. The problem will require administrator intervention. The service will retry in 56 seconds. Microsoft.Exchange.Data.Storage.MailboxUnavailableException: Cannot open mailbox

 If you have ever had to move the System Attendant mailbox before then you might recognize this one.  Restart the Microsoft System Attendant service and it will create a new mailbox.

Finally I get Event ID 9331’s –

OALGen encountered error 8004010f (internal ID 501080c) accessing the public folder store while generating the offline address list for address list '\Global Address List'.

There are followed by an Event ID 9358 –

OALGen could not update the timestamp of the previous version of offline address list '\Global Address List'.  Please check earlier event log messages for the specific error.

Updating the offline address lists manually from the Exchange Management Shell using the commands below fixed these errors.

Get-Addresslist | Update-Addresslist

Get-GlobalAddresslist | Update-GlobalAddressList

Get-offlineaddressbook | update-OfflineAddressbook

Update-FiledistrubutionServer

PF - DOES NOT - Admin

One of the very few things I’ve found in Exchange 2007 that cannot be done through the command line is check/ set the calendar permissions on everyone’s mailbox.  In 2010 this is done using the Add-MailboxFolderPermission command which unfortunately is not available in 2007. The next best option is to use PFDavAdmin.  Unfortunately it refused to work on one of my mailbox servers. I can connect. It provides a list of all the mailboxes on the server. Running the permission report creates a file with no data. When I try to expand the mailbox I get an error: Could not expand http://server.domain.doman/exadmin/domain.com/mbx/mailboxname.non_ipm_subtree/:The remote server returned an error (404) Not Found

I went through the standard troubleshooting of checking security settings, using a machine in the same physical subnet, using a different privileged account, etc.  Microsoft suggested using PFDavAdmin’s replacement ExFolders however that only works if you have a 2010 server.

3 calls and three engineers later I ended up checking the Dot Net versions and digging deeper. Finally I ended up replacing the OWA exadmin virtual directory. I found someone whoi had a similar issue here on the Exchange Server Tech Center. SO I read up on it between getting put on hold and actually fixing the issue.

First I exported the folder settings to a text file.

Get-OwaVirtualDirectory <Server>\exadmin* |fl >d:\foldersettings.txt

You need to know what format the folder is using which is done like so:

Get-OwaVirtualDirectory <Server>\exadmin*

Name                       Server                     OwaVersion

----                       ------                     ----------

Exadmin (Default Web Site) <Server>               Exchange2003or2000

I needed to know that it was using Exchange 2003or2000. Then I removed the virtual directory using this command:

Remove-OwaVirtualDirectory <Server>\exadmin* -verbose

The verbose switch gives this output:

VERBOSE: Remove-OwaVirtualDirectory : Beginning processing.

VERBOSE: Remove-OwaVirtualDirectory : Administrator Active Directory session

VERBOSE: Remove-OwaVirtualDirectory : Searching objects "<server>\exadmin*"

 of type "ADOwaVirtualDirectory" under the root "$null".

VERBOSE: Remove-OwaVirtualDirectory : Previous operation run on domain

controller 'dc.local'.

Confirm

Are you sure you want to perform this action?

The Remove-OwaVirtualDirectory cmdlet is removing Outlook Web Access virtual

directory "<server>\exadmin*".

[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help

(default is "Y"):y

VERBOSE: Remove-OwaVirtualDirectory : Deleting "<Server>\Exadmin (Default

Web Site)" of type "ADOwaVirtualDirectory".

VERBOSE: Remove-OwaVirtualDirectory : Previous operation run on domain

controller 'DC.local'.

VERBOSE: Remove-OwaVirtualDirectory : Ending processing.

The command below creates a new virtual directory, using the OWA version type displayed in the first Power Shell command.

New-OwaVirtualDirectory exadmin -owaversion Exchange2003or2000 -virtualdirectorytype exadmin –verbose

The verbose switch gives you this output.

VERBOSE: New-OwaVirtualDirectory : Beginning processing.

VERBOSE: New-OwaVirtualDirectory : Administrator Active Directory session

settings are:  View Entire Forest: 'False', Default Scope:

'Domain.local', Configuration Domain Controller:

'DC.l.local',

VERBOSE: New-OwaVirtualDirectory : Searching objects

"<Server>.Domain.local" of type "Server" under the root "$null".

VERBOSE: New-OwaVirtualDirectory : Previous operation run on domain controller

'DC.local'.

VERBOSE: New-OwaVirtualDirectory : Processing object "<Server>\exadmin".

VERBOSE: Creating Outlook Web Access virtual directory "Default Web Site" on

server "<Server>.Domain.local".

VERBOSE: New-OwaVirtualDirectory : The properties changed are: "{

OwaVersion='Exchange2003or2000', VirtualDirectoryType='Exadmin',

FolderPathname='\\.\BackOfficeStorage', Url={ 'http://<Server>:80/exadmin'

}, MetabasePath='IIS://<Server>.Domain.local/W3SVC/1/ROOT/exadmin',

AuthenticationMethodFlags='Basic, Ntlm, WindowsIntegrated',

AuthenticationMethodFlags='Fba', Id='<Server>\exadmin (Default Web Site)'

}".

VERBOSE: New-OwaVirtualDirectory : Saving object "<Server>\exadmin (Default

 Web Site)" of type "ADOwaVirtualDirectory" and state "New".

VERBOSE: New-OwaVirtualDirectory : Previous operation run on domain controller

'DC.local'.

VERBOSE: New-OwaVirtualDirectory : Searching objects "<Server>\exadmin

(Default Web Site)" of type "ADOwaVirtualDirectory" under the root "$null".

VERBOSE: New-OwaVirtualDirectory : Previous operation run on domain controller

'DC.local'.

Name                       Server                     OwaVersion

----                       ------                     ----------

exadmin (Default Web Site) <Server>               Exchange2003or2000

VERBOSE: New-OwaVirtualDirectory : Ending processing.

So why does this work?  According to this TechNet article the OWA virtual directories are used to “handle Exchange 2003 Outlook Web Access requests, Exchange 2000 Outlook Web Access requests, WebDAV requests, and some administrative functions.” Ok, I knew that but why ExAdmin? We the ExADmin virtual directory is used for Exchange Administrative (or like TechNet says here, to change administrative settings and properties.) access just like the name implies and PFDavAdmin must access it to make permissions changes etc.

Exchange 2007 – Where Nothing Happens Fast

One thing that I really like about exchange 2007 is The Exchange Management Shell (and Powershell). I try to use it for most of my daily tasks but sometimes it’s just quicker to right-click and pick new.

... until you make a mistake. I meant to create a mailbox database but I created a public folder database.  No big deal right? just delete it. There’s no data on it? Not so fast. Here’s what I get:

Remove-PublicFolderDatabase : Exchange is unable to check the public folder rep

licas. Verify the Microsoft Information Store service is running, and that the da

tabase is properly mounted

OK. FINE I’ll mount the database. GRRRR now I get this:

The public folder database contains folder replicas. Before deleting the public

folder database, remove the folders or move the replicas to another public

folder database. For detailed instructions about how to remove a public folder

database, see http://go.microsoft.com/fwlink/?linkid=81409.

I’m going to rant a little here. I’m wrapping up my 2003 to 2007 migration this month. I really liked that Exchange 2003 had pretty much done away with the little idiosyncrasies we’d come to love/hate about earlier versions. Gone were the days of “Exchange is just that way.” Well I’m starting to see that those days are back.  2007 puts a copy of the public folder hierarchy on all public folder databases.  This is a good thing. But what this means for me right now is that I have to wait for this database to replicate to all of my other public folder servers around the world. I also have to wait for the store cache to flush which may take 2 hours. Then I can delete this database.

Well I have things to do. I can’t wait around all day until Exchange decides it can find all the replicas for this database.  So firing up ADSIEDIT I find the Exchange Administrative Group for 2007, drill down to the Server CN.  Next I expand the container for the server with the wayward public folder and expand the information store container.  Expand the Storage group and delete the database. Now all that’s left is to manually delete the edb file and logs and I can fix my mistake and go home on time.

The first post - Shoulder Pad Build-up

For the guys out there – my wife introduced me to this term. It happens when every top a lady is wearing has shoulder pads so that they end up looking like an NFL linebacker. 

This week I was asked to fix a friend's computer.  The complaint was, “It’s not doing anything.” I found that it was in fact doing lots of things, all at the same time and not doing any of it very well.  It booted up quickly enough, but once I had logged in everything became agonizingly slow.  I found that disabling the virus protection “fixed” it but it took 25 mintues to get there.

The real problem was that Trend Micro’s Titanium (which I help load in a drive by, over the shoulder, can you help request), McAfee and Fix It Pro were loaded hence the “shoulder pad buildup” comment. In this case we had system protect buildup.  A quick trip to Trend’s  perfomance troubleshooting page to download the McAfee uninstaller and I saw a vast improvement.  Next I set scanning exceptions for the Fix It Pro software and we’re done. Everything’s back to normal.