ESA Sink

If you found any of the tools on this page helpful, any donations would be appreciated.

Exchange SpamAssassin SMTP Sink

Date	Comment
8/17/2007	ESA and SpamAssassin have been tested through version 3.2.3. Note that version 3.1.8 and 3.1.9 do NOT work in Windows environments.

How to use an Exchange SMTP Transport Event Sink to eliminate SPAM.

On my never ending quest for the elimination of SPAM, I've looked at a lot of products. Most of the free Anti-Spam products that I've looked at work only with POP3 email, not Exchange based.

However, I was quite interested in SpamAssassin. SpamAssassin is a perl based spam filter program that utilizes a series of rules to flag mail as Spam. It's quite a robust program, having been used in the Unix world for many years.

There is a link off of the main SpamAssassin page regarding getting SpamAssassin to work in a Windows environment. This work was done by Michael Bell, and while a little complicated, by following all the steps there, I was able to get SpamAssassin to run on a Windows machine. Now, there are a couple of issues with this.

SpamD, the SpamAssassin Daemon, doesn't compile with normal perl, so you are forced to use SpamAssassin in serial mode. This means that for each email that you test, a copy of Perl must be started. A little bit of a bummer, but my mail server load is relatively light.
None of the community checking systems (DCC, Razor, Pyzor) work.
SpamAssassin creates a batch file that reads mail from stdin, and exports the resulting mail with additional headers in stdout.

How in the heck would this work with Exchange? Michael has done some more work on getting SpamD to work under Cygwin, but I wasn't quite ready to move to this. The normal serial mode batch file would work, but how would you implement this with Exchange?

Michael did have a link on his system to a SMTP Sink written by Eric Smith. However, Eric obviously ran in to some of the more common problems with SMTP sinks - The CDO api is really poorly documented. His script did the following:

Copy the Exchange message object to a common text file.
Run SpamAssassin with the common text file as input, outputting to another common file.
Parse the output file, hunting for the spam result.
If the message is spam, either change the subject line, or reject it.
If it is not spam, just exit.

While workable, this is really not taking advantage of all the capabilities if SpamAssassin, limited as the are in the Windows world. I wanted to totally integrate SpamAssassin with the Transport Event Sink, so that for each message received, the sink would replace the original message with the one processed by SpamAssassin, just as originally intended.

Thus I began my search on SMTP Transport Event Sinks and CDO. I started off at the CDO for Windows MSDN site, which covered most of the basics for writing Event Sinks. The sink that need to be implemented is the OnArrival sink. Basically, my thoughts were the following:

Output the entire message object to a text file.
Process the text file with SpamAssassin.
Reload the resulting file into the current message object.
Parse the message object's headers for the SpamAssassin headers.
Archive the text message object to a Ham or Spam folder for later Bayesian evaluation.

After mucking around with a couple of sample scripts, I happened to read something about being able to access the message object as an ADO.Stream object. Bingo! So now, using the CDO Message's GetStream method, I could write the message object to a text file, and then using the ADO.Stream's LoadFromFile method I could reload the processed file! Instant SpamAssassin headers.

Usage:

To install the sink, download and install the ESA-setup program. Note that you will need to have a working install of SpamAssassin on your Exchange server.

I've tried to have the install script do pretty much everything it can, but you will need to edit the INI file to point to the SpamAssassin.BAT file in your perl directory.

The Exchange SpamAssassin directory structure looks like this:

C:\ESA
├───Archive
│ ├───Ham
│ └───Spam
├───Ham
├───Logs
├───New
└───Spam

ESA-Setup will install the Sink for you, or you can run Install_ESA_Sink.bat. To remove the Sink run Remove_ESA_Sink.bat.

Please see the ReadMe.txt for further instructions.

Thanks,

Chris Lewis (2004-04-13)

Download:

Installation program: ESA-setup.exe (318 KB )

Source Code: ESA-Source-2007-04-23.zip (61 KB)

Change Log:

2/7/06	Added two small fixes to the Exchange SpamAssassin Sink. 1) Added MaxMsgSizeKB item to the INI file. If the message being processed is larger then this size in KB, then the message is never sent to SpamAssassin. Should help speed up processing on large files. 2) Added a section of code to fix an issue that someone discovered regarding messages that seem to be missing a final CRLF. The fix checks to see if the message ends in CRLF and if not, adds it. This seemed to fix a hang with POP3 processing of some ESA processed files.
1/17/06	3.0.5 has been release as an upgrade to 3.0.4 in the Apache Archives. Still no word on the 3.2 branch.
11/14/05	A fix to the issue has been checked into CVS. 3.2.0 should be the release to aim for. I'll be installing the CVS version on my server this week. (update 1/3/06 CVS works fine as far as I can see...)
10/5/05	SpamAssassin has released version 3.1.0, but ESA doesn't work with this version at this time. I will be addressing this shortly.
9/27/04	Updated the source ZIP file to include the modShellAndWait.bas file. Sorry about that...
9/9/04	Small change to the XSpamStatus parsing to work with SpamAssassin 3.0
9/6/04	Updated ESA to use CreateProcess instead of WshShell. Some reports of issues with rights and the Windows Scripting Host. CreateProcess also allows us to return the actual error that is thrown by the server if the process can't be created.
6/28/04	With lots of help, the EF BB BF mystery has been solved. It's just the Unicode UTF-8 BOM indicator that the stream is being stored as unicode. Very simple code change to fix this, you just use objStream.CharSet = "ASCII".
4/20/04	Exchange SpamAssassin Sink. Some major fixes to the install (it should actually work) and a small bug fix for the headers. If anyone out there would know why CDO is adding an additional EF BB BF to the mail headers on Stream.LoadFromFile it would be appreciated.
4/21/04	Exchange SpamAssassin Sink. Another small update to fix the ReallyBadSpam threshold.
5/1/04	A lot of people are seeing the post processed files containing just the following: "You do not have Perl in your PATH". This is due to the way Windows Services get their environment. Environments are inherited from parent processes. Exchange runs as a service under Service.EXE. Service.EXE is loaded at boot-up, so in order to change the Path, you need to reboot the service. If anyone knows of a way to force a process to re-evaluate the environment, please let me know. For now, just reboot. I'll also change the code to look for this.
5/4/04	Better handling of "You do not have Perl in your PATH" error.
5/5/04	Exchange SpamAssassin Sink. Updated temp file naming for large volume sites, small fix for, International Number handling and X-Spam-Value.
5/6/04	Exchange SpamAssassin Sink. Fixed issue w/ temp file name.
5/18/04	As Daniel Owen and I figured out, you need to specify the directory for your Bayesian database in order to be able to have Exchange read the same one as you are training. Add to your \perl\share\SpamAssassin\local.cf the following: #Where is the BAYES DB? #bayes_path /path/to/file (default: ~/.spamassassin/bayes) > #Use C:\.spamassassin bayes_path /.spamassassin/bayes If anyone knows how to specify an alternate drive in the bayes_path config, send me a note.

Copyright and License:

The use and distribution terms for this software are covered by the Common Public License 1.0 (http://opensource.org/licenses/cpl.php) which can be found in the file CPL.TXT at the root of this distribution.
By using this software in any fashion, you are agreeing to be bound by the terms of this license.

You must not remove this notice, or any other, from this software.

References:

VamSoft: Scripts and Tools - Great notes on ISMTPOnArrival scripting.
Microsoft: MSDN CDO for Windows - API reference.
Michael Bell - For his 'USING SpamAssassin WITH WIN32' http://www.openhandhome.com/howtosa.html
Eric Smith - For his original VBScript http://www.cardboardutopia.com/ExchangeSpamFilter.zip
SpamAssassin Team - http://spamassassin.org/
Inno Setup - Jordan Russell's wonderful setup program. This setup tool ROCKS! http://www.jrsoftware.org/isinfo.php

Trademark Notice and Disclaimer:

Exchange® is a registered trademark of Microsoft Corporation.
Microsoft® is a registered trademark of Microsoft Corporation.
SpamAssassin™ is a trademark of Network Associates, Inc.