Blog

Here’s an updated screenshot with my progress over the past few days:

You’ll notice now that the two emails with subjects in Hebrew and Greek (probably horribly translated, I just used Google Translate) now appear correctly in the main window.

Getting these messages to appear correctly was actually quite tricky to figure out. The key, in the end, was that I had to make a copy of the subject field before the Indy routine ProcessHeaders mangled it into Ansi characters, and then call my DecodeHeader algorithm on the un-mangled header.

The ironic thing is, if I upgraded to the latest version of Delphi, and the latest version of Indy (as in Indy10 rather than as in Indy9.0.53), the whole DecodeHeader and handling international characters would be a non-issue. But then I’d have a whole different can of worms–I’d have to fix all the non-backward compatible changes between Indy 9 and 10, I’d have to figure out how to convert ActionBand Popups into the new Delphi 2010 equivalent. In the end, what it would take to port the app to a newer version of Delphi is probably more difficult than coercing the old version of Delphi and Indy into doing what I want it to do. Sure, in the end it might end up being more robust that way, but I might also break and mangle lots of existing features.

At this point, what you see in the screenshot, I am only processing the Subject field with the new technique that works for any codepage (vs: only the current one or UTF-8), so I need to extrapolate my strategy to the other header fields that might be encoded. I also need to find an equivalent strategy to do the same thing on the Preview window. In the preview window it re-downloads the email through a different Indy code-path, and I haven’t found the right place in that code-path where I can capture the un-mangled header yet. So there’s still work to do but I’m on the right path.

To convert the random code-page to Unicode, I am using the windows library function WideCharToMultiByte, which converts a string to a “wide” (Unicode) string based on a specified code page number. Getting the code page number was also a little bit of a challenge. The library with that function doesn’t have a GetCodePageNumber function to convert the code page *name* to the windows code page *number*. There is a DLL that comes with windows that has (almost) that function, but figuring out how to call it is kind of tricky, and rumors on the internet say it might be buggy in certain cases. So, I’m using the straightforward but ugly strategy: convert the table on MSDN of allowed code-page names/IDs to a data structure and look it up manually. That list isn’t the full IANA supported list of encodings (aliases/alternate names), but the cases it doesn’t handle are likely to be rare, and could be added in the future if future research doesn’t find a better strategy.

I started out storing the list in a record (ie: Delphi equivalent of a struct) with a dead simple sequential search algorithm, per a tip on StackOverflow, but I wasn’t happy with the performance on lookup, because there’s 140 different encodings in my list of encodings so far, and this method is going to be called in a loop for each header of each email unless it’s definitely not encoded. So I did some research and found TStringList which can be used like a sorted map with better string search performance, and THashedStringList, which is basically a hash-map data structure, so even better string search performance–up to the level you need for, get this, INI FILES! Then I had to do more research to figure out what the Delphi equivalent of a static initializer is to use the Hashed String List…But now that I have it working, it does seem noticeably faster even on a small number of emails, but the internet could just be less congested today, it’s hard to say.

I had an email in my inbox today from Shutterfly with a subject in Base-64 encoding. Only, the subject was so long in Base-64 that it didn’t all fit on a single line in the email headers. Which means my base-64 subject decoder needs to be a little more robust.

The raw-header looks like this:

Subject: =?UTF-8?B?WW91ciBUaGFuayBZb3UgZ2lmdCBlbmRzIFdlZC4gfCBFbmpveSA=?=
=?UTF-8?B?JDIwIGp1c3QgZm9yIHlvdcKg?=

Notice anything odd about the format of this header? Each line has it’s own encoding tags, instead of one encoding tag spanning both lines, so it’s a little different than what I’ve seen on long subjects encoded with quoted-printable. Converted from Base 64 to English, the subject should parse into:

Your Thank You gift ends Wed. | Enjoy
$20 just for you

No weird special characters or anything that really necessitates base-64, but none the less to be a robust email parser, this kind of case should probably be handled.

In Delphi, I get a single string with the entire header except the word “Subject: “, but including the line break returned from Indy. This means I can’t just say from after the B? to the very end minus two characters, I need to actually tokenize the ending question-marks and/or whitespace, and encoding begin strings, perhaps in a loop, so I can pull out just the base 64 encoded part:

WW91ciBUaGFuayBZb3UgZ2lmdCBlbmRzIFdlZC4gfCBFbmpveSA=
JDIwIGp1c3QgZm9yIHlvdcKg


And then pass that lovely string into the base64 decoder. And then instead of a subject of

=?UTF-8?B?WW91ciBUaGFuayBZb3UgZ2lmdCBlbmRzIFdlZC4gfCBFbmpveSA=?= =?UTF-8?B?JDIwIGp1c3QgZm9yIHlvdcKg?=

you could have a subject of

Your Thank You gift ends Wed. | Enjoy $20 just for you

Two bugs fixed! The first was trivial, adding one line of code for a case I had overlooked initially. The second was more subtle. I eventually tracked down the cause to a less common variant in the MIME headers.

Here’s a typical MIME header:

Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

And here’s the header I was seeing in the error producing email:

Content-Disposition: inline
Content-Transfer-Encoding: 8bit
Content-Type: text/html

Notice anything majorly different? Take a look at that content-type. In the second-example, it didn’t specify the charset, so there’s no semicolon after the content type string. And that meant that some existing code that parsed out the content-type would fail if the charset was missing. Position of semi-colon returning zero (not found) when you want it to return end of string if not found…slight difference, now it gets the full content-type in the latter case instead of a blank string and displays correctly.