PopTrayU Bugtracker

Originally, for keeping track of issues with PopTrayU I attempted to use Sourceforge’s built in bug tracker. But, as I recall, I was less than enthralled with it for reasons I’ve long since forgotten.

putbugtracker2So then I started looking into other options, PHP based options I could run from the webhosting area on Sourceforge, and ended up selecting Mantis. I think I looked into bugzilla as well and I don’t remember what else. For the most part Mantis worked quite well, and had a slick-looking interface, and I loved the dashboard view where you could sort all the bugs by their status and it color-coded them in a way that just “made sense”.

But Mantis and Sourceforge just don’t play well together. Every time you submit a change to a bug, the next page you try to load returns a 503 “Guru Meditation” error due to the Varnish Cache Server installed by Sourceforge. Apparently it’s some sort of timeout error, and I’d spent a lot of time looking into fixing the issue, possibly more than once, without success.

So what ends up happening in reality? It’s too much trouble to update the bug-tracker in everyday use So I just keep track of “active” bugs in a spiral notebook sitting on my desk with a pencil and only using the bug tracker only for keeping track of “things I might implement someday”.

So I decided to give this version tracking issue another look. Wanting to stay away from Soruceforge’s cache server issues, I started looking at the options for bug tracking used by some other open source projects I follow.

Google Code is deprecated and on the verge of being shut down and not open to new projects, so although I’ve liked them in the past that was out.

Microsoft’s CodePlex I love how they prominently display options to vote on your favorite bugs. Super-useful when you’re thinking “which bug should I tackle next”.codeplexAnd it has a nice taxonomy for marking up the bugs–separate drop-downs for component, status, type, impact, etc for well-classified bugs. But it’s kind of boring and corporate, lacking color-coding, and the captcha nearly resulted in loss of data for both of the two first test entries I entered.

And then there’s the ever-popular GitHub. It’s bug tracker is simple–at times almost too simple. Component can’t be a separate field from the bug status–it’s all just a mashup of tags. I’d rather see the list of components in my project separate from the resolutions. But overall the bug tracker has some nice CSS and is quick and easy to add new bugs and update their status.

github-bugsIn the end I decided ease of entering and updating bugs was more essential than taxonomies and voting. If it’s not practically effortless to track the bugs, there’s a good chance I won’t use the bugtracker to keep track of features I’m considering developing.

So here’s the new bug tracker: https://github.com/jojobear99/PopTrayU/issues

PopTrayU 5.1.0 Options Preview

I wanted to give you all a little preview of more of the new features I’ve been working on for the 5.1 version of PopTrayU.

To start here’s a screen-shot of a new panel/category on the Options tab to customize the Preview Window:

poptrayu_5.1_previewLots of subtle differences. Before we get too far, I’m sure someone is thinking “What’s with the font?” No, 12 point Segoe Print is not a new default (it’s something I picked arbitrarily for testing). But it does illustrate a new feature–large fonts support. 12 point isn’t exactly “that large”, but it’s large enough that things would be overlapping and cut off, and created a horrible user experience for for users who use windows large fonts feature, or just wanted a larger font in this application.

I’ve painstakingly gone through just about every screen in the application and added resizing code to make just about every screen “resize correctly” in both the vertical and horizontal directions if you use larger fonts. As an added bonus, this actually improves the user experience for users of languages other than English as well, because buttons and label areas that used to be fixed size now will resize intelligently, and increase in size, if the translated caption doesn’t fit on default sized button. At first I tried to do this with a grid layout manager, because that would have been the elegant, clean way to do things, but after wrestling with some frustrating bugs in how it auto-sizes, it ended up being more reliable to just position everything manually. So it’s quite possible somewhere I overlooked *something*, but we’ll call that a bug for now, and feedback on resizing issues that need to be addressed will be welcome.

Now, let’s talk about the list of tabs. I gave a minor overhaul to the list of categories, and made some changes to the order of the categories. Defaults has been moved to the first category, because it holds the option to change the language. And as a french-speaking user pointed out, you need to change the language before you change any other setting on a new installation. Seems sensible, so language selection is now at the very top of the first options screen. There are also two entirely new categories of options: Preview Window, and Rules.

The Rules options category gathers together existing options relating to rules that were previously spread across several tabs. Options like whether to log rules, whether messages marked as protected by rules require an extra confirmation to delete, whether to download part or all of the message body to make more sophisticated rules, and what happens when you blacklist something are all gathered in one place under a single logical category.

poptrayu_5.1_rules_panelYou may have also noticed that the old “Black List mark as Spam” checkbox has been replaced with the new “Blacklist Action” drop-down. I didn’t feel like it was intuitive what happens if you **don’t** mark blacklisted messages as spam–and the alternative involves things getting deleted. I take emails getting deleted seriously, and don’t want emails deleted because of bugs or because some configuration option was vague or unclear. So the drop down now forces an explicit choice between “Mark as Spam” or “Delete”. And I always do lots of extra testing (including on my own personal email) before releasing any changes that affect message deletion. I don’t want PopTrayU deleting your important mail–unless you really meant to do so.

The other new category in the list is Preview Window. There’s a couple of new features here, but previously, most of these options you had to change from a right-click menu on an actual preview window.

So, going back to that first screen-shot, you’ll see a new Tabs sub-category. This is for a new feature I’ve been wanting for some time. I’ve been torn over the design decision that PopTrayU should preview using the last selected tab or view type. I kept finding myself wanting PopTrayU to behave more like Outlook, where it automatically displays the email in HTML if the email has an HTML section, regardless of what view I was using to preview the previous email. But sine a lot of different people use PopTrayU in completely different ways to accomplish completely different goals, I didn’t want to eliminate the current option to have it display in the last used tab. So now you can choose between previewing with the last selected tab, or to always use a specific tab by default.

And, since Spam is sort of a special case, it gets its own independent selection. The main reason for this is that it’s likely that if you have images enabled, you may not want to display the email with images if it’s spam. So now you can choose whether you’d rather view spam as Plain Text, RAW headers, or any of the options supported for non-spam emails.

Back to the preview tab from the first screen shot. What can be customized for preview but isn’t on this tab? Fonts and Colors. When I first created this tab, initially I had put the options to customize the preview window font and colors (does not apply to HTML messages) on this new panel, however, I decided it would make more sense to have all of the font face, font size, and color customizations on the Visual appearance tab in one place.

poptrayu_5.1_visualAnd while I was at it, I finally fixed that eyesore that was the fonts selection panel. Originally I’d had the colors for the message list as separate drop-downs, which minimized the number of mouse-clicks needed to customize everything. But I decided it was getting a little too cluttered,  and confusing since only certain UI items can have the colors customized. I changed the previews to have borders–this was actually something I had wanted to do before, but there were some technical hurdles that required research to figure out how to make that happen.

Introducing the New Accounts Screen

For the upcoming 5.1 release of PopTrayU, I’ve redesigned the accounts screen:

5.1accountsIn the old design, which was basically the PopTray 3.2 layout with new features squished in around the edges wherever they’d fit without changing things too much, things didn’t resize well. Look what happens if you increase the font size:

5.0.15_largefontsEverything is overlapping, some of the captions don’t fit. It’s a bit…cluttered. And if you use a language other than English, the translations often had to be abbreviated to fit the space available.

The new design manually resizes and positions everything in both horizontal and vertical directions to accommodate the selected font size and language settings. The account settings are broken into logical groups, like splitting apart the server information for your server from the ways PopTrayU notifies you about, to make it the page easier to navigate. The sections are each collapsible as well, so the advanced options or notification options can be hidden if you don’t want to see those sections. Oh, and those buttons at the bottom? They are now intelligently sized…if the caption doesn’t fit, they will be resized larger to fit the caption. Users of languages other than English will also especially appreciate this new feature.


Disabling Auto-Check

I recently finished implementing a new feature for PopTrayU, conditionally disabling automatic checking for an email account based on the time of day. This feature already existed on a global level: you could disable checking for all accounts, so that your computer would not check for email while you’re sleeping or awaken you with a loud “you have new email” notification sound. But some users, particularly users who work from home and use the same computer for both work and pleasure, were making a feature request that they’d like to be able to turn off notifications for their work email outside of normal working hours.

The part of the implementation that was most difficult, of course, was the requirements analysis and the detailed design. On the surface, this feature seemed pretty straight-forward, but as I started to dive into actually implementing it, a lot of nuances come up where it isn’t exactly clear how this feature should work, and the design has to be hashed out in more detail, and the requirements clarified.

Each account has a status icon indicating whether the account is disabled or has new email or no new email. (For the sake of clarify, lets call this new per-account no-auto-check feature “sleeping”.) Should the visual appearance of an account that is “sleeping” appear like a normal account or a disabled account? Something else? What should the status-bar for the account say when an auto-check happens, but a single account is skipped because of the time? If the user presses “Check All Accounts” manually, should that skip or include accounts that are sleeping? What if they press “Check” for a single “sleeping” account?

Once I nailed down the design, implementation was pretty simple. Because the feature existed on a global level, most of the implementation was quite similar to the logic and UI of how it was done for the global level.

As far as how I decided to resolve some of those issues:

An account that is disabled is represented by an envelope with a red minus symbol on it, like this:  Previously, this was the only reason a specific account would be skipped when an auto-check occurred. Now we have two reasons an account might be skipped. It could, for some users, be handy to be able to differentiate the reason the account is not automatically checking, especially should they be troubleshooting settings gone awry. I experimented with making a new symbol that resembled sleeping, like something with zzz’s or a moon, but given the limited amount of space in a 16×16 image, trying to make an image that is clear and recognizable is difficult, so I decided to use the same symbol as a disabled account, but use a contrastingly different color to differentiate the different status.  

As far as when to check/not-check:

  • Automatic timer-interval checking, pretty straightforward, sleeping accounts should be skipped.
  • Manually pressing the “check all” button: more ambiguous, it’s not really an “auto-check”, but it behaves a lot like one. In the end I decided “check all” should replicate the behavior of auto-check, but on-demand, so accounts that are not in a state allowing automatic checking should be excluded, so it should behave like a disabled account.
  • Manually pressing the “Check” (this account) button: this one seemed pretty straightforward, even a disabled account will check when pressing this button, so a sleeping account should definitely check in this case.

On the status-bar for the account, I decided it ought to say *something* when an auto-check is scheduled but skipped, if only to handle the case where account1 is sleeping and account2 is enabled. The UI only shows one account at a time, so a visible indicator that an account check happened, and the time is useful. So “account skipped’ with the time. Autocheck was scheduled, but skipped beccause of the no-check hours for that account. Hopefully that’s clear, and it it wasn’t, they could read the manual (to be updated in the next release) to find out what “account skipped” means.

PopTrayU to Lazarus/Free Pascal Porting Update

Well, I’ve gotten a lot of the “cosmetic” stuff fixed. Here are updated screenshots, that you can compare to the ones my last update about this project:

First, here’s the options screen.

You’ll notice it looks pretty much about right. I had to use a third-party time control (ZVDateTimeControls) to get the time picker working. Lazarus doesn’t have an equivalent of TDateTimePicker. They have a TDateEdit, but that only does calendar and not time only. But the third party one seems to work fine, though I’m not in love with the arrows on it, I’d rather see it using the native windows picker control under the hood, since cross-platform compatibility is not a primary concern of mine at the moment. But at least it’s functional for now.

Here’s the main window at the moment:

You’ll notice the main screen now looks “almost” like the real thing now. The toolbar is there, but the labels are in the wrong spot. I stumbled almost on accident onto the solution for how you do the labels on the right. Apparently you have to use TSpeedButton(s) on a TBevel instead of using a TToolbar to get labels on the right. Sounds pretty doable, but is lower priority than some of the other issues I’m dealing with. 

Those tabs also don’t appear totally “right”. You’ll notice there’s some weirdness about the borders for them. The tabs are a TTabControl because they don’t have individual content, it’s really just an array of accounts it’s representing. But there’s a key difference in Lazarus from Delphi that the TTabControl is not designed to have child components, so it’s resizing it wrong to make room for the components that should be nested on the tab. This might actually be kind of involved to fix. One guy supposedly submitted a patch at one time that made it work more like Delphi, but it got rejected because it didn’t work “right” for all the different platforms Lazarus supports. So looks like I could try to track down that patch and make it into a third party control, or I could change the TTabControl to a TTabSheet, and replicate the buttons onto every tab, but that would lead to a lot more UI overhead. But either way, it’s not worth fixing unless I find a way to fix the blocking issue that could be a showstopper.

Right now the blocker issue is Indy 9. In 2005, Marco van de Voort ported Indy9 to Free Pascal, at least for 32 bit architecture, not 64 bit. But then he decided to upgrade to Indy10 so it wasn’t maintained. So, showstopper issue, Indy9 isn’t working right. I’m focusing on the “test account” button right now since that’s about the simplest network-operation the app does. Connect to the Pop3 server, login, and disconnect. Login is actually working. The problem is disconnecting. It’s reading garbage instead of “+OK” for the disconnect message, so it fails to disconnect and throws an exception that causes other problems, like skipping actually closing the connection. The problem is, using Wireshark to sniff what’s actually going on in the network, the server is actually sending the correct response back. But what PopTrayU is seeing when I step through the debugger is half of the account’s domain name, “bear.net” Stepping through the routines where it moves pointers around to find the response, it looks like it’s reading the wrong place in memory, it’s skipping right over where it should be reading, and reading from some random spot a little bit farther out that presumably contains memory that was previously used for something else. Tricky right? The code doing this is totally obscure, and has very few comments, so it might take a while to understand it well enough to figure out *why* it’s reading the wrong address when it goes to disconnect. The warnings in Lazarus that it’s doing Pointer to Integer conversions could be a hint though. Theoretically that shouldn’t mess it up on 32 bit, but you never know, seems to be doing some bad pointer math somewhere.

Interestingly, poking through the Indy10 codebase, they appear to have reworked the whole “managed buffer” that’s giving me issues and eliminated this weird pointer math. And Indy10 is officially supported in Free Pascal, so I may have to bite the bullet and upgrade. Apparently there were some non-trivial issues when Renier tried to port to Indy10 once long ago, but perhaps they would be less nasty to resolve than buffer overrun issues.

Oh, and unrelated to all of that, I also have the tray icon partly working now. The right click menu is working, but changing the icon images is not. The rules and about screens are also on the todo list if I can resolve blocker issues. The about screen lost all the strings in the tables, because they’re encoded in binary in Delphi, so I have to retype them. And the rules stuff was using some components that have to be manually ported/fixed.

HTML Email Preview

One of the complaints I’ve heard about the feature of having html preview of emails is that spam-bots often track whether emails are read by embedding a unique image, and tracking if the image gets downloaded. It’s a legitimate concern, one I was well aware of when I added the HTML preview option. But it came down to being a value vs. risk proposition.

The HTML preview window in PopTrayU
The HTML preview window in PopTrayU

For version 1.0 of the feature, that seemed like a perfectly reasonable risk to accept. When I check my email from my phone, it displays in HTML and doesn’t have an option to disable loading images either. And, if like me, you run your email through a really good spam filter (like Gmail), you aren’t getting a lot of spam in your inbox in the first place. But that’s not necessarily the case for people using PopTrayU to check their ISP email account, which is a very common use case.

But that doesn’t mean it shouldn’t be somehow addressed in subsequent releases.

Initially, when I’d looked into HTML components, my first choice was a native Delphi HTML rendering component. But upon investigation, the component was too old and did not support CSS at all, which was a deal-breaker since font tags are completely deprecated these days. I mean, I could have added CSS support myself, but that is an extremely complex feature to implement correctly, with many corner cases. You’re pretty much writing a web-browser minus the network connections part. Much bigger scope than I was willing to take on.

So my second strategy was to use the TWebBroswer control to render the HTML, which is essentially a wrapper for an embedded internet explorer window. You get the HTML rendering almost for free with that technique. But…Internet explorer has this known bug that the “offline mode” flag in the component is completely ignored, and it decides whether you are online by a global registry setting or whether an internet connection is present. Less than ideal for this purpose. Any external content is really undesirable for spam email previewing.

So then I looked into maybe I could just pre-process the HTML and strip out the images. Removing <img> tags wasn’t too difficult. You can either do that with a regular expression or a for loop iterating through the characters in the message. Well, it turns out to be a little more complex than that. Images don’t just come from <img> tags anymore. Images are embedded in CSS that’s slapped in the body section of the message using url(…) syntax. Images are embedded through inline CSS. Almost any CSS component could have an image as part of a background or bullet image or border or any number of other places. And then you need to strip out any javascript, because that might be loading images or external content, so there go onclick, onload, and other properties that could be on any element in the document. And you need to get rid of external css files, you need to get rid of object tags, maybe even image maps. Point is, it gets really complex really fast.

After realizing how many loopholes there could be in pre-processing, I came back to the conclusion that pre-processing isn’t the “right” strategy. It’s a hack, that might work most of the time, but it’s never going to be as sure-fire as doing it right. The right way would be to block the requests at the network level or the browser level. Oh, you want to request a file off the internet? No, you haven’t clicked the magic button that says this is trusted content, forbidden.

Which brings me back to offline mode and that annoying bug that IE ignores the offline mode parameter. Why did they even bother putting that in their spec for the interface to their WebBrowser control in the first place? I’m sure there’s some technical reason they decided to not handle the parameter correctly. They even have a knowledge base article for a very old version of IE (IE5?), explaining the issue, and that there are no known workarounds. Very helpful.

Then I started to look into other replacements for the TWebBrowser control. There was one that initially looked promising, a firefox component that attempted to be a direct replacement for the TWebBrowser. So close, in fact, that you can patch a compiled EXE to put it in without touching the code. But…there were some big buts. But the newest version it comes for is Firefox 1.5, a seven year old version. This code is not maintained, it does not have bug fixes, it’s poorly documented, nevermind the possible security issues. Oh, plus it doesn’t implement any of the “more complex” aspects of the TWebBrowser interface, for example, it can’t create the page from a memory stream rather than from a file on the hard drive, so then I’d have to start littering the hard drive with unnecessary temp files. And even if I changed the code to do that, there’s no guarantees I’d be able to get it to work at all.

Then there’s a Google Chrome Frame. It’s not a direct replacement, and is not an activex control like the TWebBrowser or the Firefox browser component. It only supports HTTP requests (not file) unless you muck around with some settings, probably have to start writing temp files, etc. I haven’t played with it enough to assess whether it would be a feasible option or not.

One thing I did implement was adding some logic that if a message is marked as spam when you hit preview, it switches to plain-text view instead of HTML automatically. At least that solves the problem of how do you preview a message that might be spam, without having to open another non-spam email and switch tabs and then close the window and preview the spam message. That was a step in the right direction.

And then I also figured out that if you set “offline mode” in any internet explorer window, they ALL go offline, including the embedded browser window in PopTrayU. So, if you really don’t want external content to load, all you have to do is set IE offline. Of course, then you cant surf the internet in IE at the same time. But if you’re pretty serious about security, there’s a good chance you don’t use IE as your primary browser anyhow. So that’s one option. A workaround. Less than ideal for some users. But may be totally sufficient for others. Potentially I could pragmatically enter offline mode before rendering the page and then return to online mode after, affecting other IE windows as well, though that would affect other IE windows doing stuff in the background (hope you don’t have a big file download going on in the background!).

So then I get back to debating whether I should just nix the whole experimental “hide images” feature entirely, because it’s just not worth the trouble of making it work, or keep pressing in to try to find a way to make it work that’s going to be a lot of work for a modest return. There are plenty of other features and bugs and refactoring that could easily occupy my time instead. Decisions…

Diving into the Bohemouth

One of the things I’ve been finding most challenging as I develop PopTrayU is how to test and debug issues with emails. There’s no easy way to select an arbitrary email and paste in it’s raw headers to test the email.

Or just in general, situations like this: a user reports some bizarre error message that doesn’t make sense and is poorly worded about how dollar sign underscore isn’t an integer…they got this email from ebay.de that caused the error message. A little searching the code and google reveals the error message is probably caused by trying to convert a string to a number. Well, ok, so have a good suspicion of a couple places that might do this, such as where it’s converting dates around…but I need an email with the bad headers to cause this behavior… no easy way in. This bug has been around for some time (since before I started messing with the code), and it seems, in my opinion, that if it’s going to give an error message it should at least make sense, eg: “this email message is corrupt” and it should try to display the bad data as best it can excepting the corrupted header if it can’t be salvaged.

On the one hand, I could modify that code to try to fix the bug, but without creating a repeatable test-case, I have no way of verifying the fix actually fixes anything and isn’t just an incorrect stab in the dark.

As I think about how could I make the code more testable, I keep coming back to the need to break the code up into modular pieces, rather than a 7000+ line uMain.pas file. If I could break apart the pieces where it’s doing the message processing from the UI and input data source, I could make a second unit test project that fakes the input data and does something different with the output rather than display it on the UI.

So, now that I fixed the internationalization code so all my new labels are translatable (unrelated, but needed to be done), I think some refactoring will be my next sub-project…see if I can break down the mammoth main file into more bite-size chewable testable sections.

Where to Store PopTrayU ini/user files?

One area where PopTray/PopTrayU is currently less than perfect when it comes to Vista/Win7 functionality is that it stores the poptray.ini file (and other files such as rules and whitelist) in the program files folder with the application. In Vista and above, writing to program files when you’re running normal user applications is a big no no. Those kinds of files are supposed to go in a per-user folder under the users/appdata folder to support multi-user environments, and protect users from malicious applications overwriting things they shouldn’t. It doesn’t make my job as a developer easier, but from a security perspective, it definitely helps cut down on certain opportunities for viruses and trojans to be destructive.

But that leaves me with a dilemma. The old design where everything was in the same folder, and the app doesn’t use the registry makes copying the app to a memory stick to use on other machines (say, internet cafe) with locked down machines pretty easy and doesn’t require an installer. Also makes side-by-side use with PopTray easy, since you can just plop poptrayu.exe in the PopTray folder and go.

But, with Vista/Win7, that model where everything goes in the app folder makes a lot of headaches for users who accidentally ended up with a virtual program files folder and can’t figure out why their changes aren’t “sticking” to poptray.ini or the plugins they are trying to install are suddenly “missing” and they have to approve all these UAC prompts to even install a plugin. Not the best of user experiences.

In order to support Vista/Win7 properly, the default storage location for those files needs to be appdata (aka a user-specific folder). I’ve been debating whether it would be best to keep things simple–now poptray.ini HAS to go in user/appdata/roaming/poptrayu/ unless they specify otherwise by using the command line options to specify the “ini folder”, or whether to come up with some more elaborate scheme such as falling back to the app directory if the user/appdata/roaming/poptrayu/ folder doesn’t exist.

Then, if I change the default storage location, there’s another ball of worms, as far as updating the installer to move/copy the ini file to the new storage location, whether it should import the ini file from poptray and/or poptrayu’s app folder if it’s there, etc.

So, please discuss in the comments, what you think the “right” behavior of the app should be. Do you actually use the use-case where having fall-back folders would be handy, or do you just run the installer and expect it’s going to magically put everything in the right place and you don’t care where it is so long as it works?

Creating an Installer

Another thing I’ve been working on lately is a couple of finishing touches to my PopTray fork, so I can distribute it (icons, installer, graphics for the installer, etc).

In particular, my top priority was creating an installer. I thought I might have to start from scratch there, but it turns out Renier was using NSIS for the installer, and the configuration file used to create the installer is included, so while I tend to prefer applications that are a bit more visual than NSIS and have menu options and check-boxes to configure how your installer will work and function, this one was almost all the way there. It was easier to just fix the problems with the existing installer script (like creating images and folders needed by the installer) than to start over from scratch with a different installer.

I see the advantages in certain environments that the install script system used by NSIS is very customizable for specific install procedure needs and would be easy to track changes to the install script in version control systems since everything is stored in one text file until you compile it into a setup.exe, however, it is also a bit fragile by the same nature.

It would be quite easy to overlook or make a typo in the name of a file that needs to be deleted when uninstalling and have it not uninstall completely because of that. That, and I’d kind of rather not have to pull out a manual every time I want to know what a particular line of the script does, or to figure out whether the line is actually necessary. And then if you comment out an image here from the script, then the macro it creates needs to be commented out there as well or it just breaks and won’t compile. It makes a nice small installer without a lot of overhead, but less flexibility would also mean less opportunity to introduce a mistake into the installer script. I don’t want to worry about registry keys needed to uninstall the app being in my script, I want the compiler to take care of that part, I want to be spending my time specifying that the following files go in program files and the following files go in application data 😉

PoptrayU: Updated Screenshot

Here’s an updated screenshot with my progress over the past few days:

You’ll notice now that the two emails with subjects in Hebrew and Greek (probably horribly translated, I just used Google Translate) now appear correctly in the main window.

Getting these messages to appear correctly was actually quite tricky to figure out. The key, in the end, was that I had to make a copy of the subject field before the Indy routine ProcessHeaders mangled it into Ansi characters, and then call my DecodeHeader algorithm on the un-mangled header.

The ironic thing is, if I upgraded to the latest version of Delphi, and the latest version of Indy (as in Indy10 rather than as in Indy9.0.53), the whole DecodeHeader and handling international characters would be a non-issue. But then I’d have a whole different can of worms–I’d have to fix all the non-backward compatible changes between Indy 9 and 10, I’d have to figure out how to convert ActionBand Popups into the new Delphi 2010 equivalent. In the end, what it would take to port the app to a newer version of Delphi is probably more difficult than coercing the old version of Delphi and Indy into doing what I want it to do. Sure, in the end it might end up being more robust that way, but I might also break and mangle lots of existing features.

At this point, what you see in the screenshot, I am only processing the Subject field with the new technique that works for any codepage (vs: only the current one or UTF-8), so I need to extrapolate my strategy to the other header fields that might be encoded. I also need to find an equivalent strategy to do the same thing on the Preview window. In the preview window it re-downloads the email through a different Indy code-path, and I haven’t found the right place in that code-path where I can capture the un-mangled header yet. So there’s still work to do but I’m on the right path.

To convert the random code-page to Unicode, I am using the windows library function WideCharToMultiByte, which converts a string to a “wide” (Unicode) string based on a specified code page number. Getting the code page number was also a little bit of a challenge. The library with that function doesn’t have a GetCodePageNumber function to convert the code page *name* to the windows code page *number*. There is a DLL that comes with windows that has (almost) that function, but figuring out how to call it is kind of tricky, and rumors on the internet say it might be buggy in certain cases. So, I’m using the straightforward but ugly strategy: convert the table on MSDN of allowed code-page names/IDs to a data structure and look it up manually. That list isn’t the full IANA supported list of encodings (aliases/alternate names), but the cases it doesn’t handle are likely to be rare, and could be added in the future if future research doesn’t find a better strategy.

I started out storing the list in a record (ie: Delphi equivalent of a struct) with a dead simple sequential search algorithm, per a tip on StackOverflow, but I wasn’t happy with the performance on lookup, because there’s 140 different encodings in my list of encodings so far, and this method is going to be called in a loop for each header of each email unless it’s definitely not encoded. So I did some research and found TStringList which can be used like a sorted map with better string search performance, and THashedStringList, which is basically a hash-map data structure, so even better string search performance–up to the level you need for, get this, INI FILES! Then I had to do more research to figure out what the Delphi equivalent of a static initializer is to use the Hashed String List…But now that I have it working, it does seem noticeably faster even on a small number of emails, but the internet could just be less congested today, it’s hard to say.