Creating an Installer

Another thing I’ve been working on lately is a couple of finishing touches to my PopTray fork, so I can distribute it (icons, installer, graphics for the installer, etc).

In particular, my top priority was creating an installer. I thought I might have to start from scratch there, but it turns out Renier was using NSIS for the installer, and the configuration file used to create the installer is included, so while I tend to prefer applications that are a bit more visual than NSIS and have menu options and check-boxes to configure how your installer will work and function, this one was almost all the way there. It was easier to just fix the problems with the existing installer script (like creating images and folders needed by the installer) than to start over from scratch with a different installer.

I see the advantages in certain environments that the install script system used by NSIS is very customizable for specific install procedure needs and would be easy to track changes to the install script in version control systems since everything is stored in one text file until you compile it into a setup.exe, however, it is also a bit fragile by the same nature.

It would be quite easy to overlook or make a typo in the name of a file that needs to be deleted when uninstalling and have it not uninstall completely because of that. That, and I’d kind of rather not have to pull out a manual every time I want to know what a particular line of the script does, or to figure out whether the line is actually necessary. And then if you comment out an image here from the script, then the macro it creates needs to be commented out there as well or it just breaks and won’t compile. It makes a nice small installer without a lot of overhead, but less flexibility would also mean less opportunity to introduce a mistake into the installer script. I don’t want to worry about registry keys needed to uninstall the app being in my script, I want the compiler to take care of that part, I want to be spending my time specifying that the following files go in program files and the following files go in application data 😉

PoptrayU: Updated Screenshot

Here’s an updated screenshot with my progress over the past few days:

You’ll notice now that the two emails with subjects in Hebrew and Greek (probably horribly translated, I just used Google Translate) now appear correctly in the main window.

Getting these messages to appear correctly was actually quite tricky to figure out. The key, in the end, was that I had to make a copy of the subject field before the Indy routine ProcessHeaders mangled it into Ansi characters, and then call my DecodeHeader algorithm on the un-mangled header.

The ironic thing is, if I upgraded to the latest version of Delphi, and the latest version of Indy (as in Indy10 rather than as in Indy9.0.53), the whole DecodeHeader and handling international characters would be a non-issue. But then I’d have a whole different can of worms–I’d have to fix all the non-backward compatible changes between Indy 9 and 10, I’d have to figure out how to convert ActionBand Popups into the new Delphi 2010 equivalent. In the end, what it would take to port the app to a newer version of Delphi is probably more difficult than coercing the old version of Delphi and Indy into doing what I want it to do. Sure, in the end it might end up being more robust that way, but I might also break and mangle lots of existing features.

At this point, what you see in the screenshot, I am only processing the Subject field with the new technique that works for any codepage (vs: only the current one or UTF-8), so I need to extrapolate my strategy to the other header fields that might be encoded. I also need to find an equivalent strategy to do the same thing on the Preview window. In the preview window it re-downloads the email through a different Indy code-path, and I haven’t found the right place in that code-path where I can capture the un-mangled header yet. So there’s still work to do but I’m on the right path.

To convert the random code-page to Unicode, I am using the windows library function WideCharToMultiByte, which converts a string to a “wide” (Unicode) string based on a specified code page number. Getting the code page number was also a little bit of a challenge. The library with that function doesn’t have a GetCodePageNumber function to convert the code page *name* to the windows code page *number*. There is a DLL that comes with windows that has (almost) that function, but figuring out how to call it is kind of tricky, and rumors on the internet say it might be buggy in certain cases. So, I’m using the straightforward but ugly strategy: convert the table on MSDN of allowed code-page names/IDs to a data structure and look it up manually. That list isn’t the full IANA supported list of encodings (aliases/alternate names), but the cases it doesn’t handle are likely to be rare, and could be added in the future if future research doesn’t find a better strategy.

I started out storing the list in a record (ie: Delphi equivalent of a struct) with a dead simple sequential search algorithm, per a tip on StackOverflow, but I wasn’t happy with the performance on lookup, because there’s 140 different encodings in my list of encodings so far, and this method is going to be called in a loop for each header of each email unless it’s definitely not encoded. So I did some research and found TStringList which can be used like a sorted map with better string search performance, and THashedStringList, which is basically a hash-map data structure, so even better string search performance–up to the level you need for, get this, INI FILES! Then I had to do more research to figure out what the Delphi equivalent of a static initializer is to use the Hashed String List…But now that I have it working, it does seem noticeably faster even on a small number of emails, but the internet could just be less congested today, it’s hard to say.

Before And After Screenshots

I thought I’d share some before and after screenshots of the work I’ve done so far.

Here’s the main screen of the application before:

And here is the after shot:

You’ll notice two of the emails look like gibberish both in the before and after shots. Those were test emails I created with subjects in a single foreign language (one is in Hebrew, and one in Greek), so Outlook decided to encode them with the encodings windows-1255 and iso-8859-7 respectively. This is a case I’d like to handle better in the future. But even without that, still a much more usable version.

And for fun, here’s another screen I’ve been working on. It is supposed to look like this (in the old version of PopTray):

But if you’re using Vista or above, and using the Aero theme, because of some screen refresh bugs (not specific to the app) usually it looks more like this:

So my changes to this screen include moving the alignment of the buttons to “cling” to the right side of the window, and adding code in the window resize to stretch the textboxes when you resize the window. Textboxes that are too small that don’t resize when you make the window bigger is a pet peeve of mine. So I thought I’d fix it.

And while I was at it, I looked into workarounds for the refresh issues, and found that adding a call to refresh the inner panel (aka “frame” in delphi) with the missing labels and buttons on create and after the window is resized reduces the problem by about 95%, as in, it displays correctly on load and it’s only a problem now if you drag the window off the screen and back, but if you resize the window it redraws it properly, so it’s much easier to work around. If I ported the app from Delphi 7 to a newer version of Delphi this problem would probably go away entirely, so it’s not really worth the investment to try to workaround that last 5% issue.

New Bug to Fix

I had an email in my inbox today from Shutterfly with a subject in Base-64 encoding. Only, the subject was so long in Base-64 that it didn’t all fit on a single line in the email headers. Which means my base-64 subject decoder needs to be a little more robust.

The raw-header looks like this:

Subject: =?UTF-8?B?WW91ciBUaGFuayBZb3UgZ2lmdCBlbmRzIFdlZC4gfCBFbmpveSA=?=
=?UTF-8?B?JDIwIGp1c3QgZm9yIHlvdcKg?=

Notice anything odd about the format of this header? Each line has it’s own encoding tags, instead of one encoding tag spanning both lines, so it’s a little different than what I’ve seen on long subjects encoded with quoted-printable. Converted from Base 64 to English, the subject should parse into:

Your Thank You gift ends Wed. | Enjoy
$20 just for you

No weird special characters or anything that really necessitates base-64, but none the less to be a robust email parser, this kind of case should probably be handled.

In Delphi, I get a single string with the entire header except the word “Subject: “, but including the line break returned from Indy. This means I can’t just say from after the B? to the very end minus two characters, I need to actually tokenize the ending question-marks and/or whitespace, and encoding begin strings, perhaps in a loop, so I can pull out just the base 64 encoded part:

WW91ciBUaGFuayBZb3UgZ2lmdCBlbmRzIFdlZC4gfCBFbmpveSA=
JDIwIGp1c3QgZm9yIHlvdcKg

And then pass that lovely string into the base64 decoder. And then instead of a subject of

=?UTF-8?B?WW91ciBUaGFuayBZb3UgZ2lmdCBlbmRzIFdlZC4gfCBFbmpveSA=?= =?UTF-8?B?JDIwIGp1c3QgZm9yIHlvdcKg?=

you could have a subject of

Your Thank You gift ends Wed. | Enjoy $20 just for you

Bugs Fixed!

Two bugs fixed! The first was trivial, adding one line of code for a case I had overlooked initially. The second was more subtle. I eventually tracked down the cause to a less common variant in the MIME headers.

Here’s a typical MIME header:

Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

And here’s the header I was seeing in the error producing email:

Content-Disposition: inline
Content-Transfer-Encoding: 8bit
Content-Type: text/html

Notice anything majorly different? Take a look at that content-type. In the second-example, it didn’t specify the charset, so there’s no semicolon after the content type string. And that meant that some existing code that parsed out the content-type would fail if the charset was missing. Position of semi-colon returning zero (not found) when you want it to return end of string if not found…slight difference, now it gets the full content-type in the latter case instead of a blank string and displays correctly.

Adding an HTML Preview Window

I decided to take a break from fixing up the subject line decoding algorithms (I think it could be more efficient and limitations that could be potentially eliminated) and see how difficult it would be to add HTML message preview to the application.

I started out with some online research about what sort of component could display HTML. The first thing I came across was a third party component that was eventually released into the public domain. It looked promising and easy to use, but I had a couple concerns. First, was overcoming an issue with installing components in the first place, probably a compatibility issue between Delphi 7 and Windows 7. Eventually I realized the solution was to delete the file it was giving an error about and then ignore the error message. Not obvious but it worked. But even at the point where I got to where I could install components, I had concerns about whether the component would meet my needs for rendering HTML emails. It looked in their demo app as if it was rendering all the HTML from scratch, in a very HTML 3.0 kind of way, with lots of emphasis on frames and displaying frame-sets…stuff that’s kind of obsolete these days. But being able to handle frames isn’t the problem. My concern was that, because the component appeared to be dated, it might not handle certain modern widely used HTML features such as stylesheets.

So then I decided to look into an alternative approach, to see whether there was a way to embed the native OS web-browser into the application to render the HTML. It did not take much research to determine that this was quite possible, and with that using only stock components included with Delphi.

Of course, it’s always easier said than done, right? First it turned out when I installed Delphi and skipped a bunch of stuff I didn’t anticipate needing (database and web stuff mostly) to speed up the install, I had not installed a piece I needed to use the TWebBrowser component. And then it turns out that even after you install it, you still have to select “Import ActiveX Control” from the Component menu and select the component to get it to appear on the Components toolbar.

And after I got that fixed, when I added it to the form it added an extra “uses” import to the project for some OleControls thing that I didn’t need, which had an extra feature of causing the entire application to crash immediately on debug. I commented out all the super-questionable new code I’d written, but it still crashed. So I commented out more changes in cycles until I got to the point of diffing the entire file and commenting out anything that wasn’t in the original when I discovered the suspect import (I’m guessing that “import” is probably not the proper Delphi term for it, but coming from a java background that’s how I mentally understand it). At that point it stopped crashing so I started un-commenting the sections of code one at a time to see where it broke down, and I was able to uncomment everything except my suspect import without issue, so I was able to move on.

After that, I had to learn how to add a tab to the tab control, which was pretty straightforward. Though I still have an outstanding issue to look into how translation of the tab names in handled and regression test translation of the tabs to make sure my new tab doesn’t cause the tabs to have the wrong names if the app is not being displayed in English. It would be easy enough to ignore that case for my needs since I speak English, but since I do hope to release my code improvements pubically, I need to think like a developer, and consider other users with use cases other than my own personal one, and not break features that would be important to the users of the application.

Displaying raw HTML in the component was not too difficult, though it relies on a snippet of code from the internet on how to “properly” load raw HTML directly into the browser. It’s a little complicated, but it worked right out of the box. My understanding is that the code chunk I used was based on directions in Microsoft’s knowledge-base about the proper way to display HTML that’s not in a file, starting with showing a blank page and then some other stuff including checking the browser’s ready state, and creating an “IPersistStreamInit” stream to stream the HTML to the page. Having worked on writing Internet Explorer Plugins before, you have to kind of expect there to be some ugly boilerplate code like that to connect to Internet Explorer with long interface names starting with I’s. I found some other similar code that was a bit more object-oriented that would become a wrapper class for the TWebBrowser to give it pretty functions like LoadHtmlFromString(…) instead of having a procedure do it yourself, but for 15 lines of code, using a pre-made wrapper class with a bunch of other pretty functions I don’t need and am unlikely to need in the future might be overkill.

Next roadblock? MIME. Lot of technical information about the Email spec that you have to worry about. You don’t just want to display the entire email in the browser window, because often the message is included in both plaintext and HTML formats, so you only want to display the HTML format part, so you don’t have the whole email twice. Luckily, since the application has options for saving attachments already, there was a place I could hook into to grab the MIME section with the HTML attachment. Unfortunately, it’s also not quite that simple and I still have some corner cases I need to fix.

With the test bank of emails of “whatever is in my inbox today” that I tested with, about 90% of them display correctly with the current code. It’s those corner cases though. At least, email is easy to get more of to test with ;-). There’s this one message that isn’t reading the MIME type correctly, but I’m slightly suspicious that the application isn’t reading the MIME type of that email correctly for other reasons unrelated to my new code. But I’ll still have to look into why it’s picking up the second MIME section as “text” instead of “html” and fix it to make the preview robust and work for as many emails as possible. The other message I’m having trouble with appears to not have a plaintext section, and everything is HTML, and I think it’s an edge case that’s falling through and not hitting the code that copies the HTML chunk into the preview window.

But all in all, I’m very excited that after only a couple hours of mucking around I have a preview window that can display HTML emails for about 90% of what’s in my inbox! That’s leaps and bounds of progress and makes the app even more powerful than it was before and will be a very useful additional feature for the application.

Fixing PopTray: My Christmas Project

Over my Christmas “vacation” I decided I needed a project. Something to do during my son’s naps for my own enjoyment while visiting the in-laws. After a short amount of mulling it over, I decided to attempt to tackle a project I’d been idly thinking about for quite some time.

And when I say quite some time, I mean it. I actually started looking into solving this problem three years ago, and had come up with a proposal back then of how to fix it and posted that to the application’s forum site. But…the original developer of the application stopped updating the app in 2006, rather abruptly.

It was a little bit of a mystery why he stopped developing it, at the time, for a while there were new editions every few months, keeping up with the latest version of the programming environment, adding and improving features with a nice wishlist of changes for future releases, and then all of a sudden, it just stopped in it’s tracks. I never really knew why he stopped–on the forums I saw something about the application “met his needs sufficiently” so he decided to stop updating it. But that was three operating systems ago! And that was before e-mail was frequently sent with subjects encoded in UTF-8, that look almost illegible when displayed as plain-text.

I kind of understand why open-source development can just spontaneously stop, especially when there’s really only one guy doing everything. Maybe it starts out as side-project that just gets out of hand, and starts sapping your free time, until you no longer enjoy working on it. Maybe you get a new job, or your life circumstances change, and you no longer can devote the time. Maybe your computer breaks down or gets stolen, and you no longer have access to the development tools you need. Things happen. Its life.

But that doesn’t mean that one little nagging un-fixed bug in the application doesn’t keep nagging at you. Back in 2008, I took a look at the source-code to see whether I could find the cause of the bug. I had a pretty good hunch where you needed to fix the problem, and what needed to be done. But I didn’t take it a step farther and implement it at that time because I didn’t know the programming language, I didn’t have the programming suite I needed, and there were several steps in my outlined solution that relied on knowledge I didn’t know. There was a little bit of hand-waving as in, I’m not sure how you do this in Delphi, but in programming languages I’m familiar with it’s not too hard, so this should be doable, I think.

So at that time I proposed my solution outline on the forum, hoping someone else who might not want to spend as much time figuring out why it’s a bug, or even the original developer, could take my outline of a solution and implement it.

Three years go by. During that time I’d considered changing mail notifiers a couple times. I even downloaded just about every other free one I could find. But none of them were as good as the one I’d been using for the last ten years. Yes, some of them handled international characters much better, but I was quite content with the user-interface and functionality of the application I was using, and didn’t really want to get used to a different clunkier interface.

If in all this time, nobody else has taken the time to solve the problem and fix it, and nobody else has come out with a better alternative application, maybe it’s just time to bite the bullet, overcome the hurdles, and fix it myself.