PopTrayU Bugtracker

Originally, for keeping track of issues with PopTrayU I attempted to use Sourceforge’s built in bug tracker. But, as I recall, I was less than enthralled with it for reasons I’ve long since forgotten.

putbugtracker2So then I started looking into other options, PHP based options I could run from the webhosting area on Sourceforge, and ended up selecting Mantis. I think I looked into bugzilla as well and I don’t remember what else. For the most part Mantis worked quite well, and had a slick-looking interface, and I loved the dashboard view where you could sort all the bugs by their status and it color-coded them in a way that just “made sense”.

But Mantis and Sourceforge just don’t play well together. Every time you submit a change to a bug, the next page you try to load returns a 503 “Guru Meditation” error due to the Varnish Cache Server installed by Sourceforge. Apparently it’s some sort of timeout error, and I’d spent a lot of time looking into fixing the issue, possibly more than once, without success.

So what ends up happening in reality? It’s too much trouble to update the bug-tracker in everyday use So I just keep track of “active” bugs in a spiral notebook sitting on my desk with a pencil and only using the bug tracker only for keeping track of “things I might implement someday”.

So I decided to give this version tracking issue another look. Wanting to stay away from Soruceforge’s cache server issues, I started looking at the options for bug tracking used by some other open source projects I follow.

Google Code is deprecated and on the verge of being shut down and not open to new projects, so although I’ve liked them in the past that was out.

Microsoft’s CodePlex I love how they prominently display options to vote on your favorite bugs. Super-useful when you’re thinking “which bug should I tackle next”.codeplexAnd it has a nice taxonomy for marking up the bugs–separate drop-downs for component, status, type, impact, etc for well-classified bugs. But it’s kind of boring and corporate, lacking color-coding, and the captcha nearly resulted in loss of data for both of the two first test entries I entered.

And then there’s the ever-popular GitHub. It’s bug tracker is simple–at times almost too simple. Component can’t be a separate field from the bug status–it’s all just a mashup of tags. I’d rather see the list of components in my project separate from the resolutions. But overall the bug tracker has some nice CSS and is quick and easy to add new bugs and update their status.

github-bugsIn the end I decided ease of entering and updating bugs was more essential than taxonomies and voting. If it’s not practically effortless to track the bugs, there’s a good chance I won’t use the bugtracker to keep track of features I’m considering developing.

So here’s the new bug tracker: https://github.com/jojobear99/PopTrayU/issues

Add/Remove Gmail Labels using Indy 10 in Delphi

According to support threads, Indy’s TIdImap4 supports fetching Gmail labels but doesn’t have support built in for adding and removing labels. I’m still working on figuring out how to fetch and read the fetched labels, but by using the SendCmd function to manually send the IMAP commands I got adding and removing labels working. So I thought I’d share that code:

function AddGmailLabelToMsgs(const uidList: TStrings; labelname : string) : boolean;
    if HasCapa('X-GM-EXT-1') and (uidList.Count > 0) and (labelname <> '') then begin
      IMAP.SendCmd(ImapCmdNum(),'UID STORE '+uidList.CommaText+' +X-GM-LABELS ("'+ labelname + '")',['OK','BAD','NO'], true);
      Result := IMAP.LastCmdResult.Code = 'OK';
    end else
      Result := false;
    on E : Exception do
       //Dialogs.ShowMessage('Exception class name = '+E.ClassName);
       //Dialogs.ShowMessage('Exception message = '+E.Message);
       Result := false;

At the heart of it, I am using Indy’s SendCmd function to manually send the command. It was a little tricky figuring out how to use this function since documentation and examples are a little scarce. The first parameter is a sequence number so that the response can be matched up with the command. Second is the remainder of the actual command. Third parameter is what responses to look for to know this command is “complete”, and the fourth is whether to look for a single line response or multi-line response.

I also have some commented out exception handling that was helpful to debugging when my SendCmd wasn’t formatted correctly, but is neither needed nor useful for production code.

Remove label is basically the same thing but with a minus sign:

function RemoveGmailLabelFromMsgs(const uidList: TStrings; labelname : string): boolean;
    if HasCapa('X-GM-EXT-1') and (uidList.Count >0) and (labelname <> '')  then begin
      IMAP.SendCmd(ImapCmdNum(),'UID STORE '+uidList.CommaText+' -X-GM-LABELS ("'+ labelname + '")',['OK','BAD','NO'], true);
      Result := IMAP.LastCmdResult.Code = 'OK';
    end else
      Result := false;
    Result := false;

And here’s the helper functions, minus the global/class/whatever variables:

ImapCmdNum() generates a sequence number for the IMAP transaction, essentially a replacement for NewCmdCounter in IdImap4, since it’s a protected property and client code can’t use it. At first I tried using “A1”, “A2”, etc. but it turned out that using a C prefix, “C1″, C2”, etc. like Indy was important for using Indy to process the response when I got into debugging why things weren’t working right at first…but now I’ve completely forgot exactly where it goes wrong if you don’t use a C.

function ImapCmdNum(): string;
  Result := 'C'+IntToStr(cmdNum);

And HasCapa() is a function I added to make sure that the Gmail capabilities are actually available before trying to set/unset labels…Might not be totally necessary, but it seemed like a good idea. Under the hood, I just used Indy’s built in Capability function that calls CAPABILITY on the server, with a little caching so I wouldn’t need to do this more than once per session.

function TProtocolIMAP4.HasCapa(capability: string) : boolean;
  if (capabilities.count = 0) then begin
  Result := (capabilities.IndexOf(capability)<>-1);

How to Use Indy 10 Imap Intercept function for Logging

It took me a while to track down enough information to figure out how to use Imap.Intercept to log raw IMAP client/server communication to troubleshoot an SSL account issue, so thought I’d share a basic code snippet on how to have IMAP log to disk the communications.

idLogFile1 : TidLogFile;
IMAP := TIdIMAP4.Create(nil);

idLogFile1 := TidLogFile.Create(nil);
idLogFile1.Filename := 'C:\temp\imaplog.txt';
idLogFile1.active := True;

IMAP.Intercept := idLogFile1;


Installing Indy9 from Source Code

I’m finally working on getting Delphi 7 set up on my Win7 x64 machine. I think it’s been over a year since I last did this on my Vista machine or my Win7 x32 netbook, so my mind totally went blank trying to remember how to install the lastest version of Indy 9.

I already had pulled the code from Indy’s SVN server and been able to transfer that from the old machine, so I was one step ahead there.

I pulled up the “ManualInstall.txt” document in the Indy source directory and tried to follow the directions, only to find it didn’t work. At first it looked like it was a permissions issue, I needed to be running a batch file called fulld7.bat from a command line running on elevated privileges to get the script to run without errors, however, even after sorting all that out, when you get to the step of Installing the *.bpl, even after you’ve correctly remembered which one is the right menu option install such, it just doesn’t work. Complains about a missing file (that just doesn’t seem to exist in the provided repository or compiled files directory).

Well, after much fiddling and trying to figure it out on my own, since the internet had no solutions, it turns out, those directions probably are obsolete, and doing it the way that actually *worked* was much easier…just not obvious if you haven’t been working with Delphi recently.

In the Indy 9 directory from the SVN, there’s a .DPK package file for each delphi version. So, since I’m using Delphi 7, double click Indy70.DPK, it opens in Delphi with a package window visible displaying a list of all the files. At this point you’re two clicks away from a working Indy install. Click “Compile” and then “Install” on the package window, and dismiss the dialogs that confirm those steps work, and suddenly Indy is on the component toolbar without even restarting Delphi.

So there you have it. I’m thinking documenting the steps on installing this that work will be helpful, because by the next time I get around to trying to install Indy on a new machine, I almost certainly will have forgotten how you install it, since you do it once and never touch it again, and odds are good that it will need re-installing after a hiatus where I haven’t touched Delphi in a while period and can’t remember off the top of my head the difference  between installing a component and installing a package, since that’s very delphi-specific terminology.

New Project: Android Apps?

Another project idea that crossed my mind was getting back into doing Android programming. Android continues to be gaining in market share, and I happen to have an Android phone I could test on. Writing Android apps is a potentially marketable skill, and one that would potentially lend itself to part-time contract work, should I wish to look for that sort of work. A project in this arena would both refresh my memory, expand my knowledge, and give me a portfolio to show for my skills. “Oh, yes, I could write an app for you. You want to know I’m capable? Well, have a look at some of the stuff I’ve done before”

I have been pondering what sort of app I’d want to write. A game? Maybe a game timer? The last time I visited my sister we were playing Taboo, and the sand timer wasn’t cutting it…the timer-watcher would forget to keep watching. So my dad ended up pulling out his iphone and setting a stopwatch alarm on there to time the remaining rounds. Would a dedicated android app potentially be more useful or work better? There seem to be chess timers out there, various other sorts of game timers, and the like. You can even download an app that IS Taboo in non-box form. Each one was riddled with its own sorts of issues, that one looks to specialized, not easy to use for just replacing the timer from this or that game, not pretty to look at, not intuitive, etc. Things have been written, but nothing that would be as simple as I’d like, while adding more useful functionality than just using the stopwatch. Like I might want the game-time for popular games pre-populated so I don’t have to remember whether boggle uses a 3 minute timer or a 1 minute timer. I might want optional “loud ticking” or some other auditory warning when time is running out. I want to just click a button or two when I launch the app so I can get going quick. 3 minute timer, go. Easy to pause. Without a lot of fuss out of the box, making it take less time to set up than a general stopwatch program. If you could make the user-interface dead-simple and easy to use, AND aesthetically pleasing, there might be an app worth the trouble of making.

Trying to solve an actual problem on my phone revealed another potential project. My phone’s been having frequent sudden drops in amount of battery available. It’s also had some problems with wake-lock freezing the phone, and some dead-zones due to some geographical features of where I’m trying to use my phone. So I’ve been trying to passively monitor why my phone’s battery is going from lots of juice to “charge me RIGHT now”. Android has a built in battery monitor graph, but you can’t zoom in on the graph, which seems to be a big problem because I see this big drop that happened four hours ago and I can’t zoom in to see what time that was…was that when someone called or texted? Or around the time of day I pulled the battery? Was the screen on? Was I using the phone? Did I receive a voice call? And for that matter, why is my phone being awake so much when I’m not using it? Is there something eating my battery that needs to be uninstalled Some of these things you can tell from the built in battery monitor, some you can’t directly. I looked at 3rd party apps, and there was one that had a semi-decent historical graph which showed voice calls in addition to what android shows by default, but I didn’t like some of the other things it came with, like doing it’s own logging (wasting battery) and monitoring other apps for hogging the battery (again, wasting more battery). So I keep finding myself wanting better battery use graph that allows zooming. When you start showing 8 hours on a 2″ wide screen, you start to not see a lot of detail. I’d want something that just looks at existing system logs if they’re exposed (they must be somehow because one of the apps I downloaded appears to be replicating the battery use module slightly better) and gives clues as to what was going on then, so I can trace it to whether that unexpected drop was not so unexpected after all, or whether I need to look into whether something is using too much battery that shouldn’t be for unnecessary updates, mis-configured etc.

New Project: Porting PopTrayU to FreePascal?

Poking around on the internet for one reason or another (maybe I was browsing posts about Delphi on the Programmers Stack Overflow/Exchange site?), I came across something interesting. Lazarus, the open-source IDE for Free Pascal. 

Lazarus is an open source equivalent to Delphi. Well, not exactly equivalent, but pretty close.. It comes with (not terribly well documented at the moment) tools to port Delphi code to Free Pascal, and comes with the Lazarus Component Library (LCL) which is very similar to Delphi’s Visual Component Library (VCL).

The idea of porting PopTrayU to an open source development platform instead of an expensive closed source one (namely: Delphi XE) is very much in the spirit of open-source development. It was always one of the dreams of Renier (the original PopTray developer) to port PopTray to a free platform, and he indicated he might be interested in continuing to develop the app if it could be ported. And it would save me the expense of upgrading to Delphi XE to fix some various bugs.

I’d previously looked into porting PopTray to the free version of Turbo Delphi 2006, but my feasibility analysis results were that it was not feasible. That version of Delphi, aside from being discontinued/no longer supported or “officially” available, was that it left out a key feature (namely, 3rd party component support).

Free Pascal and Lazarus, from what I’ve said so far looks more promising as far as being feasible. Indy components has been ported to Free Pascal, which is a key stepping stone, though investing whether I’d have to port to Indy 10 first or not remains to be determined. So far everything else I’ve been reading looks like it would “probably” be possible without significant loss of critical features, but it wouldn’t be a plug and play transition, you have to rename a bunch of things and change the imports, and some of the UI might get mangled porting it so you might have to redraw parts of the UI. But so far haven’t come across any definite showstoppers.

At the moment, the biggest risk for showstopping problems would be if there is a missing library eg “things that are very windows specific” may be missing. But nothing specific yet. There would definitely be some hurdles to fix the unicode issues, but that’s to be expected and certainly fixable. Even though Free Pascal is multi-platform, the probram would still only run on Windows, because of Indy and it’s limited compatibility with other platforms.

So that’s on my todo list. And in between downloading Lazarus and now, they’ve already gone from version 0.8 to 1.0 of the IDE. Actively developed and being improved regularly is a positive sign to look for.

New Project?

I’ve been thinking for a while about finding a secondary project to get involved in. While I could probably find plenty of things to do with PopTrayU, it would be more practical, career-wise to also keep myself fresh with a programming in a language that’s a bit more in demand for paying jobs, so I’m fresh on the draw when I find something aweseome I want to interview for.

So I started thinking about what sort of project to do. At first nothing was coming to mind, and browsing open source project sites wasn’t particularly inspiring. But I gave the idea of a project some time to simmer. And then I ended up coming up more than enough ideas which I will write more about in separate posts :)

HTML Email Preview

One of the complaints I’ve heard about the feature of having html preview of emails is that spam-bots often track whether emails are read by embedding a unique image, and tracking if the image gets downloaded. It’s a legitimate concern, one I was well aware of when I added the HTML preview option. But it came down to being a value vs. risk proposition.

The HTML preview window in PopTrayU
The HTML preview window in PopTrayU

For version 1.0 of the feature, that seemed like a perfectly reasonable risk to accept. When I check my email from my phone, it displays in HTML and doesn’t have an option to disable loading images either. And, if like me, you run your email through a really good spam filter (like Gmail), you aren’t getting a lot of spam in your inbox in the first place. But that’s not necessarily the case for people using PopTrayU to check their ISP email account, which is a very common use case.

But that doesn’t mean it shouldn’t be somehow addressed in subsequent releases.

Initially, when I’d looked into HTML components, my first choice was a native Delphi HTML rendering component. But upon investigation, the component was too old and did not support CSS at all, which was a deal-breaker since font tags are completely deprecated these days. I mean, I could have added CSS support myself, but that is an extremely complex feature to implement correctly, with many corner cases. You’re pretty much writing a web-browser minus the network connections part. Much bigger scope than I was willing to take on.

So my second strategy was to use the TWebBroswer control to render the HTML, which is essentially a wrapper for an embedded internet explorer window. You get the HTML rendering almost for free with that technique. But…Internet explorer has this known bug that the “offline mode” flag in the component is completely ignored, and it decides whether you are online by a global registry setting or whether an internet connection is present. Less than ideal for this purpose. Any external content is really undesirable for spam email previewing.

So then I looked into maybe I could just pre-process the HTML and strip out the images. Removing <img> tags wasn’t too difficult. You can either do that with a regular expression or a for loop iterating through the characters in the message. Well, it turns out to be a little more complex than that. Images don’t just come from <img> tags anymore. Images are embedded in CSS that’s slapped in the body section of the message using url(…) syntax. Images are embedded through inline CSS. Almost any CSS component could have an image as part of a background or bullet image or border or any number of other places. And then you need to strip out any javascript, because that might be loading images or external content, so there go onclick, onload, and other properties that could be on any element in the document. And you need to get rid of external css files, you need to get rid of object tags, maybe even image maps. Point is, it gets really complex really fast.

After realizing how many loopholes there could be in pre-processing, I came back to the conclusion that pre-processing isn’t the “right” strategy. It’s a hack, that might work most of the time, but it’s never going to be as sure-fire as doing it right. The right way would be to block the requests at the network level or the browser level. Oh, you want to request a file off the internet? No, you haven’t clicked the magic button that says this is trusted content, forbidden.

Which brings me back to offline mode and that annoying bug that IE ignores the offline mode parameter. Why did they even bother putting that in their spec for the interface to their WebBrowser control in the first place? I’m sure there’s some technical reason they decided to not handle the parameter correctly. They even have a knowledge base article for a very old version of IE (IE5?), explaining the issue, and that there are no known workarounds. Very helpful.

Then I started to look into other replacements for the TWebBrowser control. There was one that initially looked promising, a firefox component that attempted to be a direct replacement for the TWebBrowser. So close, in fact, that you can patch a compiled EXE to put it in without touching the code. But…there were some big buts. But the newest version it comes for is Firefox 1.5, a seven year old version. This code is not maintained, it does not have bug fixes, it’s poorly documented, nevermind the possible security issues. Oh, plus it doesn’t implement any of the “more complex” aspects of the TWebBrowser interface, for example, it can’t create the page from a memory stream rather than from a file on the hard drive, so then I’d have to start littering the hard drive with unnecessary temp files. And even if I changed the code to do that, there’s no guarantees I’d be able to get it to work at all.

Then there’s a Google Chrome Frame. It’s not a direct replacement, and is not an activex control like the TWebBrowser or the Firefox browser component. It only supports HTTP requests (not file) unless you muck around with some settings, probably have to start writing temp files, etc. I haven’t played with it enough to assess whether it would be a feasible option or not.

One thing I did implement was adding some logic that if a message is marked as spam when you hit preview, it switches to plain-text view instead of HTML automatically. At least that solves the problem of how do you preview a message that might be spam, without having to open another non-spam email and switch tabs and then close the window and preview the spam message. That was a step in the right direction.

And then I also figured out that if you set “offline mode” in any internet explorer window, they ALL go offline, including the embedded browser window in PopTrayU. So, if you really don’t want external content to load, all you have to do is set IE offline. Of course, then you cant surf the internet in IE at the same time. But if you’re pretty serious about security, there’s a good chance you don’t use IE as your primary browser anyhow. So that’s one option. A workaround. Less than ideal for some users. But may be totally sufficient for others. Potentially I could pragmatically enter offline mode before rendering the page and then return to online mode after, affecting other IE windows as well, though that would affect other IE windows doing stuff in the background (hope you don’t have a big file download going on in the background!).

So then I get back to debating whether I should just nix the whole experimental “hide images” feature entirely, because it’s just not worth the trouble of making it work, or keep pressing in to try to find a way to make it work that’s going to be a lot of work for a modest return. There are plenty of other features and bugs and refactoring that could easily occupy my time instead. Decisions…

Writing a string to a stream in Delphi

There are two tricks to this. Most of this comes from what Embarcadero suggests on how to write a string to a stream. Another website suggested multiplying the length times the number of bytes to make sure the code works on both unicode and non-unicode builds, something one would be wise to consider when developing legacy code on Delphi 7, that may someday get ported to a newer version. One less thing to worry about later.

procedure WriteStringToStream(stream: TStream; const appendText: string);
  stream.WriteBuffer(Pointer(appendText)^, Length(appendText)*SizeOf(Char));

Show Modal in Delphi

I was looking today at whether I could fix that annoying bug where a modal error message is shown, and if you switch to another application, when you switch back, the modal dialog gets “lost” behind the main window that’s now inactive, so you have to jump through alt-tab hoops to re-activate the window that you’re trying to acknowledge/close.

The first relevant post I came across on Stack Exchange basically said, newer versions of Delphi add some properties like setting the parent of a popup, to overcome this problem. If on the other hand you are using Delphi 7…

Again? Another problem that just upgrading Delphi would fix without a convoluted workaround? It’s not a trivial upgrade to make the app work on a newer version of Delphi, but I could be trading one bucket of problems for another….I just wish there was an edition of Delphi that wasn’t so expensive, or I probably would have already.