Add/Remove Gmail Labels using Indy 10 in Delphi

According to support threads, Indy’s TIdImap4 supports fetching Gmail labels but doesn’t have support built in for adding and removing labels. I’m still working on figuring out how to fetch and read the fetched labels, but by using the SendCmd function to manually send the IMAP commands I got adding and removing labels working. So I thought I’d share that code:

function AddGmailLabelToMsgs(const uidList: TStrings; labelname : string) : boolean;
begin
  try
    if HasCapa('X-GM-EXT-1') and (uidList.Count > 0) and (labelname <> '') then begin
      IMAP.SendCmd(ImapCmdNum(),'UID STORE '+uidList.CommaText+' +X-GM-LABELS ("'+ labelname + '")',['OK','BAD','NO'], true);
      Result := IMAP.LastCmdResult.Code = 'OK';
    end else
      Result := false;
  except
    on E : Exception do
     begin
       //Dialogs.ShowMessage('Exception class name = '+E.ClassName);
       //Dialogs.ShowMessage('Exception message = '+E.Message);
       Result := false;
     end;
  end;
end;

At the heart of it, I am using Indy’s SendCmd function to manually send the command. It was a little tricky figuring out how to use this function since documentation and examples are a little scarce. The first parameter is a sequence number so that the response can be matched up with the command. Second is the remainder of the actual command. Third parameter is what responses to look for to know this command is “complete”, and the fourth is whether to look for a single line response or multi-line response.

I also have some commented out exception handling that was helpful to debugging when my SendCmd wasn’t formatted correctly, but is neither needed nor useful for production code.

Remove label is basically the same thing but with a minus sign:

function RemoveGmailLabelFromMsgs(const uidList: TStrings; labelname : string): boolean;
begin
  try
    if HasCapa('X-GM-EXT-1') and (uidList.Count >0) and (labelname <> '')  then begin
      IMAP.SendCmd(ImapCmdNum(),'UID STORE '+uidList.CommaText+' -X-GM-LABELS ("'+ labelname + '")',['OK','BAD','NO'], true);
      Result := IMAP.LastCmdResult.Code = 'OK';
    end else
      Result := false;
  except
    Result := false;
  end;
end;

And here’s the helper functions, minus the global/class/whatever variables:

ImapCmdNum() generates a sequence number for the IMAP transaction, essentially a replacement for NewCmdCounter in IdImap4, since it’s a protected property and client code can’t use it. At first I tried using “A1”, “A2”, etc. but it turned out that using a C prefix, “C1″, C2”, etc. like Indy was important for using Indy to process the response when I got into debugging why things weren’t working right at first…but now I’ve completely forgot exactly where it goes wrong if you don’t use a C.

function ImapCmdNum(): string;
begin
  Result := 'C'+IntToStr(cmdNum);
  inc(cmdNum)
end;

And HasCapa() is a function I added to make sure that the Gmail capabilities are actually available before trying to set/unset labels…Might not be totally necessary, but it seemed like a good idea. Under the hood, I just used Indy’s built in Capability function that calls CAPABILITY on the server, with a little caching so I wouldn’t need to do this more than once per session.

function TProtocolIMAP4.HasCapa(capability: string) : boolean;
begin
  if (capabilities.count = 0) then begin
    IMAP.Capability(capabilities);
  end;
  Result := (capabilities.IndexOf(capability)<>-1);
end;

How to Use Indy 10 Imap Intercept function for Logging

It took me a while to track down enough information to figure out how to use Imap.Intercept to log raw IMAP client/server communication to troubleshoot an SSL account issue, so thought I’d share a basic code snippet on how to have IMAP log to disk the communications.

var
idLogFile1 : TidLogFile;
begin
IMAP := TIdIMAP4.Create(nil);

idLogFile1 := TidLogFile.Create(nil);
idLogFile1.Filename := 'C:\temp\imaplog.txt';
idLogFile1.active := True;

IMAP.Intercept := idLogFile1;

//...
end;

Installing Indy9 from Source Code

I’m finally working on getting Delphi 7 set up on my Win7 x64 machine. I think it’s been over a year since I last did this on my Vista machine or my Win7 x32 netbook, so my mind totally went blank trying to remember how to install the lastest version of Indy 9.

I already had pulled the code from Indy’s SVN server and been able to transfer that from the old machine, so I was one step ahead there.

I pulled up the “ManualInstall.txt” document in the Indy source directory and tried to follow the directions, only to find it didn’t work. At first it looked like it was a permissions issue, I needed to be running a batch file called fulld7.bat from a command line running on elevated privileges to get the script to run without errors, however, even after sorting all that out, when you get to the step of Installing the *.bpl, even after you’ve correctly remembered which one is the right menu option install such, it just doesn’t work. Complains about a missing file (that just doesn’t seem to exist in the provided repository or compiled files directory).

Well, after much fiddling and trying to figure it out on my own, since the internet had no solutions, it turns out, those directions probably are obsolete, and doing it the way that actually *worked* was much easier…just not obvious if you haven’t been working with Delphi recently.

In the Indy 9 directory from the SVN, there’s a .DPK package file for each delphi version. So, since I’m using Delphi 7, double click Indy70.DPK, it opens in Delphi with a package window visible displaying a list of all the files. At this point you’re two clicks away from a working Indy install. Click “Compile” and then “Install” on the package window, and dismiss the dialogs that confirm those steps work, and suddenly Indy is on the component toolbar without even restarting Delphi.

So there you have it. I’m thinking documenting the steps on installing this that work will be helpful, because by the next time I get around to trying to install Indy on a new machine, I almost certainly will have forgotten how you install it, since you do it once and never touch it again, and odds are good that it will need re-installing after a hiatus where I haven’t touched Delphi in a while period and can’t remember off the top of my head the difference  between installing a component and installing a package, since that’s very delphi-specific terminology.

New Project: Porting PopTrayU to FreePascal?

Poking around on the internet for one reason or another (maybe I was browsing posts about Delphi on the Programmers Stack Overflow/Exchange site?), I came across something interesting. Lazarus, the open-source IDE for Free Pascal. 

Lazarus is an open source equivalent to Delphi. Well, not exactly equivalent, but pretty close.. It comes with (not terribly well documented at the moment) tools to port Delphi code to Free Pascal, and comes with the Lazarus Component Library (LCL) which is very similar to Delphi’s Visual Component Library (VCL).

The idea of porting PopTrayU to an open source development platform instead of an expensive closed source one (namely: Delphi XE) is very much in the spirit of open-source development. It was always one of the dreams of Renier (the original PopTray developer) to port PopTray to a free platform, and he indicated he might be interested in continuing to develop the app if it could be ported. And it would save me the expense of upgrading to Delphi XE to fix some various bugs.

I’d previously looked into porting PopTray to the free version of Turbo Delphi 2006, but my feasibility analysis results were that it was not feasible. That version of Delphi, aside from being discontinued/no longer supported or “officially” available, was that it left out a key feature (namely, 3rd party component support).

Free Pascal and Lazarus, from what I’ve said so far looks more promising as far as being feasible. Indy components has been ported to Free Pascal, which is a key stepping stone, though investing whether I’d have to port to Indy 10 first or not remains to be determined. So far everything else I’ve been reading looks like it would “probably” be possible without significant loss of critical features, but it wouldn’t be a plug and play transition, you have to rename a bunch of things and change the imports, and some of the UI might get mangled porting it so you might have to redraw parts of the UI. But so far haven’t come across any definite showstoppers.

At the moment, the biggest risk for showstopping problems would be if there is a missing library eg “things that are very windows specific” may be missing. But nothing specific yet. There would definitely be some hurdles to fix the unicode issues, but that’s to be expected and certainly fixable. Even though Free Pascal is multi-platform, the probram would still only run on Windows, because of Indy and it’s limited compatibility with other platforms.

So that’s on my todo list. And in between downloading Lazarus and now, they’ve already gone from version 0.8 to 1.0 of the IDE. Actively developed and being improved regularly is a positive sign to look for.

HTML Email Preview

One of the complaints I’ve heard about the feature of having html preview of emails is that spam-bots often track whether emails are read by embedding a unique image, and tracking if the image gets downloaded. It’s a legitimate concern, one I was well aware of when I added the HTML preview option. But it came down to being a value vs. risk proposition.

The HTML preview window in PopTrayU
The HTML preview window in PopTrayU

For version 1.0 of the feature, that seemed like a perfectly reasonable risk to accept. When I check my email from my phone, it displays in HTML and doesn’t have an option to disable loading images either. And, if like me, you run your email through a really good spam filter (like Gmail), you aren’t getting a lot of spam in your inbox in the first place. But that’s not necessarily the case for people using PopTrayU to check their ISP email account, which is a very common use case.

But that doesn’t mean it shouldn’t be somehow addressed in subsequent releases.

Initially, when I’d looked into HTML components, my first choice was a native Delphi HTML rendering component. But upon investigation, the component was too old and did not support CSS at all, which was a deal-breaker since font tags are completely deprecated these days. I mean, I could have added CSS support myself, but that is an extremely complex feature to implement correctly, with many corner cases. You’re pretty much writing a web-browser minus the network connections part. Much bigger scope than I was willing to take on.

So my second strategy was to use the TWebBroswer control to render the HTML, which is essentially a wrapper for an embedded internet explorer window. You get the HTML rendering almost for free with that technique. But…Internet explorer has this known bug that the “offline mode” flag in the component is completely ignored, and it decides whether you are online by a global registry setting or whether an internet connection is present. Less than ideal for this purpose. Any external content is really undesirable for spam email previewing.

So then I looked into maybe I could just pre-process the HTML and strip out the images. Removing <img> tags wasn’t too difficult. You can either do that with a regular expression or a for loop iterating through the characters in the message. Well, it turns out to be a little more complex than that. Images don’t just come from <img> tags anymore. Images are embedded in CSS that’s slapped in the body section of the message using url(…) syntax. Images are embedded through inline CSS. Almost any CSS component could have an image as part of a background or bullet image or border or any number of other places. And then you need to strip out any javascript, because that might be loading images or external content, so there go onclick, onload, and other properties that could be on any element in the document. And you need to get rid of external css files, you need to get rid of object tags, maybe even image maps. Point is, it gets really complex really fast.

After realizing how many loopholes there could be in pre-processing, I came back to the conclusion that pre-processing isn’t the “right” strategy. It’s a hack, that might work most of the time, but it’s never going to be as sure-fire as doing it right. The right way would be to block the requests at the network level or the browser level. Oh, you want to request a file off the internet? No, you haven’t clicked the magic button that says this is trusted content, forbidden.

Which brings me back to offline mode and that annoying bug that IE ignores the offline mode parameter. Why did they even bother putting that in their spec for the interface to their WebBrowser control in the first place? I’m sure there’s some technical reason they decided to not handle the parameter correctly. They even have a knowledge base article for a very old version of IE (IE5?), explaining the issue, and that there are no known workarounds. Very helpful.

Then I started to look into other replacements for the TWebBrowser control. There was one that initially looked promising, a firefox component that attempted to be a direct replacement for the TWebBrowser. So close, in fact, that you can patch a compiled EXE to put it in without touching the code. But…there were some big buts. But the newest version it comes for is Firefox 1.5, a seven year old version. This code is not maintained, it does not have bug fixes, it’s poorly documented, nevermind the possible security issues. Oh, plus it doesn’t implement any of the “more complex” aspects of the TWebBrowser interface, for example, it can’t create the page from a memory stream rather than from a file on the hard drive, so then I’d have to start littering the hard drive with unnecessary temp files. And even if I changed the code to do that, there’s no guarantees I’d be able to get it to work at all.

Then there’s a Google Chrome Frame. It’s not a direct replacement, and is not an activex control like the TWebBrowser or the Firefox browser component. It only supports HTTP requests (not file) unless you muck around with some settings, probably have to start writing temp files, etc. I haven’t played with it enough to assess whether it would be a feasible option or not.

One thing I did implement was adding some logic that if a message is marked as spam when you hit preview, it switches to plain-text view instead of HTML automatically. At least that solves the problem of how do you preview a message that might be spam, without having to open another non-spam email and switch tabs and then close the window and preview the spam message. That was a step in the right direction.

And then I also figured out that if you set “offline mode” in any internet explorer window, they ALL go offline, including the embedded browser window in PopTrayU. So, if you really don’t want external content to load, all you have to do is set IE offline. Of course, then you cant surf the internet in IE at the same time. But if you’re pretty serious about security, there’s a good chance you don’t use IE as your primary browser anyhow. So that’s one option. A workaround. Less than ideal for some users. But may be totally sufficient for others. Potentially I could pragmatically enter offline mode before rendering the page and then return to online mode after, affecting other IE windows as well, though that would affect other IE windows doing stuff in the background (hope you don’t have a big file download going on in the background!).

So then I get back to debating whether I should just nix the whole experimental “hide images” feature entirely, because it’s just not worth the trouble of making it work, or keep pressing in to try to find a way to make it work that’s going to be a lot of work for a modest return. There are plenty of other features and bugs and refactoring that could easily occupy my time instead. Decisions…

Writing a string to a stream in Delphi

There are two tricks to this. Most of this comes from what Embarcadero suggests on how to write a string to a stream. Another website suggested multiplying the length times the number of bytes to make sure the code works on both unicode and non-unicode builds, something one would be wise to consider when developing legacy code on Delphi 7, that may someday get ported to a newer version. One less thing to worry about later.

procedure WriteStringToStream(stream: TStream; const appendText: string);
begin
  stream.WriteBuffer(Pointer(appendText)^, Length(appendText)*SizeOf(Char));
end;

Show Modal in Delphi

I was looking today at whether I could fix that annoying bug where a modal error message is shown, and if you switch to another application, when you switch back, the modal dialog gets “lost” behind the main window that’s now inactive, so you have to jump through alt-tab hoops to re-activate the window that you’re trying to acknowledge/close.

The first relevant post I came across on Stack Exchange basically said, newer versions of Delphi add some properties like setting the parent of a popup, to overcome this problem. If on the other hand you are using Delphi 7…

Again? Another problem that just upgrading Delphi would fix without a convoluted workaround? It’s not a trivial upgrade to make the app work on a newer version of Delphi, but I could be trading one bucket of problems for another….I just wish there was an edition of Delphi that wasn’t so expensive, or I probably would have already.