PopTrayU Bugtracker

Originally, for keeping track of issues with PopTrayU I attempted to use Sourceforge’s built in bug tracker. But, as I recall, I was less than enthralled with it for reasons I’ve long since forgotten.

putbugtracker2So then I started looking into other options, PHP based options I could run from the webhosting area on Sourceforge, and ended up selecting Mantis. I think I looked into bugzilla as well and I don’t remember what else. For the most part Mantis worked quite well, and had a slick-looking interface, and I loved the dashboard view where you could sort all the bugs by their status and it color-coded them in a way that just “made sense”.

But Mantis and Sourceforge just don’t play well together. Every time you submit a change to a bug, the next page you try to load returns a 503 “Guru Meditation” error due to the Varnish Cache Server installed by Sourceforge. Apparently it’s some sort of timeout error, and I’d spent a lot of time looking into fixing the issue, possibly more than once, without success.

So what ends up happening in reality? It’s too much trouble to update the bug-tracker in everyday use So I just keep track of “active” bugs in a spiral notebook sitting on my desk with a pencil and only using the bug tracker only for keeping track of “things I might implement someday”.

So I decided to give this version tracking issue another look. Wanting to stay away from Soruceforge’s cache server issues, I started looking at the options for bug tracking used by some other open source projects I follow.

Google Code is deprecated and on the verge of being shut down and not open to new projects, so although I’ve liked them in the past that was out.

Microsoft’s CodePlex I love how they prominently display options to vote on your favorite bugs. Super-useful when you’re thinking “which bug should I tackle next”.codeplexAnd it has a nice taxonomy for marking up the bugs–separate drop-downs for component, status, type, impact, etc for well-classified bugs. But it’s kind of boring and corporate, lacking color-coding, and the captcha nearly resulted in loss of data for both of the two first test entries I entered.

And then there’s the ever-popular GitHub. It’s bug tracker is simple–at times almost too simple. Component can’t be a separate field from the bug status–it’s all just a mashup of tags. I’d rather see the list of components in my project separate from the resolutions. But overall the bug tracker has some nice CSS and is quick and easy to add new bugs and update their status.

github-bugsIn the end I decided ease of entering and updating bugs was more essential than taxonomies and voting. If it’s not practically effortless to track the bugs, there’s a good chance I won’t use the bugtracker to keep track of features I’m considering developing.

So here’s the new bug tracker: https://github.com/jojobear99/PopTrayU/issues

PopTrayU Progress: Unread Message Filtering

There was a user requested feature for PopTrayU to add filtering based on read/unread status. This is something that is only technically feasible for IMAP since POP servers don’t store read/unread status. Anyway, I started working on implementing this feature, and decided that if you are only showing unread messages, you might need a mechanism to change the read/unread status–especially if previewing automatically marks the message read on the server.

So whip up some code for this…go to test it…and…nothing. No change to the message status.  Turn on some logging of server communication, and something looks off about the command set to the server…and the server response? Yeah, pretty sure that’s not what’s supposed to happen either.

Sent: C4 UID STORE 1264  (\Seen)<EOL>
Recv: C4 BAD Error in IMAP command UID STORE: Invalid STORE modifiers.<EOL>

This may not be so apparent in a non-monospaced font, but there’s a double space between 1264 and (\Seen) like a token got left out. But not being intimately aquainted with every nuance of the IMAP spec, I had to lookup the section on setting flags in the IMAP RFC for IMAP to be sure I even knew what the command was supposed to look like. It’s supposed to have a +FLAGS.SILENT in that missing token.

So, long story short, I had to update my copy of Indy from SVN to make sure the bug still existed on top of trunk, then I had to debug a variable type that disappeared in the new version of indy (it got replaced with something else), after retesting and trying a couple variants to see if it only exhibited this behavior when certain parameters were used, I filed a bug report to let the Indy team know about this issue, and left my code using a workaround with the better code commented out in the meantime. Remy and the Indy team are pretty responsive to fixing actual bugs, especially ones that are probably pretty trivial to fix like this (now incredibly useful but difficult to implement enhancements on the other hand, like IMAP IDLE I’m not about to hold my breath over…).

Aside from this particular sub-feature, there’s still quite a bit that needs to be done to make “unread only” mode happen. Like, the main window needs a whole new paradigm of basing the read/unread status off of the server for IMAP and off internal tracking for POP. And the message list needs to be using UIDs instead of relative message numbers for a bunch of different commands including but not limited to preview and delete.

And while I’m at changing it to download by UIDs, this would be a good time to evaluate whether it should be downloading only the “message envelope” (a limited set of headers that basically are the headers you see in your email client message list) instead of the “full headers” (including all the headers you usually don’t read like spam checker versions and hosts that redirected the email and so on that could be useful for rules) the way it does now. If you’re not making rules that use the extended headers, this could cut data transfer in about half, which could be a significant time saver on large inboxes, so it’s worth looking into, even though I’d probably still have to also support having the existing modes to support users who use those headers to create complex rules. And while I’m doing that…and so on and so forth it goes 😉

PopTrayU 5.1.0 Options Preview

I wanted to give you all a little preview of more of the new features I’ve been working on for the 5.1 version of PopTrayU.

To start here’s a screen-shot of a new panel/category on the Options tab to customize the Preview Window:

poptrayu_5.1_previewLots of subtle differences. Before we get too far, I’m sure someone is thinking “What’s with the font?” No, 12 point Segoe Print is not a new default (it’s something I picked arbitrarily for testing). But it does illustrate a new feature–large fonts support. 12 point isn’t exactly “that large”, but it’s large enough that things would be overlapping and cut off, and created a horrible user experience for for users who use windows large fonts feature, or just wanted a larger font in this application.

I’ve painstakingly gone through just about every screen in the application and added resizing code to make just about every screen “resize correctly” in both the vertical and horizontal directions if you use larger fonts. As an added bonus, this actually improves the user experience for users of languages other than English as well, because buttons and label areas that used to be fixed size now will resize intelligently, and increase in size, if the translated caption doesn’t fit on default sized button. At first I tried to do this with a grid layout manager, because that would have been the elegant, clean way to do things, but after wrestling with some frustrating bugs in how it auto-sizes, it ended up being more reliable to just position everything manually. So it’s quite possible somewhere I overlooked *something*, but we’ll call that a bug for now, and feedback on resizing issues that need to be addressed will be welcome.

Now, let’s talk about the list of tabs. I gave a minor overhaul to the list of categories, and made some changes to the order of the categories. Defaults has been moved to the first category, because it holds the option to change the language. And as a french-speaking user pointed out, you need to change the language before you change any other setting on a new installation. Seems sensible, so language selection is now at the very top of the first options screen. There are also two entirely new categories of options: Preview Window, and Rules.

The Rules options category gathers together existing options relating to rules that were previously spread across several tabs. Options like whether to log rules, whether messages marked as protected by rules require an extra confirmation to delete, whether to download part or all of the message body to make more sophisticated rules, and what happens when you blacklist something are all gathered in one place under a single logical category.

poptrayu_5.1_rules_panelYou may have also noticed that the old “Black List mark as Spam” checkbox has been replaced with the new “Blacklist Action” drop-down. I didn’t feel like it was intuitive what happens if you **don’t** mark blacklisted messages as spam–and the alternative involves things getting deleted. I take emails getting deleted seriously, and don’t want emails deleted because of bugs or because some configuration option was vague or unclear. So the drop down now forces an explicit choice between “Mark as Spam” or “Delete”. And I always do lots of extra testing (including on my own personal email) before releasing any changes that affect message deletion. I don’t want PopTrayU deleting your important mail–unless you really meant to do so.

The other new category in the list is Preview Window. There’s a couple of new features here, but previously, most of these options you had to change from a right-click menu on an actual preview window.

So, going back to that first screen-shot, you’ll see a new Tabs sub-category. This is for a new feature I’ve been wanting for some time. I’ve been torn over the design decision that PopTrayU should preview using the last selected tab or view type. I kept finding myself wanting PopTrayU to behave more like Outlook, where it automatically displays the email in HTML if the email has an HTML section, regardless of what view I was using to preview the previous email. But sine a lot of different people use PopTrayU in completely different ways to accomplish completely different goals, I didn’t want to eliminate the current option to have it display in the last used tab. So now you can choose between previewing with the last selected tab, or to always use a specific tab by default.

And, since Spam is sort of a special case, it gets its own independent selection. The main reason for this is that it’s likely that if you have images enabled, you may not want to display the email with images if it’s spam. So now you can choose whether you’d rather view spam as Plain Text, RAW headers, or any of the options supported for non-spam emails.

Back to the preview tab from the first screen shot. What can be customized for preview but isn’t on this tab? Fonts and Colors. When I first created this tab, initially I had put the options to customize the preview window font and colors (does not apply to HTML messages) on this new panel, however, I decided it would make more sense to have all of the font face, font size, and color customizations on the Visual appearance tab in one place.

poptrayu_5.1_visualAnd while I was at it, I finally fixed that eyesore that was the fonts selection panel. Originally I’d had the colors for the message list as separate drop-downs, which minimized the number of mouse-clicks needed to customize everything. But I decided it was getting a little too cluttered,  and confusing since only certain UI items can have the colors customized. I changed the previews to have borders–this was actually something I had wanted to do before, but there were some technical hurdles that required research to figure out how to make that happen.

Introducing the New Accounts Screen

For the upcoming 5.1 release of PopTrayU, I’ve redesigned the accounts screen:

5.1accountsIn the old design, which was basically the PopTray 3.2 layout with new features squished in around the edges wherever they’d fit without changing things too much, things didn’t resize well. Look what happens if you increase the font size:

5.0.15_largefontsEverything is overlapping, some of the captions don’t fit. It’s a bit…cluttered. And if you use a language other than English, the translations often had to be abbreviated to fit the space available.

The new design manually resizes and positions everything in both horizontal and vertical directions to accommodate the selected font size and language settings. The account settings are broken into logical groups, like splitting apart the server information for your server from the ways PopTrayU notifies you about, to make it the page easier to navigate. The sections are each collapsible as well, so the advanced options or notification options can be hidden if you don’t want to see those sections. Oh, and those buttons at the bottom? They are now intelligently sized…if the caption doesn’t fit, they will be resized larger to fit the caption. Users of languages other than English will also especially appreciate this new feature.


Indy 10 & Security Enhancements in PopTrayU

So to be a little more detailed from my last post about how upgrading Indy affects the protocol plugins for PopTrayU…

PopTray, historically, only included POP3 as a “core” protocol. Anything else (Pop3 with SSL, Pop3 with APOP, IMAP, IMAP with SSL, etc) had to be configured as a seperate plugin. There’s a DLL interface that the plugin has to implement (though many of the methods are optional), which can be implemented in either Delphi or C++.

PopTray had a set of “example” plugins that included the most common additional protocol needs (those mentioned above), implemented in Delphi.

The POP3 SSL plugin for PopTray (a seperate DLL) directly extended pop3.pas (one of the source files for the core executable). That class has a reference to an Indy callback object that updates the a progress meteron the UI. In Indy 10, one of the parameters was changed from Int to Int64 and changed whether it was const. So the older DLL + the newer EXE (or conversely new DLL + older EXE) just don’t quite mix well and initial testing was showing “random” crashes and such undesirable behavior.

Well, if we’re going to have to break the existing plugins for SSL and IMAP, now would be the time to make any other changes/additions to the plugin interface, so we don’t have too many different versions of the plugin interface, wouldn’t it?

One change I’ve been contemplating for some time is including IMAP and SSL as “core” features rather than plugins. It’s becoming increasingly common for mail providers (Gmail, for example) to require SSL to even get POP access to an email account.

Another change I’d been contamplating is offering an “advanced connection options” feature. Almost every email client (even the one on my phone) has a menu or dialog where you can tinker with some various cryptic sounding settings beyond “SSL/TLS enabled”, like STARTTLS, Explicit TLS, SASL, APOP, etc. In PopTray, you had to have a lengthy list of protocols to have a seperate protocol for every permutation of feature combinations you needed, and not only does that lead to having to make the protocol support only the least common denominator of desirable settings, it made configuring options on a per account basis difficult. The plugins were set up so there was a configuration page for the plugin itself (which to be honest I’ve never seen actually used for more than an “about” dialog), but no way to configure settings differently between two different accounts that both require SSL connections, even if I put the plugin settings dialog to better use.

So I spent a bit of time researching what all these crazy-sounding advanced settings acronyms mean that other mail clients use, and then tried to create a list of which ones would be useful to support. One in particular that I was initially undecided about was APOP. Even the iPhone’s default mail app doesn’t have APOP support. In the days of SSL, does anyone really use APOP instead (APOP just uses an MD5 hash to obscure the password on a plain-text connection, which is obviously better than sending any passwords in plaintext over public wifi where there might be hackers looking for your passwords, but far less secure than SSL. But some research seemed to indicate that there is at least one major ISP (Earthlink) that, as best I can tell, flat out does not support SSL even in this day and age, but does support APOP. Well, for the sake of anyone who unfortunate enough to have such an account, I’d rather not deprecate increased security that was already available in previous versions of PopTray and PopTrayU.

And then after deciding which advanced options I’d like to support, the more difficult decision becomes how to group and arrange and label them on the UI. Should I label a checkbox “STARTTLS” or “Explicit TLS” or put slashes between alternate version of the name? They mean the same thing, but different apps label it differently. That particular option opens the connection over the standard insecure port (110 for POP) and then sends the server a command to “explicitly” enter encrypted connection mode, rather than requiring you to use a different port if you want a secure connection. Apparently, there’s some debate about which way is the better way to do things, but if you want to be flexible and accomodate the widest assortment of mail server configurations, ideally you’d support both options. and just to make things confusing, there’s also a mode called Implicit TLS. That means that you still use a TLS secure connection, but the connection doesn’t make an insecure “handshake” first, you connect over the secure port and it’s just assumed you’re going to be in secure mode because you connected on the secure port.

Anyway, I don’t want to bore you with all the details of every single email connection option I ended up learning about and what all those options mean exactly…for most users, they just want to check the box their ISP or geek friend/adviser told them to use, and not know a thing about what it means 😉

But it suffices to say, there’s no one standard UI for setting all those advanced settings. They have different labels and groupings depending on the app and which features it supports. So one of the hardest aspects of addding all these extra advanced connection options is deciding on a usable interface for setting the options, not to mention how the plugins can interact with the options available to enable or disable options that protocol doesn’t support. So I’m taking my time with this and doing a lot of testing, and well, some of this secure stuff is tricky, and I’m still working on some of the kinks that happen if you mix and match combinations of features that just aren’t compatible and shouldn’t be available at the same time.

One of the features in particular that I’m excited to add support for is SASL authentication. SASL wasn’t even invented until 2006, so obviously, it was after PopTray’s time, and still new enough that everyone doesn’t automatically expect SASL support, let alone know what it does. The layman’s version is you don’t need to know a thing about how it works, you just need to know that that SASL + SSL can be more secure than SSL alone, so you should absolutely be using it if it’s an option available. Internally, there’s a whole bunch of different “authentication mechanisms” the app can choose from, and it will send your password using the most secure one of those mechanisms that the server supports. So it’s going to be one of those “enabled by default” kind of features that will make newer version more secure without having to be an IT professional to know what settings to pick.

The one thing I’m probably not going to do is include OpenSSL DLL’s in the base installer for PopTrayU, simply because, it’s a liability to bundle SSL abilities. The US and many other countries have all kinds of weird rules about importing and exporting software that include “strong” encryption. So some people may want the “insecure only” version, to avoid the whole issue of whether it’s legal to travel to xyz country with this software or it’s installer loaded on their PC.

Overall, I am attempting to minimizing changes to the plugin interface, and add the new features as optional methods, in hopes of breaking as few existing plugins as possible. There are poptray plugins for various webmail interfaces that I have no interest in supporting or developing, but if they continue to work as-is, I’d rather leave things that way.

Latest on my PopTrayU development

So in a slight twist of irony since my last post about working on PopTrayU, after all the trouble of getting Indy 9 set up on my current machine, I’ve decided to port PopTrayU from Indy 9 to Indy 10.

For those of you unfamiliar with Indy (which is probably pretty much everyone who is not an experienced Delphi developer), Indy is an open source networking library that is available both for .NET and Delphi platforms. It does most of the heavy lifting as far as the technical implementation of connecting to a POP3 server, downloading and parsing the email headers (though I had to override a few little pieces of that to workaround Delphi 7 international character support issues) and that sort of stuff.

Version 9 is, well, a legacy version at this point. You can’t escape being asked “why are you still using Indy 9 anyhow?” if you ask a question pertaining to it on a programming forum. Version 10 has a lot of new features, and performance improvements and so forth. But, they made a lot of changes to the interfaces, so it’s not like you can just download version 10, and your code for version 9 will just work without changes.

I’d already done the vast majority of the porting to version 10 legwork when I was working on my Lazarus port, so there were only a few holes (“todos”) here and there that needed to be fixed. It was probably more painful to actually do enough research/reading to even figure out how to get versions 9 and 10 “somewhat” concurrently installed in delphi (to the degree that’s possible), than it was to makde PopTrayU built on Indy 10 itself work.

But I’ve been holding off on releasing these changes as the next version, because aside from wanting to make sure it’s stable and not buggy, it introduces the first change to the plugin interface since branching from PopTray that will make certain older plugins no longer compatible. I’ll write more about those changes in my next post 😉

Disabling Auto-Check

I recently finished implementing a new feature for PopTrayU, conditionally disabling automatic checking for an email account based on the time of day. This feature already existed on a global level: you could disable checking for all accounts, so that your computer would not check for email while you’re sleeping or awaken you with a loud “you have new email” notification sound. But some users, particularly users who work from home and use the same computer for both work and pleasure, were making a feature request that they’d like to be able to turn off notifications for their work email outside of normal working hours.

The part of the implementation that was most difficult, of course, was the requirements analysis and the detailed design. On the surface, this feature seemed pretty straight-forward, but as I started to dive into actually implementing it, a lot of nuances come up where it isn’t exactly clear how this feature should work, and the design has to be hashed out in more detail, and the requirements clarified.

Each account has a status icon indicating whether the account is disabled or has new email or no new email. (For the sake of clarify, lets call this new per-account no-auto-check feature “sleeping”.) Should the visual appearance of an account that is “sleeping” appear like a normal account or a disabled account? Something else? What should the status-bar for the account say when an auto-check happens, but a single account is skipped because of the time? If the user presses “Check All Accounts” manually, should that skip or include accounts that are sleeping? What if they press “Check” for a single “sleeping” account?

Once I nailed down the design, implementation was pretty simple. Because the feature existed on a global level, most of the implementation was quite similar to the logic and UI of how it was done for the global level.

As far as how I decided to resolve some of those issues:

An account that is disabled is represented by an envelope with a red minus symbol on it, like this:  Previously, this was the only reason a specific account would be skipped when an auto-check occurred. Now we have two reasons an account might be skipped. It could, for some users, be handy to be able to differentiate the reason the account is not automatically checking, especially should they be troubleshooting settings gone awry. I experimented with making a new symbol that resembled sleeping, like something with zzz’s or a moon, but given the limited amount of space in a 16×16 image, trying to make an image that is clear and recognizable is difficult, so I decided to use the same symbol as a disabled account, but use a contrastingly different color to differentiate the different status.  

As far as when to check/not-check:

  • Automatic timer-interval checking, pretty straightforward, sleeping accounts should be skipped.
  • Manually pressing the “check all” button: more ambiguous, it’s not really an “auto-check”, but it behaves a lot like one. In the end I decided “check all” should replicate the behavior of auto-check, but on-demand, so accounts that are not in a state allowing automatic checking should be excluded, so it should behave like a disabled account.
  • Manually pressing the “Check” (this account) button: this one seemed pretty straightforward, even a disabled account will check when pressing this button, so a sleeping account should definitely check in this case.

On the status-bar for the account, I decided it ought to say *something* when an auto-check is scheduled but skipped, if only to handle the case where account1 is sleeping and account2 is enabled. The UI only shows one account at a time, so a visible indicator that an account check happened, and the time is useful. So “account skipped’ with the time. Autocheck was scheduled, but skipped beccause of the no-check hours for that account. Hopefully that’s clear, and it it wasn’t, they could read the manual (to be updated in the next release) to find out what “account skipped” means.

PopTrayU to Lazarus/Free Pascal Porting Update

Well, I’ve gotten a lot of the “cosmetic” stuff fixed. Here are updated screenshots, that you can compare to the ones my last update about this project:

First, here’s the options screen.

You’ll notice it looks pretty much about right. I had to use a third-party time control (ZVDateTimeControls) to get the time picker working. Lazarus doesn’t have an equivalent of TDateTimePicker. They have a TDateEdit, but that only does calendar and not time only. But the third party one seems to work fine, though I’m not in love with the arrows on it, I’d rather see it using the native windows picker control under the hood, since cross-platform compatibility is not a primary concern of mine at the moment. But at least it’s functional for now.

Here’s the main window at the moment:

You’ll notice the main screen now looks “almost” like the real thing now. The toolbar is there, but the labels are in the wrong spot. I stumbled almost on accident onto the solution for how you do the labels on the right. Apparently you have to use TSpeedButton(s) on a TBevel instead of using a TToolbar to get labels on the right. Sounds pretty doable, but is lower priority than some of the other issues I’m dealing with.

Those tabs also don’t appear totally “right”. You’ll notice there’s some weirdness about the borders for them. The tabs are a TTabControl because they don’t have individual content, it’s really just an array of accounts it’s representing. But there’s a key difference in Lazarus from Delphi that the TTabControl is not designed to have child components, so it’s resizing it wrong to make room for the components that should be nested on the tab. This might actually be kind of involved to fix. One guy supposedly submitted a patch at one time that made it work more like Delphi, but it got rejected because it didn’t work “right” for all the different platforms Lazarus supports. So looks like I could try to track down that patch and make it into a third party control, or I could change the TTabControl to a TTabSheet, and replicate the buttons onto every tab, but that would lead to a lot more UI overhead. But either way, it’s not worth fixing unless I find a way to fix the blocking issue that could be a showstopper.

Right now the blocker issue is Indy 9. In 2005, Marco van de Voort ported Indy9 to Free Pascal, at least for 32 bit architecture, not 64 bit. But then he decided to upgrade to Indy10 so it wasn’t maintained. So, showstopper issue, Indy9 isn’t working right. I’m focusing on the “test account” button right now since that’s about the simplest network-operation the app does. Connect to the Pop3 server, login, and disconnect. Login is actually working. The problem is disconnecting. It’s reading garbage instead of “+OK” for the disconnect message, so it fails to disconnect and throws an exception that causes other problems, like skipping actually closing the connection. The problem is, using Wireshark to sniff what’s actually going on in the network, the server is actually sending the correct response back. But what PopTrayU is seeing when I step through the debugger is half of the account’s domain name, “bear.net” Stepping through the routines where it moves pointers around to find the response, it looks like it’s reading the wrong place in memory, it’s skipping right over where it should be reading, and reading from some random spot a little bit farther out that presumably contains memory that was previously used for something else. Tricky right? The code doing this is totally obscure, and has very few comments, so it might take a while to understand it well enough to figure out *why* it’s reading the wrong address when it goes to disconnect. The warnings in Lazarus that it’s doing Pointer to Integer conversions could be a hint though. Theoretically that shouldn’t mess it up on 32 bit, but you never know, seems to be doing some bad pointer math somewhere.

Interestingly, poking through the Indy10 codebase, they appear to have reworked the whole “managed buffer” that’s giving me issues and eliminated this weird pointer math. And Indy10 is officially supported in Free Pascal, so I may have to bite the bullet and upgrade. Apparently there were some non-trivial issues when Renier tried to port to Indy10 once long ago, but perhaps they would be less nasty to resolve than buffer overrun issues.

Oh, and unrelated to all of that, I also have the tray icon partly working now. The right click menu is working, but changing the icon images is not. The rules and about screens are also on the todo list if I can resolve blocker issues. The about screen lost all the strings in the tables, because they’re encoded in binary in Delphi, so I have to retype them. And the rules stuff was using some components that have to be manually ported/fixed.

Initial Stages of Porting PopTrayU to Lazarus

Yesterday I started working on trying to port PopTrayU to Free Pascal/Lazarus. The built-in converters seemed to do a good job for the most part, though they did leave hundreds of errors for me to clean up before I could even say it compiled, and its pretty far from, well, working.

But here’s where we’re at right now:

This is the main screen, and it does look fairly close to what it’s supposed to look like, but as you’ll notice, a few key UI items (the toolbars and list of messages) are missing.

On the options screen, things look even worse:

Though I’d be willing to bet that in the end, the problems with the options screen turn out to mostly be cosmetic (a list that needs to be manually repopulated), and that the main screen will turn out to be way more work to get working.

So what issues have I been running into as I port? Thanks for asking.

The CoolTrayIcon component that PopTrayU relies on for the tray icon in the windows tray is super-broken to the point where it was easier to comment out everything relating to the tray icon rather than cobble the Tray Icon component into running skeleton code. The Tray icon relies heavily on Application.Handle and Application.Hook methods to get references to the application to do snazzy things like minimizing to the tray. Those methods are conveniently not available in Lazarus. So I’m going to have to do more research about workarounds for that, because there doesn’t appear to be a straightforward “just do this instead” option.

When the Lazarus porting guide said most of the things that aren’t implemented are “things that are very windows specific” it turns out to be quite the list, and some of those components that are “wontfix” missing components are ones that the app uses.

For example, THotKey has no equivalent. THotKey is a class that hooks into the system settings to set system wide hotkeys to control PopTrayU when you’re using other programs. That’s not to say it can’t be done, but if it can be done the code is probably going to look a lot more ugly. But that’s a low-priority feature compared to say, checking your email and the tray icon being also broken.

I actually came across this handly list that shows which Delphi components do and do not have equivalents in Lazarus. Unfortunately for me, the “not included” list includes TActionManager, TActionToolBar, TPopUpActionBar, and TCustomizeDlg. I may be forgetting others. So basically, in Delphi, all the menus and toolbars used to use Actions to link the menu items to the code it runs when you click on them, and then you could drag and drop menu items into toolbars and vice versa and fully customize the toolbars, which is a really nice to have but not essential feature. Having menus and toolbars is, however, an essential feature, so I’m going to have to check out TToolbar, TActionList (that is available) and TPopupMenu which are candidate replacement classes.

Some of the more straightforward replacements that the auto-code-fixer couldn’t figure out for me were things like:

  • TTabSet -> replace with TTabControl
  • TColorBox -> add import ColorBox
  • GetTime() in SysUtils -> replace w/ Time()

There were also a couple type (struct) definitions that I had to manually add (related to hotkeys), which wasn’t a big deal other than figuring out what was missing.

One of the tricks I found handy for fixing things was keeping Delphi open at the same time and using the “Browse Symbol at Cursor” feature to plug in the name of whatever it was complaining was missing to find out where that was defined, so I could figure out whether it’s an import name that’s missing or something from one of the dependencies or delphi or wherever it’s from.

But overall, most of the issues have been with UI stuff and NOT indy stuff, so it’s looking optimistic that this is a feasible port worth continuing to pursue. I was having a couple minor issues with indy, but I think they’re minor, figuring out where to store the data.

On with the porting!

HTML Email Preview

One of the complaints I’ve heard about the feature of having html preview of emails is that spam-bots often track whether emails are read by embedding a unique image, and tracking if the image gets downloaded. It’s a legitimate concern, one I was well aware of when I added the HTML preview option. But it came down to being a value vs. risk proposition.

The HTML preview window in PopTrayU
The HTML preview window in PopTrayU

For version 1.0 of the feature, that seemed like a perfectly reasonable risk to accept. When I check my email from my phone, it displays in HTML and doesn’t have an option to disable loading images either. And, if like me, you run your email through a really good spam filter (like Gmail), you aren’t getting a lot of spam in your inbox in the first place. But that’s not necessarily the case for people using PopTrayU to check their ISP email account, which is a very common use case.

But that doesn’t mean it shouldn’t be somehow addressed in subsequent releases.

Initially, when I’d looked into HTML components, my first choice was a native Delphi HTML rendering component. But upon investigation, the component was too old and did not support CSS at all, which was a deal-breaker since font tags are completely deprecated these days. I mean, I could have added CSS support myself, but that is an extremely complex feature to implement correctly, with many corner cases. You’re pretty much writing a web-browser minus the network connections part. Much bigger scope than I was willing to take on.

So my second strategy was to use the TWebBroswer control to render the HTML, which is essentially a wrapper for an embedded internet explorer window. You get the HTML rendering almost for free with that technique. But…Internet explorer has this known bug that the “offline mode” flag in the component is completely ignored, and it decides whether you are online by a global registry setting or whether an internet connection is present. Less than ideal for this purpose. Any external content is really undesirable for spam email previewing.

So then I looked into maybe I could just pre-process the HTML and strip out the images. Removing <img> tags wasn’t too difficult. You can either do that with a regular expression or a for loop iterating through the characters in the message. Well, it turns out to be a little more complex than that. Images don’t just come from <img> tags anymore. Images are embedded in CSS that’s slapped in the body section of the message using url(…) syntax. Images are embedded through inline CSS. Almost any CSS component could have an image as part of a background or bullet image or border or any number of other places. And then you need to strip out any javascript, because that might be loading images or external content, so there go onclick, onload, and other properties that could be on any element in the document. And you need to get rid of external css files, you need to get rid of object tags, maybe even image maps. Point is, it gets really complex really fast.

After realizing how many loopholes there could be in pre-processing, I came back to the conclusion that pre-processing isn’t the “right” strategy. It’s a hack, that might work most of the time, but it’s never going to be as sure-fire as doing it right. The right way would be to block the requests at the network level or the browser level. Oh, you want to request a file off the internet? No, you haven’t clicked the magic button that says this is trusted content, forbidden.

Which brings me back to offline mode and that annoying bug that IE ignores the offline mode parameter. Why did they even bother putting that in their spec for the interface to their WebBrowser control in the first place? I’m sure there’s some technical reason they decided to not handle the parameter correctly. They even have a knowledge base article for a very old version of IE (IE5?), explaining the issue, and that there are no known workarounds. Very helpful.

Then I started to look into other replacements for the TWebBrowser control. There was one that initially looked promising, a firefox component that attempted to be a direct replacement for the TWebBrowser. So close, in fact, that you can patch a compiled EXE to put it in without touching the code. But…there were some big buts. But the newest version it comes for is Firefox 1.5, a seven year old version. This code is not maintained, it does not have bug fixes, it’s poorly documented, nevermind the possible security issues. Oh, plus it doesn’t implement any of the “more complex” aspects of the TWebBrowser interface, for example, it can’t create the page from a memory stream rather than from a file on the hard drive, so then I’d have to start littering the hard drive with unnecessary temp files. And even if I changed the code to do that, there’s no guarantees I’d be able to get it to work at all.

Then there’s a Google Chrome Frame. It’s not a direct replacement, and is not an activex control like the TWebBrowser or the Firefox browser component. It only supports HTTP requests (not file) unless you muck around with some settings, probably have to start writing temp files, etc. I haven’t played with it enough to assess whether it would be a feasible option or not.

One thing I did implement was adding some logic that if a message is marked as spam when you hit preview, it switches to plain-text view instead of HTML automatically. At least that solves the problem of how do you preview a message that might be spam, without having to open another non-spam email and switch tabs and then close the window and preview the spam message. That was a step in the right direction.

And then I also figured out that if you set “offline mode” in any internet explorer window, they ALL go offline, including the embedded browser window in PopTrayU. So, if you really don’t want external content to load, all you have to do is set IE offline. Of course, then you cant surf the internet in IE at the same time. But if you’re pretty serious about security, there’s a good chance you don’t use IE as your primary browser anyhow. So that’s one option. A workaround. Less than ideal for some users. But may be totally sufficient for others. Potentially I could pragmatically enter offline mode before rendering the page and then return to online mode after, affecting other IE windows as well, though that would affect other IE windows doing stuff in the background (hope you don’t have a big file download going on in the background!).

So then I get back to debating whether I should just nix the whole experimental “hide images” feature entirely, because it’s just not worth the trouble of making it work, or keep pressing in to try to find a way to make it work that’s going to be a lot of work for a modest return. There are plenty of other features and bugs and refactoring that could easily occupy my time instead. Decisions…