troydavisson.com - my thoughts written down http://blog.troydavisson.com/ en-US Copyright 2008 Tue, 29 Jul 2008 00:49:15 -0500 http://www.sixapart.com/movabletype/ http://www.rssboard.org/rss-specification One giant leap
So many times as I'm working on web applications, I end up thinking to myself "there's got to be an easier way".  I've only been seriously programming for about 10 years now and my first introduction language to programming was Perl.  For those that know and work with Perl, you know that "there's got to be an easier way" is a recurring theme.  Perl is a very powerful language that has a lot of flexibility, but for my uses and the types of things I've found myself building, it was almost always too much.  I don't need a programming language that can do everything because I don't do everything.  One of the common things known about Perl is that there are many different ways to do the same thing, and it's not difficult at all to have 10 different methods for doing something that are completely cryptic to anyone other than the developer that wrote it.  This is a great idea.  It gives freedom of coding style, freedom of choice and flexibility in how something works.  The problem is that for the majority of the time, it's simply too much for me to handle.

For the past couple of years, I've passively watched the growth of PHP as a programming language and would constantly find myself criticizing it for it's lack of flexibility or being "too easy".  I've seen demos and examples of platforms like Ruby on Rails where you can create an entire blog in 5 lines of code.  I like to maintain a lot of programming control in applications I write, and that's doing way too much for me (something about most of my code being put in place to reverse something that it did automatically for me just doesn't seem right).  I knew Perl was making me work harder than I wanted to but I didn't know of a different language that I felt would strike the right balance for me in terms of programming control and ease of use.

About 2-1/2 years ago, I was working on a side project that involved integration with Wordpress (and then later, phpBB) which was sort of forcing me into a position of needing to know a little bit about PHP.  The end goal of what I was trying to do involved tracking a user's session (in the case of phpBB, actually using phpBB's session) through a site that included iframes so the system needed to either be able to save the cookie, or, if the browser rights were too strict, at least be able to pass the session ID along with every link so every other page would still have session data.  It was when I discovered PHP's use_trans_sid setting that I was hooked.  With that feature enabled, PHP would automatically buffer anything I was echoing and would go back in and add the session data to the end of any link I was echoing and it would automatically add a hidden form element containing the session information to any form I was echoing so that forms could also continue the same session.  This feature got really close to the "doing too much" line, but it ended up saving me a lot of time and the end result was exactly what I'd hoped for (and it did it automatically!).

There have been countless numbers of examples of the same type of situation over the last 24 months or so.  number_format(), trim(), setcookie() and array_unique(), just to name a few.

So, with my new excitement about PHP only a few months old, I was sitting in the lobby of the Embassy Suites in Chicago last year on the final day of the RETS conference killing time before my 6pm train while still digesting everything that'd happened the last few days.  I'd seen some demos of some nice applications and other tools, but most of them were for Windows or needed to be run as complete programs rather than smaller tools that a developer could use.  I wasn't unfamiliar with working with RETS at that point because I'd written a few different internal clients in Perl, but I still couldn't say that I had a good grasp on the big picture yet.  I ended up taking out a pad and pen and made a full page of notes.  I was determined to find an easier way that would both make sense to me and make sense to someone else that had very little experience with RETS.

That piece of paper is what eventually became the list of goals and general structure layout of PHRETS, a free, open source PHP library for working with RETS.  In working with Perl, I knew that there were a ton of Perl modules that could be used to introduce functionality to a program but I had no idea how to write one.  I knew that other functions or classes could be written which would do similar things but I'd never needed to write them so I didn't know how those would be structured or designed.  I knew that in order for PHRETS to be easy for someone to use, it needed to be fairly easy for me to make.  I knew that if I tried to get too tricky with it, it would be too tricky for someone else to understand.  I knew that if average Joe Programmer me could use it, any average Joe Programmer could.

A few weeks, I posted the latest version of PHRETS which included the last major part of RETS I'd originally set out to support (GetObject).  Personally, this was a pretty big accomplishment because I finally felt like the code could be used by someone to do something useful.  The code works, it manages a session successfully, it retrieves and processes metadata (data formats) correctly, it can do queries and retrieve back any type of record from a RETS 1.x server, and it can query for and retrieve back photos.  It assumes as little as possible to remain as flexible as possible.

I think I've created an easier way and I encourage you, if you're even somewhat familiar with PHP, to give it a spin.  It'll only take a few minutes to know if it'll be successful or not, and I think you'll find an incredible amount of flexibility using the features it has.  For example, for retrieving photos, it assume nothing.  You tell it what to get from the server, it gets it and returns the actual image data back to you to do whatever you want with it.  If you want to put the image data into a file on your system, go ahead (you're a quick file_put_contents() away) and use whatever filenaming scheme you can dream up.  If you want to send the image data to Amazon S3 for storage and serving, you can do that, too.

Development on PHRETS isn't done as things are constantly changing and improving.  Additional support for versions other than RETS 1.5 are needed and the first steps towards those has already started.  Additional error checking and error reporting is needed so those are being added as well.  If there are features or items that you see that you'd like to suggest, please let me know.  All of the information about PHRETS as available at the project site, located at http://troda.com/projects/phrets .

Look for some additional PHRETS related posts coming over the next few weeks.  I'm scheduled to be part of a presentation during this year's RETS Conference in about a week where tools such as PHRETS will be showcased.
]]> http://blog.troydavisson.com/2008/07/one-giant-leap.html http://blog.troydavisson.com/2008/07/one-giant-leap.html RETS programming Tue, 29 Jul 2008 00:49:15 -0500 Things are different
About 2 weeks ago, we had an issue with one of our servers which caused some data loss.  Regular accounts were restored and things are pretty much back to normal, except that my personal accounts weren't in the backup rotation so a restore wasn't possible for these.  I did happen to have a recent backup of the database itself but the MovableType tables were for version 3.3.  The only version of MT I could find was the newest 4.1 and after looking at the table differences, it seemed like more of a challenge than I was interested in taking.

So, the blog is back up and running on 4.1 with the previous entries hand-copy/pasted into the new tables from a raw .sql file so I could at least have the posts back.  I may go through and do the same with the comments but I haven't gotten there yet.

We're obviously looking at our entire backup strategy very closely and hope to have something much more complete in the works in the coming days and weeks to avoid the situation that happened while I was attending the NAR Midyear conference and trade expo.  Unfortunately, due to said problems, I was only able to attend about 5 hours of the conference so I don't have the type of reaction or information I was hoping to have and post.
]]> http://blog.troydavisson.com/2008/05/things-are-different.html http://blog.troydavisson.com/2008/05/things-are-different.html work Wed, 28 May 2008 00:06:26 -0500 Introduction to PHRETS
PHRETS is a PHP library written for RETS. This library adds easy-to-use PHP functions for working with RETS.

My experience with RETS and real estate data in general is from a 3rd party "data consumer" role so the initial release of this PHP library is sort of targeted to that. As time goes on and more feedback is received, it may branch into different directions to offer a much more complete set of tools and features for using RETS.

I'm also fairly new to PHP myself so these functions are being written from a PHP "newbie" which should be easy for anyone remotely familiar with PHP to work with. The current code being developed is already passing the RETS compliance test so it's just a matter of cleaning things up and offering some better features and usable functions.

I've posted some introduction material at http://troda.com/projects/phrets/

If you're interested in being an early tester or being notified when I have some source code to release, sign up for the PHRETS mailing list (Google Group) and more information will be released as things develop.]]> http://blog.troydavisson.com/2008/04/introduction-to-phrets.html http://blog.troydavisson.com/2008/04/introduction-to-phrets.html RETS Fri, 18 Apr 2008 23:02:22 -0500 When Something Goes Wrong
Windows Vista has been a frustrating experience for me. I purchased a brand new laptop from Best Buy last July that came with Windows Vista Home Premium pre-loaded. I wasn't thrilled at the time knowing it was coming with Vista but I decided to try it out. What a horrible experience. After suffering through it for a few weeks, I couldn't take it any more and put Windows XP Professional on it. It worked great after that.

That whole experience opened my eyes a little. I remember a few years ago when games like FarCry were coming out and everyone was so impressed with what the game demanded out of a computer. Many people couldn't run FarCry at the time because it demanded more than what a typical PC had at the time. Once people upgraded, things were better and they could enjoy the game as it was originally intended.

When did we start treating Windows, a computer operating system, like a game? What's different between Windows Vista and FarCry is that people wanted that better, visual experience with FarCry and so they went for it. With Vista, it's basically forced on you which creates an instant negative reaction.

Brand new AMD Athlon64 X2 dual-core TK-53 1.7Ghz laptop with 1GB of RAM and it has trouble running an operating system. An operating system shouldn't be pushing the limits of what my computer can handle. An operating system shouldn't be having trouble doing the basic things that computer users use a PC for. If you can do those things well AND have a good visual experience doing it then great, but be an operating, working system first.]]> http://blog.troydavisson.com/2008/04/when-something-goes-wrong.html http://blog.troydavisson.com/2008/04/when-something-goes-wrong.html personal Fri, 04 Apr 2008 20:34:43 -0500 Sliced bread has found it's match
Couple of examples: upgrading from dialup to broadband, going from regular TV to TV with a DVR, going from a hardwired phone to a cellphone, upgrading from standard definition TV to HD, etc.

Once you go, you can't go back. It just doesn't happen. The thought of it makes you twitch. Maybe some are more severe than others like living with a regular TV in a bedroom while your home theater system has a DVR, but the feeling is still there. So, what am I adding to the list?

Threaded email.

Yeah, I know. You probably think I'm crazy, but try it and you'll see what I'm talking about.

I've made several other posts about our transition to using Google Apps for email (the other services are nice, but email is where it's at) so my opinions about this are based on my experience using Gmail. There are a few small features that Google has added to Gmail which makes this concept even better, but the threaded conversation is where it's at.

I don't have the most stable sleep schedule in the world. People that I communicate often with normally don't expect me to be available at a certain time of the day and usual just base it off of how long my IM client shows I've been idle. Because of this, it might be "late morning" (after lunch) by the time I'm sitting down for the first time to get my day rolling. Often times, someone with a lot more self-discipline starts an email conversation between 3 or 4 people at the start of the day and a few emails have already circled around with thoughts or add-ons by the time I can look at it. Typically, you'd have a few emails in your inbox that you'd have to read.

If you sort by date received, you find the bottom one that's still bold and read them one by one working your way up. Because they aren't grouped by topic (or email subject), you might be reading through 2 or 3 different conversations at once, or you're reading an email and scanning up the list to find the next with a similar subject to read it.

When using a program with threading capabilities, you have 1 -conversation- to read. When you select that subject, it brings up the body of all of those emails in a neat list so you can easily read what's been said while you weren't around. Now, you can quickly reply knowing that you're up-to-date on the conversation instead of replying to an old email that doesn't need answering anymore (I can't even count how many times this has happened to me).

2 additional pieces that make Gmail's service extra-useful with conversations: 1) use of labels, and 2) quote hiding.

Labels work like a typical "folder" would in Outlook or other program, except labels are more "web 2.0". Just like you can't put a piece of paper into multiple folders at once, you can't put an email into multiple folders at once. With labels, you can assign any number of labels to an email to keep yourself organized.

Quote hiding is the big feature for me. Typically, when someone replies to your email, their message is included at the top and what you originally wrote is included in the bottom. Gmail will automatically recognize those sections of the email and hide them from your view (there's a link you can click to make them show if needed). That way, even if the "true" email back to you includes replies of replies of replies of replies, Gmail will hide those so you only see the new text and can easily read through the entire conversation.

I highly recommend you at least give it a shot. Gmail has a lot of email management features like those mentioned above that make it a great email client (I've used Gmail's web interface exclusively since changing over to Google Apps) but I know other email clients may have similar features like that as well.

If you know of another program that you like that's integrated this concept, post a comment. I'm always willing to try something new that might be better than what I've got.

UPDATE: I found a good screenshot of Gmail's system showing the threaded view along with the view of all conversations.]]> http://blog.troydavisson.com/2008/03/sliced-bread-has-found-its-mat.html http://blog.troydavisson.com/2008/03/sliced-bread-has-found-its-mat.html work Wed, 26 Mar 2008 02:19:14 -0500 Communication using RETS













If you have any feedback for future audio or video pieces you'd like to see made, please let me know. I'm always looking for new ideas.]]> http://blog.troydavisson.com/2008/01/communication-using-rets.html http://blog.troydavisson.com/2008/01/communication-using-rets.html RETS Tue, 22 Jan 2008 00:15:19 -0500 Email and RETS
During that meeting last week, I was struck by a question that was asked.

"What all would change for our users in order for them to check their email?"

Because we've answered dozens of calls and emails from this company about email troubleshooting, we're very aware that everyone in the office checks email via Outlook (either Express or 2003).

The answer: "Nothing. We host our email in a standard format the same way that Google does. All we need to do is change a few fields one time within Outlook to point Outlook to Google instead of us and they're set. Everything functions the same from that point on."

It seemed so simple. It's something that I've often lost sight of. Running a POP3 server for email makes it so users can use Outlook, Eudora, Thunderbird, Blackberry, Opera Mail, etc. since they all work with POP3. You don't even question anymore if your email client (or email service) supports POP3. You just assume they do.

RETS, the Real Estate Transaction Standard, is the standard method used for clients and servers to talk to each other to exchange real estate data. If you have a RETS program, you access a RETS server and get what you need. That's what RETS does.

Right now, the interchangeability of RETS isn't totally there yet. Various implementations of RETS can cause issues from client to client. Various RETS workgroups are working on solidifying standards so that everyone plays by the same rules. It's hard to believe that a 150 page PDF document can have holes and remaining questions, but it does. They aren't major ones and are often easily identifiable and fixable to make it work but its hard to get something to work on the first try right out of the box.

To anyone using RETS or thinking about using RETS, follow the standard. It doesn't do any good for you to implement a POP3 server for email that requires special, unique authentication from a client because, chances are, programs won't be able to do it and people can't get their email. The advantage of investing time in implementing a standard is that it can be used as a standard and not just another crazy upgrade.

If you're an MLS with an IDX feed upgrading from FTP to RETS, I encourage you to follow the standard and implement what the standard defines. If you have extra business needs or requirements, please share those with the RETS community so those types of things can be implemented in the standard. If you decide that you aren't going to use DMQL for your queries, there won't be any RETS-compliant clients that can use your system out-of-the-box, so what's the point? Your custom server would require someone to write a custom client which involves huge time and cost investments and would require a LOT more support from your technical staff.

Compare it to email. In the near future, an agent will be able to access the MLS's data using a front-end program of their choice that gives them the features they're interested in. I like the features of Outlook. Maybe you like the features of Thunderbird more. Why should you be forced to use Outlook and why should I be forced to use Thunderbird? Maybe you're looking for a complete solution with a lot of features and I just want a simple way to search for property data for my clients.

It's an exciting time to be involved during a big change in this industry. NAR's sponsorship of RETS along with the support and services already offered by most of the major MLS vendor companies shows that it's only a matter of time.

The RETS meeting in Chicago last month opened my eyes even more to the overall picture of RETS and it's potential. Even weeks later, I'm still trying to digest a lot of the information shared and discussed. As time goes on, I'll post more and more of my thoughts of the trip.

One thing I do know is that the train ride was an experience. First time on a train (the train from Indianapolis to Chicago was changed to a bus ride so a train was only involved on the return trip) was interesting. About 5 and a half hours of Indiana countryside.]]> http://blog.troydavisson.com/2008/01/email-and-rets.html http://blog.troydavisson.com/2008/01/email-and-rets.html RETS Wed, 16 Jan 2008 23:27:42 -0500 Why use RETS and what are the advantages?


If you can think of other advantages that I missed, please post it in the comments for others to see.





Loading audio player...



]]> http://blog.troydavisson.com/2008/01/why-use-rets-and-what-are-the.html http://blog.troydavisson.com/2008/01/why-use-rets-and-what-are-the.html RETS Wed, 16 Jan 2008 23:24:02 -0500 Technical Introduction to RETS


Technical Introduction to RETS (my perspective)





Loading audio player...



]]> http://blog.troydavisson.com/2008/01/technical-introduction-to-rets.html http://blog.troydavisson.com/2008/01/technical-introduction-to-rets.html RETS Wed, 16 Jan 2008 23:20:00 -0500 New Audio/Video Series
Hope you enjoy. If you have any comments or feedback, please post.]]> http://blog.troydavisson.com/2008/01/new-audiovideo-series.html http://blog.troydavisson.com/2008/01/new-audiovideo-series.html RETS Wed, 16 Jan 2008 23:11:36 -0500 Progress Report: Google Apps
So far, we've transferred 3 "large" accounts over to Google Apps for email hosting (none really use the other features to my knowledge). The first account was my company's which only has 3 real addresses on it but it's high volume and it's important to me so I'm calling it "large". The second account was a domain entirely made up of forwarders to remote email accounts (about 40 in all). The third was a real estate company which is made up of about 25 POP accounts and 25 forwarders.

Before I get into that though, I guess a little background information is in order. Basically, spam lists don't seem to like us very much. On each real estate site we have, site users have the ability to register and be notified automatically when new properties come on the market that match their search criteria. Many users have pretty broad search parameters so we often times have a lot of emails going out each morning. All of these are opt-in with clear directions to turn off the emails. Seems simple and harmless enough but blacklists have had fun with us I think. The emails themselves don't normally end up being reported (I can't recall any time I've gotten a report from a spamlist and it's been related to these emails) but I believe they end up playing an important role in our problem. Read on.

So account 1 was changed. It involved making setting changes for 3 accounts on 2 different computers within Outlook. The setting changes and MX entry changes were done at the same time on the computers to keep everything consistent so we didn't lose any emails in the process. Pretty easy and slick.

Account 2 was a little more work but it was still pretty easy. Created the Google Apps account for them, setup all of the forwarders within the admin dashboard and we were ready to go. The MX record change didn't cause any problems because either emails were being forwarded by us or by Google. Either way, the end result is that the email is in the same place. We let one of the main people know what we were doing just so they knew (we emailed them after the change actually) but the change was easy.

Account 3 was a completely different story. The 25 email forwarders weren't much of a problem (just like Account 2) but getting 25 pretty computer illiterate people to change their Outlook settings was an event like none other. We made multiple meetings with the company about the change to ensure that they were comfortable with what we were going to be doing. About 12 hours before the planned changeover, I setup forwarders on our system to go to the temporary email addresses Google provides so emails could start going there (and so screens weren't blank when they changed their settings). On the day of the planned change, I spent almost the entire day in their office walking around to the different computers behind each user so they could log into their Windows network profile, launch the email program they use (was either Outlook Express or Outlook XP) and instruct them through the process of making the setting changes while I'm standing over their shoulder.

Long story short, that was about 3 weeks ago and the full MX record change didn't happen until a few days ago. There were a few users who weren't there that day and we've had to wait until there was a good time to walk them through the process.

So far....

The Good

  1. Solid solution. No downtime or access-related problems so far that I've noticed. Helps me sleep better
  2. Good spam filtering (more on this below). Filters both accounts and emails being forwarded on (we could too but we had to create a POP3 account for the forwarder to get our spam killing solution to zap the emails before sending them onto the remote destination and the local POP3 account would max it's quota and emails would start failing)
  3. Easy to use admin dashboard
  4. Integrated webmail and POP access. Very nice to have emails sent via Outlook appear in webmail also. The other webmail solutions were totally disconnected from POP clients
  5. Lots of space and growing. Up to almost 4,500MB at the time of writing this (and that's for the free account)


The Bad

  1. Minor "feature" is causing a lot of headaches. When you send yourself an email, it will appear as a new email within the webmail interface but it won't be downloaded via POP3. I think I understand the reason for doing this BUT when someone sends an email to themselves and it doesn't come back to them, it looks like there's a problem. If they could get rid of this feature and start allowing those emails to be available via POP3 access, that would make my life a lot easier. There is no good solution I've found to put a workaround in place for this that doesn't involve making changes to each individual user's settings.
  2. They're in control now instead of me. I've gotten a few questions about email delivery recently that I'd normally have no problem getting information about because I have access to all of our server logs but I'm a little in the dark when it comes to Gmail's system. Although this may seem like a big deal, it's not that bad because their solution is solid and it doesn't seem to suffer from weird glitches. I've eventually tracked down the issue of delivery with each of those reported problems and they've never been because of Gmail's system. I just wish I could punch in a few commands and find out what was going on myself, but I've found myself trusting their system to do what it's supposed to so I've focused my troubleshooting steps on everything else
  3. Advanced filtering and whitelist features are only available for the Google Apps paid accounts. Any person in a user's contact list will be automatically whitelisted (which is a good feature, by the way) but the advanced features within the paid service allow domain-wide settings instead of a per-account basis.


Quick, final note about the spam filtering. I'm still trying to learn how it works, how strict it is, etc. While we had the 3rd account (mentioned above) still forwarding using the temporary addresses, we popped onto a spamlist for a few hours which caused all email leaving our server to Gmail's servers to be marked as spam. It's good that they went through to their system and weren't totally rejected, but the spam folder isn't normally available via 3rd party POP3 clients so it's almost as bad as them being deleted and gone forever.

All 3 accounts mentioned above are fully pointed to Gmail now via their MX records. If it wasn't such a painful process of have their domain's DNS zones hosted elsewhere, I'd probably do that also but things seem to be going well so far. Now that most of our off-site email forwarders are off our system, I'm hoping we're done with these spamlists for a while. DNSstuff.com and I are becoming pretty good friends unfortunately.

Small side thought: I'm not convinced yet that it actually wasn't Google that got us listed with the spamlists this last time, but that's the name of the game with forwarders I guess. I'm hoping that they'll start seeing our daily email notifications as more relevant emails and not consider that excessive volume which would look like spam. Fingers crossed. I feel like I'm back in high school trying to please everybody so everyone will like me.]]> http://blog.troydavisson.com/2007/10/progress-report-google-apps.html http://blog.troydavisson.com/2007/10/progress-report-google-apps.html work Sat, 27 Oct 2007 23:06:06 -0500 Recent Move I'm just finishing up with my yearly move (or what seems like anyway). Going from a decent size place with 2 roommates to a bigger place by myself. Kind of quiet. Kind of weird. Very empty. Gives me lots of time to think so I'll be posting more in recent days and weeks. http://blog.troydavisson.com/2007/10/recent-move.html http://blog.troydavisson.com/2007/10/recent-move.html personal Sat, 27 Oct 2007 22:09:28 -0500 RETS @ Chicago 2007
This'll be the first RETS meeting I've attended and, really, is the first time I've had anything other than text conversations with people involved with RETS. I'm not sure what to expect and I have no idea how many other people are going to be there. I know the Inman conference was recently so there may be people who don't want to come to both. Who knows. We'll see.

I'm not much of an idea person but there are a lot of things about RETS that strike me a little odd so I hope to bring up some good points during the smaller discussions (assuming I don't chicken out and keep my mouth shut). At this point, I'm not that interested in the RETS2 Payload discussions (or really RETS2 that much overall) yet so I'll probably be at most of the other sessions, including ones I probably won't have anything to contribute to.

I've thought of some more RETS-related content I can put together to hopefully help other people getting a fresh start with RETS. I was recently able to help a developer from Massachusetts that opened my eyes to a lot of things. I'm not nearly as knowledgeable about RETS as probably most of the other people at this meeting I'm going to, but I'm also not brand new to it either. I think I'm in a unique position to be able to bridge the ever-increasing gap between those 2 sides and have the tools and knowledge to pull it off.

Stay tuned. We'll see how this goes.]]> http://blog.troydavisson.com/2007/08/rets-chicago-2007.html http://blog.troydavisson.com/2007/08/rets-chicago-2007.html RETS Mon, 06 Aug 2007 16:48:36 -0500 Google Maps & Interesting Places
3 spots of interest I've found so far:
- Venice, Italy: Very cool
- Madrid, Spain: Gooooooooal!
- A few miles north of the stadium shown in the above link is the construction site (just west of the train station) for the Torre del Espacio, a building featured on Discovery Channel's "Build It Bigger".]]> http://blog.troydavisson.com/2007/07/google-maps-interesting-places.html http://blog.troydavisson.com/2007/07/google-maps-interesting-places.html personal Mon, 30 Jul 2007 08:09:38 -0500 Email Forwarders and Spam
One of the many things I do is administer the servers we have that run the web services we host. Those services include website functions, database hosting and of course email.

Website functions include running and maintaining Apache. Because we pay fees to other companies for software and such, we get automatic updates as part of the package. This is great because we, honestly, don't have to do much as far as this service goes.

Database hosting includes making sure MySQL is up and healthy. For the same reasons above, there isn't much you have to do here; however, backups are even more important for this type of data because it's always changing (where maybe your actual code might not be).

Email is a completely different story. Completely. One of the few services that a server performs that relies on everyone playing nice is email. The problem is that not everyone does. Any machine on the internet can be turned into a spam spewing nightmare for system administrators.

The main way that spam is combated is by using spam filters at the server level. Yes, there are some email clients like Outlook 2003 that have built-in junk mail filters but most spam is blocked by the server so that you don't know about it. On those servers, they use various rules for checking spam. Some might look for the word "viagra" or "free prescriptions" and block email based on that. Some might look at the "From:" and determine if it's a valid address. Some might do a combination of all of that plus thousands of other rules.

The most popular (or annoying depending on your job title) is probably spam blacklists. These are lists that mail servers use to help share information with other mail servers about spammers. Just because your mail server hasn't gotten spammed by a notorious spammer doesn't mean that you have to in order to know about them. When emails come in, their criteria is checked against worldwide lists and is sometimes blocked based on that.

Now for the purpose of this article: email forwarders. Almost everyone uses them. If you have a company email address and a personal email address, you often just want your mail in one place so you ask your IT guy to forward your company email to your personal inbox. So, when an email is sent to you, it:

- leaves your friend's computer and heads to their ISP
- leaves your friend's ISP headed for your employer's server
- is received by your employer's server and redirected to your ISP's server
- is received by your ISP's server and made available to you

Great. Easy enough. Like an address change at the post office, mail is simply redirected to a new address. Here's the problem:

- spam leaves spammer's computer headed to their ISP
- spam leaves spammer's ISP headed for your employer's server
- spam is received by your employer's server and redirected to your ISP's server
- spam is received by your ISP's server, seen as spam and blocked/deleted (you never see it)

Sounds harmless unless you realize what your ISP is doing. When they get a spam email, they check to see who sent it to them. In any other case (when not using a forwarder), it's probably a server from another country. Eventually, if they get enough bad email from that server, they'll stop accepting email from it altogether. When you use forwarders though, the email your ISP blocks looks like it came from your employer and they think your employer is the spammer.

Eventually, once enough ISPs block spam emails from your employer's server, they are reported to the spam blacklists I mentioned earlier. This causes many other servers that use those lists for spam checking to block your employer's emails. Bad news.

This is my headache this last week. It's not the first time it's happened (and in fact, happened twice this week).

There are 3 possible solutions we're looking at. If you can think of others, please leave a comment on this article:

1) Stop allowing forwarders altogether (or stop allowing forwarders that redirect to remote systems). This fixes the problem altogether.

2) Reconfigure our system to scan forwarded emails before they're sent off. This has proven to be very difficult to do and, since no spam filter is perfect, spam could still get through which could give us a bad name.

3) Use an outside email hosting service that allows forwarders. Fixes the problem altogether (or at least makes it someone else's problem).

I think we're heavily considering option #3 at the moment. More specifically, Google Apps. Leverage the power of Google's already established Gmail email service (and their spam filters) with the size and scale of Google to provide a pretty powerful and feature-rich email experience for our clients.

Good luck to any other system admins that are fighting this right now. It's a tough position to be in. Maybe we'd be better off if we hadn't started off offering forwarders to ours users but we did so here we are. Maybe offloading email to an established service like Google for everyone's email (regardless of forwarder use) would be better for everyone and remove us from this equation.]]> http://blog.troydavisson.com/2007/07/email-forwarders-and-spam.html http://blog.troydavisson.com/2007/07/email-forwarders-and-spam.html work Sat, 14 Jul 2007 11:27:43 -0500