May 2007 Archives

Here's a little about the guy behind this blog:

My name is Troy Davisson and I currently live in Indianapolis, Indiana. I'm 22 years old and can almost always be found in front of a computer monitor somewhere. Everyone that knows me knows that I'm very rarely not doing something work-related which makes me late to just about everything not work-related. I truly enjoy the work that I find myself doing (most of the time) so it's easy to spend a ton of hours on things.

As far as work goes, I'm the owner of an IDX services company. My company has been around for about 5 years now. I've been a professional web developer and programmer for about 10 years. My first exposure to programming in general was with Delrina FormFlow many, many years ago. At the time, I was too young to connect the dots in my head of where I might be able to go with that interest but once I was exposed to programming on the Internet, I was hooked. Those adventures have lead me to learn almost all aspects of computer and network technology from operating system support (Windows and Linux) to networking (protocols, wiring, installation, etc.) to server technology (IIS, Apache, DNS, load balancing, high availability solutions, backup services, etc.) to web site creation (design, graphics, Perl, PHP, Javascript, CSS, MySQL, etc.).

I got started in the IDX services industry (it has to do with real estate and property data for those that don't know) when we (my business partner at the time and I) won the bid on a random elance.com project we bid on. Once we were exposed to IDX, we continued to build out our service around what we thought would be good features to have and created a business out of it. Shortly after that, my business partner decided to continue down the career path that most of his family was already involved in and became involved full-time with real estate as an agent. Since then, I've been doing what I can to keep everything afloat which has proven to be very difficult at times. Recently, I (personally) have been focusing less on offering IDX services and trying to get involved more with general consulting and development work for people that might find my skill set useful. With that, I've had various opportunities to explore unknown waters dealing with technology and even business in general so this blog will probably be mostly about that.

Anyway, my email address is troy.davisson@gmail.com if you're dying to know more about me.

There's so much more to RETS than can be described here and there are many possible applications for it that many don't even realize yet.

RETS is backed by the National Association of REALTORS® and is here to stay. Work is continuing on RETS 2.0 which has many enhancements over the 1.x versions of RETS.

For more information about RETS and how to get started, visit www.rets.org.

Other links for relevant information mentioned here:

- StandardNames
- RETS-compliant clients
- RETS-compliant server vendors

If there are other questions you have that you can't find answers for, email me and I'll try and point you in the right direction.

RETS has a lot of benefits but let's talk about the big ones and the problems they solve.

How do I know how to connect to get the data I need?
With RETS, all you need to know from the MLS is the RETS login URL, your username and your password. From there, you can learn everything else on your own almost entirely using code if you want.

It seems like using a complex system would be harder for an MLS to support. How is this easier?
In most cases, if you have a question about how to log into a typical FTP server and process their data, you'll end up talking to someone that didn't actually create it so your answers will be scattered (if you get an answer at all). Maybe someone went through the MLS software and enabled new fields to be included in the feed. You can't expect everyone at the MLS to know about this so you may be disappointed with the answers you get.

With RETS, it is a more complex system, but it's well documented and as long as both the client and the server are compliant with the standard, you should have little trouble getting setup. You don't need to request help or documentation from them because it's either online (at the rets.org website) or available through the RETS server you're connecting to.

Logging in is pretty easy but then what do I do?
The process is very similar to what you'd do when you first get FTP access to an IDX feed for the first time. Once you login, you figure out what information is available. In RETS, these are called Resources. Once you request the Resources, you'll get back a list of information types that you can learn more about. From there, you can learn about the format of that data and learn how to search it for what you need.

Don't I need to understand the query language used by the MLS's database software to be able to get information?
Nope, that's a major benefit of RETS. RETS uses it's own query language and, when combined with StandardNames, allows you very easy access to pull property information that you need. The RETS server translates your RETS query into something it can understand, pulls the information out of it's database (the way it knows how) and morphs it into the RETS format so it's easy for you to understand when it sends it to you.

If RETS is so powerful and is truly live access to the data, can't I just have a website that connects directly to the RETS server instead of downloading a copy of the data locally?
Sure, and there are products that exist that do just that. Your website might have search criteria for listing price and number of bedrooms. That information could be put into a RETS query, sent to the RETS server, and the response could be displayed back to the user of the site.

The problem is that your MLS could catch onto what you're doing and could limit the amount of information you can pull (using queries) from the RETS server. They may even revoke your access to their system. In most cases, when you want to use data on your website, it's best to download a local copy of the data and access it from there.

You keep mentioning building a website using RETS access. Aren't there other applications for this system?
Absolutely. Having multiple MLS's in the same geographical area provides the potential for those MLS's to share property information between them so that agents in those areas have more to show potential buyers and provides agents with more exposure for their sellers. This is just 1 example. There are many others.

I really, really don't like the MLS software the MLS has for me to access it. Does RETS do anything in that situation?
In short, yes. If your MLS has a RETS server that you have access to, you could use any RETS-compliant program (called RETS clients) to access that data. This introduces more options for agents and brokers because they aren't tied to single user interface.

There are drawbacks, however. In most cases, the MLS software offers a lot of features. So far, it's difficult for a RETS client to offer the same features that the MLS might be able to offer through their own software because their software is made specifically to work with their data in their area. Work is continuing with RETS to try and introduce more capabilities for information sharing.

(continued in Part 6)

RETS, the Real Estate Transaction Standard, is our solution.

This standard allows the MLS's to meet you halfway. Maybe they won't give you direct access to their database but you can have live access to their database through a protected system interface.

Now we're talking.

As you can probably tell by the name, RETS is the standard used to move real estate information back and forth. The idea is that RETS defines:

1. how the 2 systems (client-server) connect and communicate back and forth
   
2. the format of the information being passed back and forth
   

Maybe you would normally download the once-daily file from an FTP server somewhere. 1 MLS you work with provides it in the /IDX/download/daily directory with a filename like "listings-residential.tar.gz" and another MLS provides it in the /NEW_IDX directory with a filename like "PropResi-5_11_2007-06_27_01-full.zip". With RETS, you connect and communicate in a common way so you don't have to deal with unique situations like this. RETS replaces the need for an FTP server with a completely separate authentication system. In addition, because your client RETS program logs into a server RETS program, the server can apply business-specific rules and restrictions on you which might give you more access to more data than someone else might have.

Like we mentioned before, the data you normally get could be in any format with any number of fields with any kind of data type (text, numbers, etc.). With RETS, the format of the data is available in a live, technically accessible format so you know exactly what you're going to get. If the MLS adds some new fields to the feed, information about those fields is available so you can make the necessary changes on your end (hopefully using a program that does it automatically).

If this wasn't enough, RETS also defines common names for fields called StandardNames. This allows more standardization among all of the MLSs that support RETS. Normally, one MLS might provide you with a field called MLS_Number and another might give you MLNUM. These are the same fields but they have different labels. With RETS using StandardNames, you can know and expect that ListingID will be the MLS# for that property. This allows for much greater interoperability so you can use the same RETS client with any RETS server and have the same results. Try that with your FTP downloader!

(continued in Part 5)

Getting a little more technical now.....

Let's assume you're developing a website for someone. Naturally, the best way to have updated information in a solid, technically noticeable format is to have direct access to the MLS's database. If you're betting on the MLS giving this to you, I think you'll be disappointed.

Furthermore, consider the extremely possible scenario where you're running one type of system and the MLS is running another. Trying to simply copy their data into your format is trying to fit a square peg into a round hole. You can't do it without a lot of work and finesse.

Maybe you get "BedsAll BathsAll StreetAddress BathsPartial RoomsAll" from the MLS (maybe an Oracle database) and you're trying to fit that into a "Bedrooms TotalRooms FullBathrooms HalfBathrooms StreetNumber StreetDirection StreetName" format on your system (maybe a MySQL database).

Square peg, round hole.

So...

Direct access to a database is too much. Access to an FTP server somewhere with once-daily updated data isn't enough.

Meet RETS.

(continued in Part 4)

So, now, you've got data somewhere online in an exported format from the MLS. Great! That just saved a large amount of time, but we aren't done yet.

The good:

• You can actually see information about properties!
  
• The MLS didn't have to do much to make that happen because they probably allow many people to have access to that information so you probably aren't charged anything (or at least, not as much) to make that happen. Saved money!
  
• They automatically update the data every day (usually) so you can expect updated property data that often. Sure beats copy/pasting information every day.
  

The bad:

• You've got a lot of information but it's hard to read. You need to figure out how to break the file apart so you can know what each piece of data means. Not too bad.
  
• You've got a file in a semi-programmable format but you don't know what fields are where and what each piece of information is. In most cases, the information for actually understanding the file is available in PDF format from your MLS. Having this documentation is extremely helpful but isn't updated often and it's in a format that you can't have a program understand so you end up reading it and interpreting it manually. Yuck.
  
• Data that's updated daily is decent in most cases, but what if you want something a little more often? You could download the file every 60 seconds if you wanted but unless it's updated, it doesn't do anything for you. You'd need the MLS's cooperation for that. Probably not going to happen.
  
• When the MLS makes changes to the data (i.e. they add a new field to the feed), there's very little you can do to know about this. If they don't tell you ahead of time, you've just been blindsided. You're expecting 150 pieces of information and you get 151 so things start to break down. You can't see what changed because, chances are, you don't get a new PDF every evening and if you request it from them again, it's probably not updated.
  

(Note that everything above is just talking about getting and processing the data portion of the feed. Getting and processing photos for those properties is normally a totally different process with it's own challenges. For the sake of keeping this simple, we'll ignore this aspect of it for now.)

If you're shopping around the Internet for an IDX vendor for your IDX-enabled website so you can show site users properties from your local MLS, the above process is the challenge. If you're the IDX vendor, you're all too familiar with this process because you probably manage dozens of data feeds and experience these kinds of problems almost on a daily basis. My heart goes out to you.

So, what can we do to make this even better? This is typically where you stop if you've never heard of RETS, but we're not stopping because you're about to.

(continued in Part 3)

For a moment, forget about protocols, character lengths, abstraction layers, integer fields, HTTP headers and all of that, and let's look at the problem starting from the very beginning:

If you're an agent or broker (or a developer for one), the MLS has data and you want it.

They could print off each property sheet and mail it to you but that's too slow and inefficient and doesn't allow you to really do anything with that information once you have it.

They could call you over the phone and read off everything to someone that would re-type it all in a format that you can use, but that takes even longer and requires phone time with people that would probably takes weeks to get that information, and by the time you're done, it's out of date.

They could copy/paste the property sheets into emails and email them to you. Even though this is a little faster, that still turns into a TON of emails and you'd have to copy/paste that information into something that makes sense to you.

You could copy/paste the property sheets using your own MLS access software but that's really sloppy, takes a ton of time and still doesn't get you the information you want so you can use it without a lot more work.

Let's get a little more technical.....

They could create an exported version of the MLS data and make it available somewhere online. You'd somehow grab that data, figure out what it all meant (what format it was in, how things were separated, etc.) and then turn it into something you could use. This is a much better solution compared to the ones above but still has a few drawbacks. We're getting close, though.

(continued in Part 2)

Many times, as you work through the architecture of your site, you wonder how to control access to certain functions or abilities. If your site has basic articles of text for example, you might think of the following:

• Users: basic read privileges with no admin rights
• Editors: read, write and edit privileges with limited admin rights
• Admins: all rights, including read/write/edit, add/edit/delete editors, etc.

The problem is that, even though this is what comes to mind the most, this is often the least ideal way to do it.

Recently, I've been building projects using a different scheme that I'd like to share. This isn't something I came up with but maybe it'll save you some time and headaches and will allow you to break free of the traditional way of doing things.

First, you define all of the available functions you'd like to control. Going along with the example above, we want to control:

1. who can create articles
2. who can modify existing articles
3. who can delete articles
   
4. who can add/edit/delete people with privileged access to do #1 - #3

The really cool part about this method is that you can extend it as much as you want. You'll see.

Ok, going through the list again. Starting at 1, double the number each time and give that number to each item. So we end up with:

1 - create articles
2 - edit articles
4 - delete articles
8 - admin rights
(next would be 16 if we had one, etc.)

Now, for each user, you give them an "access level". If you want someone to only be able to edit articles, their access level is 2. If you want someone to just have admin rights, their access level is 8. If you want someone to have access to create and delete (but not edit) articles, their access level is 5 (1+4). If you want someone with all access, their access level is 15 (1+2+4+8). To make it easier on yourself, you could make a program that automatically does the math here based on what boxes you have checked. Figuring out someone's access level at first isn't too bad but when you go back later to modify the level, it gets a little frustrating without code to help. Just a suggestion.

Now, in your code, the easiest thing to do is create a small function that breaks it down. For example (in PHP):
----------------------------------------------------------------------------

function check_access($check_level) {

$highest_level_control = 16384; // why not?
$users_level = $users_access_level_pulled_from_somewhere;

for ($i = $highest_level_control; $i >= 1; $i = $i / 2) {
  if ($i <= $users_level) {
    $level_track{$i} = 1;
    $users_level = ($users_level - $i);
  }
}

if ($level_track{$check_level} == 1) {
  return 1;
}
else {
  return 0;
}

}

----------------------------------------------------------------------------

In the above code, I used 16384 as the highest level control (this would be the value of the 15th control in your list if you had that many). If you really only plan on using the 4 mentioned in the example, you could change 16384 to 8.

Now, in the section of your code used to delete articles (remember, access control value "4"), you'd do something like:
----------------------------------------------------------------------------

if (check_access(4) == 0) {
  print "Go away";
  exit;
}

----------------------------------------------------------------------------

Pretty cool, huh?

This lets you break free of the hierarchy that you're used to. You can give certain people very specific control over something. You could even go as far as giving certain people certain access to specific fields within a form. Obviously, your access control numbers get a little large because of the number of controls you have but it doesn't matter. The number shouldn't ever be seen if you have code put together to help you add and edit those.

Also, something to keep in mind is that the number assigned to your control has no meaning at all except that it has to be unique and is hard to change once you get started. "admin rights" could just as easily be a value of 1 and "create articles" could be 8. You totally break free from the hierarchy thinking you're used to because everyone's access can be unique and very specific. Very cool.