Recently in work Category
If you've looked at this blog before, you've probably noticed that the design has changed. While I'd love to say that this was a calm and peaceful upgrade to the newest version of MovableType, it was anything but peaceful.
About 2 weeks ago, we had an issue with one of our servers which caused some data loss. Regular accounts were restored and things are pretty much back to normal, except that my personal accounts weren't in the backup rotation so a restore wasn't possible for these. I did happen to have a recent backup of the database itself but the MovableType tables were for version 3.3. The only version of MT I could find was the newest 4.1 and after looking at the table differences, it seemed like more of a challenge than I was interested in taking.
So, the blog is back up and running on 4.1 with the previous entries hand-copy/pasted into the new tables from a raw .sql file so I could at least have the posts back. I may go through and do the same with the comments but I haven't gotten there yet.
We're obviously looking at our entire backup strategy very closely and hope to have something much more complete in the works in the coming days and weeks to avoid the situation that happened while I was attending the NAR Midyear conference and trade expo. Unfortunately, due to said problems, I was only able to attend about 5 hours of the conference so I don't have the type of reaction or information I was hoping to have and post.
About 2 weeks ago, we had an issue with one of our servers which caused some data loss. Regular accounts were restored and things are pretty much back to normal, except that my personal accounts weren't in the backup rotation so a restore wasn't possible for these. I did happen to have a recent backup of the database itself but the MovableType tables were for version 3.3. The only version of MT I could find was the newest 4.1 and after looking at the table differences, it seemed like more of a challenge than I was interested in taking.
So, the blog is back up and running on 4.1 with the previous entries hand-copy/pasted into the new tables from a raw .sql file so I could at least have the posts back. I may go through and do the same with the comments but I haven't gotten there yet.
We're obviously looking at our entire backup strategy very closely and hope to have something much more complete in the works in the coming days and weeks to avoid the situation that happened while I was attending the NAR Midyear conference and trade expo. Unfortunately, due to said problems, I was only able to attend about 5 hours of the conference so I don't have the type of reaction or information I was hoping to have and post.
For me, there are a lot of things that fit this category. Yours will undoubtedly be different from mine, but you should be able to relate to these things and understand the feelings behind it based on your own experiences.
Couple of examples: upgrading from dialup to broadband, going from regular TV to TV with a DVR, going from a hardwired phone to a cellphone, upgrading from standard definition TV to HD, etc.
Once you go, you can't go back. It just doesn't happen. The thought of it makes you twitch. Maybe some are more severe than others like living with a regular TV in a bedroom while your home theater system has a DVR, but the feeling is still there. So, what am I adding to the list?
Threaded email.
Yeah, I know. You probably think I'm crazy, but try it and you'll see what I'm talking about.
I've made several other posts about our transition to using Google Apps for email (the other services are nice, but email is where it's at) so my opinions about this are based on my experience using Gmail. There are a few small features that Google has added to Gmail which makes this concept even better, but the threaded conversation is where it's at.
I don't have the most stable sleep schedule in the world. People that I communicate often with normally don't expect me to be available at a certain time of the day and usual just base it off of how long my IM client shows I've been idle. Because of this, it might be "late morning" (after lunch) by the time I'm sitting down for the first time to get my day rolling. Often times, someone with a lot more self-discipline starts an email conversation between 3 or 4 people at the start of the day and a few emails have already circled around with thoughts or add-ons by the time I can look at it. Typically, you'd have a few emails in your inbox that you'd have to read.
If you sort by date received, you find the bottom one that's still bold and read them one by one working your way up. Because they aren't grouped by topic (or email subject), you might be reading through 2 or 3 different conversations at once, or you're reading an email and scanning up the list to find the next with a similar subject to read it.
When using a program with threading capabilities, you have 1 -conversation- to read. When you select that subject, it brings up the body of all of those emails in a neat list so you can easily read what's been said while you weren't around. Now, you can quickly reply knowing that you're up-to-date on the conversation instead of replying to an old email that doesn't need answering anymore (I can't even count how many times this has happened to me).
2 additional pieces that make Gmail's service extra-useful with conversations: 1) use of labels, and 2) quote hiding.
Labels work like a typical "folder" would in Outlook or other program, except labels are more "web 2.0". Just like you can't put a piece of paper into multiple folders at once, you can't put an email into multiple folders at once. With labels, you can assign any number of labels to an email to keep yourself organized.
Quote hiding is the big feature for me. Typically, when someone replies to your email, their message is included at the top and what you originally wrote is included in the bottom. Gmail will automatically recognize those sections of the email and hide them from your view (there's a link you can click to make them show if needed). That way, even if the "true" email back to you includes replies of replies of replies of replies, Gmail will hide those so you only see the new text and can easily read through the entire conversation.
I highly recommend you at least give it a shot. Gmail has a lot of email management features like those mentioned above that make it a great email client (I've used Gmail's web interface exclusively since changing over to Google Apps) but I know other email clients may have similar features like that as well.
If you know of another program that you like that's integrated this concept, post a comment. I'm always willing to try something new that might be better than what I've got.
UPDATE: I found a good screenshot of Gmail's system showing the threaded view along with the view of all conversations.
Couple of examples: upgrading from dialup to broadband, going from regular TV to TV with a DVR, going from a hardwired phone to a cellphone, upgrading from standard definition TV to HD, etc.
Once you go, you can't go back. It just doesn't happen. The thought of it makes you twitch. Maybe some are more severe than others like living with a regular TV in a bedroom while your home theater system has a DVR, but the feeling is still there. So, what am I adding to the list?
Threaded email.
Yeah, I know. You probably think I'm crazy, but try it and you'll see what I'm talking about.
I've made several other posts about our transition to using Google Apps for email (the other services are nice, but email is where it's at) so my opinions about this are based on my experience using Gmail. There are a few small features that Google has added to Gmail which makes this concept even better, but the threaded conversation is where it's at.
I don't have the most stable sleep schedule in the world. People that I communicate often with normally don't expect me to be available at a certain time of the day and usual just base it off of how long my IM client shows I've been idle. Because of this, it might be "late morning" (after lunch) by the time I'm sitting down for the first time to get my day rolling. Often times, someone with a lot more self-discipline starts an email conversation between 3 or 4 people at the start of the day and a few emails have already circled around with thoughts or add-ons by the time I can look at it. Typically, you'd have a few emails in your inbox that you'd have to read.
If you sort by date received, you find the bottom one that's still bold and read them one by one working your way up. Because they aren't grouped by topic (or email subject), you might be reading through 2 or 3 different conversations at once, or you're reading an email and scanning up the list to find the next with a similar subject to read it.
When using a program with threading capabilities, you have 1 -conversation- to read. When you select that subject, it brings up the body of all of those emails in a neat list so you can easily read what's been said while you weren't around. Now, you can quickly reply knowing that you're up-to-date on the conversation instead of replying to an old email that doesn't need answering anymore (I can't even count how many times this has happened to me).
2 additional pieces that make Gmail's service extra-useful with conversations: 1) use of labels, and 2) quote hiding.
Labels work like a typical "folder" would in Outlook or other program, except labels are more "web 2.0". Just like you can't put a piece of paper into multiple folders at once, you can't put an email into multiple folders at once. With labels, you can assign any number of labels to an email to keep yourself organized.
Quote hiding is the big feature for me. Typically, when someone replies to your email, their message is included at the top and what you originally wrote is included in the bottom. Gmail will automatically recognize those sections of the email and hide them from your view (there's a link you can click to make them show if needed). That way, even if the "true" email back to you includes replies of replies of replies of replies, Gmail will hide those so you only see the new text and can easily read through the entire conversation.
I highly recommend you at least give it a shot. Gmail has a lot of email management features like those mentioned above that make it a great email client (I've used Gmail's web interface exclusively since changing over to Google Apps) but I know other email clients may have similar features like that as well.
If you know of another program that you like that's integrated this concept, post a comment. I'm always willing to try something new that might be better than what I've got.
UPDATE: I found a good screenshot of Gmail's system showing the threaded view along with the view of all conversations.
A couple of months into our small deployment of Google Apps and I figure it's about time for an update with my experience on how it's been going.
So far, we've transferred 3 "large" accounts over to Google Apps for email hosting (none really use the other features to my knowledge). The first account was my company's which only has 3 real addresses on it but it's high volume and it's important to me so I'm calling it "large". The second account was a domain entirely made up of forwarders to remote email accounts (about 40 in all). The third was a real estate company which is made up of about 25 POP accounts and 25 forwarders.
Before I get into that though, I guess a little background information is in order. Basically, spam lists don't seem to like us very much. On each real estate site we have, site users have the ability to register and be notified automatically when new properties come on the market that match their search criteria. Many users have pretty broad search parameters so we often times have a lot of emails going out each morning. All of these are opt-in with clear directions to turn off the emails. Seems simple and harmless enough but blacklists have had fun with us I think. The emails themselves don't normally end up being reported (I can't recall any time I've gotten a report from a spamlist and it's been related to these emails) but I believe they end up playing an important role in our problem. Read on.
So account 1 was changed. It involved making setting changes for 3 accounts on 2 different computers within Outlook. The setting changes and MX entry changes were done at the same time on the computers to keep everything consistent so we didn't lose any emails in the process. Pretty easy and slick.
Account 2 was a little more work but it was still pretty easy. Created the Google Apps account for them, setup all of the forwarders within the admin dashboard and we were ready to go. The MX record change didn't cause any problems because either emails were being forwarded by us or by Google. Either way, the end result is that the email is in the same place. We let one of the main people know what we were doing just so they knew (we emailed them after the change actually) but the change was easy.
Account 3 was a completely different story. The 25 email forwarders weren't much of a problem (just like Account 2) but getting 25 pretty computer illiterate people to change their Outlook settings was an event like none other. We made multiple meetings with the company about the change to ensure that they were comfortable with what we were going to be doing. About 12 hours before the planned changeover, I setup forwarders on our system to go to the temporary email addresses Google provides so emails could start going there (and so screens weren't blank when they changed their settings). On the day of the planned change, I spent almost the entire day in their office walking around to the different computers behind each user so they could log into their Windows network profile, launch the email program they use (was either Outlook Express or Outlook XP) and instruct them through the process of making the setting changes while I'm standing over their shoulder.
Long story short, that was about 3 weeks ago and the full MX record change didn't happen until a few days ago. There were a few users who weren't there that day and we've had to wait until there was a good time to walk them through the process.
So far....
The Good
The Bad
Quick, final note about the spam filtering. I'm still trying to learn how it works, how strict it is, etc. While we had the 3rd account (mentioned above) still forwarding using the temporary addresses, we popped onto a spamlist for a few hours which caused all email leaving our server to Gmail's servers to be marked as spam. It's good that they went through to their system and weren't totally rejected, but the spam folder isn't normally available via 3rd party POP3 clients so it's almost as bad as them being deleted and gone forever.
All 3 accounts mentioned above are fully pointed to Gmail now via their MX records. If it wasn't such a painful process of have their domain's DNS zones hosted elsewhere, I'd probably do that also but things seem to be going well so far. Now that most of our off-site email forwarders are off our system, I'm hoping we're done with these spamlists for a while. DNSstuff.com and I are becoming pretty good friends unfortunately.
Small side thought: I'm not convinced yet that it actually wasn't Google that got us listed with the spamlists this last time, but that's the name of the game with forwarders I guess. I'm hoping that they'll start seeing our daily email notifications as more relevant emails and not consider that excessive volume which would look like spam. Fingers crossed. I feel like I'm back in high school trying to please everybody so everyone will like me.
So far, we've transferred 3 "large" accounts over to Google Apps for email hosting (none really use the other features to my knowledge). The first account was my company's which only has 3 real addresses on it but it's high volume and it's important to me so I'm calling it "large". The second account was a domain entirely made up of forwarders to remote email accounts (about 40 in all). The third was a real estate company which is made up of about 25 POP accounts and 25 forwarders.
Before I get into that though, I guess a little background information is in order. Basically, spam lists don't seem to like us very much. On each real estate site we have, site users have the ability to register and be notified automatically when new properties come on the market that match their search criteria. Many users have pretty broad search parameters so we often times have a lot of emails going out each morning. All of these are opt-in with clear directions to turn off the emails. Seems simple and harmless enough but blacklists have had fun with us I think. The emails themselves don't normally end up being reported (I can't recall any time I've gotten a report from a spamlist and it's been related to these emails) but I believe they end up playing an important role in our problem. Read on.
So account 1 was changed. It involved making setting changes for 3 accounts on 2 different computers within Outlook. The setting changes and MX entry changes were done at the same time on the computers to keep everything consistent so we didn't lose any emails in the process. Pretty easy and slick.
Account 2 was a little more work but it was still pretty easy. Created the Google Apps account for them, setup all of the forwarders within the admin dashboard and we were ready to go. The MX record change didn't cause any problems because either emails were being forwarded by us or by Google. Either way, the end result is that the email is in the same place. We let one of the main people know what we were doing just so they knew (we emailed them after the change actually) but the change was easy.
Account 3 was a completely different story. The 25 email forwarders weren't much of a problem (just like Account 2) but getting 25 pretty computer illiterate people to change their Outlook settings was an event like none other. We made multiple meetings with the company about the change to ensure that they were comfortable with what we were going to be doing. About 12 hours before the planned changeover, I setup forwarders on our system to go to the temporary email addresses Google provides so emails could start going there (and so screens weren't blank when they changed their settings). On the day of the planned change, I spent almost the entire day in their office walking around to the different computers behind each user so they could log into their Windows network profile, launch the email program they use (was either Outlook Express or Outlook XP) and instruct them through the process of making the setting changes while I'm standing over their shoulder.
Long story short, that was about 3 weeks ago and the full MX record change didn't happen until a few days ago. There were a few users who weren't there that day and we've had to wait until there was a good time to walk them through the process.
So far....
The Good
- Solid solution. No downtime or access-related problems so far that I've noticed. Helps me sleep better
- Good spam filtering (more on this below). Filters both accounts and emails being forwarded on (we could too but we had to create a POP3 account for the forwarder to get our spam killing solution to zap the emails before sending them onto the remote destination and the local POP3 account would max it's quota and emails would start failing)
- Easy to use admin dashboard
- Integrated webmail and POP access. Very nice to have emails sent via Outlook appear in webmail also. The other webmail solutions were totally disconnected from POP clients
- Lots of space and growing. Up to almost 4,500MB at the time of writing this (and that's for the free account)
The Bad
- Minor "feature" is causing a lot of headaches. When you send yourself an email, it will appear as a new email within the webmail interface but it won't be downloaded via POP3. I think I understand the reason for doing this BUT when someone sends an email to themselves and it doesn't come back to them, it looks like there's a problem. If they could get rid of this feature and start allowing those emails to be available via POP3 access, that would make my life a lot easier. There is no good solution I've found to put a workaround in place for this that doesn't involve making changes to each individual user's settings.
- They're in control now instead of me. I've gotten a few questions about email delivery recently that I'd normally have no problem getting information about because I have access to all of our server logs but I'm a little in the dark when it comes to Gmail's system. Although this may seem like a big deal, it's not that bad because their solution is solid and it doesn't seem to suffer from weird glitches. I've eventually tracked down the issue of delivery with each of those reported problems and they've never been because of Gmail's system. I just wish I could punch in a few commands and find out what was going on myself, but I've found myself trusting their system to do what it's supposed to so I've focused my troubleshooting steps on everything else
- Advanced filtering and whitelist features are only available for the Google Apps paid accounts. Any person in a user's contact list will be automatically whitelisted (which is a good feature, by the way) but the advanced features within the paid service allow domain-wide settings instead of a per-account basis.
Quick, final note about the spam filtering. I'm still trying to learn how it works, how strict it is, etc. While we had the 3rd account (mentioned above) still forwarding using the temporary addresses, we popped onto a spamlist for a few hours which caused all email leaving our server to Gmail's servers to be marked as spam. It's good that they went through to their system and weren't totally rejected, but the spam folder isn't normally available via 3rd party POP3 clients so it's almost as bad as them being deleted and gone forever.
All 3 accounts mentioned above are fully pointed to Gmail now via their MX records. If it wasn't such a painful process of have their domain's DNS zones hosted elsewhere, I'd probably do that also but things seem to be going well so far. Now that most of our off-site email forwarders are off our system, I'm hoping we're done with these spamlists for a while. DNSstuff.com and I are becoming pretty good friends unfortunately.
Small side thought: I'm not convinced yet that it actually wasn't Google that got us listed with the spamlists this last time, but that's the name of the game with forwarders I guess. I'm hoping that they'll start seeing our daily email notifications as more relevant emails and not consider that excessive volume which would look like spam. Fingers crossed. I feel like I'm back in high school trying to please everybody so everyone will like me.
Wow, what a week.
One of the many things I do is administer the servers we have that run the web services we host. Those services include website functions, database hosting and of course email.
Website functions include running and maintaining Apache. Because we pay fees to other companies for software and such, we get automatic updates as part of the package. This is great because we, honestly, don't have to do much as far as this service goes.
Database hosting includes making sure MySQL is up and healthy. For the same reasons above, there isn't much you have to do here; however, backups are even more important for this type of data because it's always changing (where maybe your actual code might not be).
Email is a completely different story. Completely. One of the few services that a server performs that relies on everyone playing nice is email. The problem is that not everyone does. Any machine on the internet can be turned into a spam spewing nightmare for system administrators.
The main way that spam is combated is by using spam filters at the server level. Yes, there are some email clients like Outlook 2003 that have built-in junk mail filters but most spam is blocked by the server so that you don't know about it. On those servers, they use various rules for checking spam. Some might look for the word "viagra" or "free prescriptions" and block email based on that. Some might look at the "From:" and determine if it's a valid address. Some might do a combination of all of that plus thousands of other rules.
The most popular (or annoying depending on your job title) is probably spam blacklists. These are lists that mail servers use to help share information with other mail servers about spammers. Just because your mail server hasn't gotten spammed by a notorious spammer doesn't mean that you have to in order to know about them. When emails come in, their criteria is checked against worldwide lists and is sometimes blocked based on that.
Now for the purpose of this article: email forwarders. Almost everyone uses them. If you have a company email address and a personal email address, you often just want your mail in one place so you ask your IT guy to forward your company email to your personal inbox. So, when an email is sent to you, it:
- leaves your friend's computer and heads to their ISP
- leaves your friend's ISP headed for your employer's server
- is received by your employer's server and redirected to your ISP's server
- is received by your ISP's server and made available to you
Great. Easy enough. Like an address change at the post office, mail is simply redirected to a new address. Here's the problem:
- spam leaves spammer's computer headed to their ISP
- spam leaves spammer's ISP headed for your employer's server
- spam is received by your employer's server and redirected to your ISP's server
- spam is received by your ISP's server, seen as spam and blocked/deleted (you never see it)
Sounds harmless unless you realize what your ISP is doing. When they get a spam email, they check to see who sent it to them. In any other case (when not using a forwarder), it's probably a server from another country. Eventually, if they get enough bad email from that server, they'll stop accepting email from it altogether. When you use forwarders though, the email your ISP blocks looks like it came from your employer and they think your employer is the spammer.
Eventually, once enough ISPs block spam emails from your employer's server, they are reported to the spam blacklists I mentioned earlier. This causes many other servers that use those lists for spam checking to block your employer's emails. Bad news.
This is my headache this last week. It's not the first time it's happened (and in fact, happened twice this week).
There are 3 possible solutions we're looking at. If you can think of others, please leave a comment on this article:
1) Stop allowing forwarders altogether (or stop allowing forwarders that redirect to remote systems). This fixes the problem altogether.
2) Reconfigure our system to scan forwarded emails before they're sent off. This has proven to be very difficult to do and, since no spam filter is perfect, spam could still get through which could give us a bad name.
3) Use an outside email hosting service that allows forwarders. Fixes the problem altogether (or at least makes it someone else's problem).
I think we're heavily considering option #3 at the moment. More specifically, Google Apps. Leverage the power of Google's already established Gmail email service (and their spam filters) with the size and scale of Google to provide a pretty powerful and feature-rich email experience for our clients.
Good luck to any other system admins that are fighting this right now. It's a tough position to be in. Maybe we'd be better off if we hadn't started off offering forwarders to ours users but we did so here we are. Maybe offloading email to an established service like Google for everyone's email (regardless of forwarder use) would be better for everyone and remove us from this equation.
One of the many things I do is administer the servers we have that run the web services we host. Those services include website functions, database hosting and of course email.
Website functions include running and maintaining Apache. Because we pay fees to other companies for software and such, we get automatic updates as part of the package. This is great because we, honestly, don't have to do much as far as this service goes.
Database hosting includes making sure MySQL is up and healthy. For the same reasons above, there isn't much you have to do here; however, backups are even more important for this type of data because it's always changing (where maybe your actual code might not be).
Email is a completely different story. Completely. One of the few services that a server performs that relies on everyone playing nice is email. The problem is that not everyone does. Any machine on the internet can be turned into a spam spewing nightmare for system administrators.
The main way that spam is combated is by using spam filters at the server level. Yes, there are some email clients like Outlook 2003 that have built-in junk mail filters but most spam is blocked by the server so that you don't know about it. On those servers, they use various rules for checking spam. Some might look for the word "viagra" or "free prescriptions" and block email based on that. Some might look at the "From:" and determine if it's a valid address. Some might do a combination of all of that plus thousands of other rules.
The most popular (or annoying depending on your job title) is probably spam blacklists. These are lists that mail servers use to help share information with other mail servers about spammers. Just because your mail server hasn't gotten spammed by a notorious spammer doesn't mean that you have to in order to know about them. When emails come in, their criteria is checked against worldwide lists and is sometimes blocked based on that.
Now for the purpose of this article: email forwarders. Almost everyone uses them. If you have a company email address and a personal email address, you often just want your mail in one place so you ask your IT guy to forward your company email to your personal inbox. So, when an email is sent to you, it:
- leaves your friend's computer and heads to their ISP
- leaves your friend's ISP headed for your employer's server
- is received by your employer's server and redirected to your ISP's server
- is received by your ISP's server and made available to you
Great. Easy enough. Like an address change at the post office, mail is simply redirected to a new address. Here's the problem:
- spam leaves spammer's computer headed to their ISP
- spam leaves spammer's ISP headed for your employer's server
- spam is received by your employer's server and redirected to your ISP's server
- spam is received by your ISP's server, seen as spam and blocked/deleted (you never see it)
Sounds harmless unless you realize what your ISP is doing. When they get a spam email, they check to see who sent it to them. In any other case (when not using a forwarder), it's probably a server from another country. Eventually, if they get enough bad email from that server, they'll stop accepting email from it altogether. When you use forwarders though, the email your ISP blocks looks like it came from your employer and they think your employer is the spammer.
Eventually, once enough ISPs block spam emails from your employer's server, they are reported to the spam blacklists I mentioned earlier. This causes many other servers that use those lists for spam checking to block your employer's emails. Bad news.
This is my headache this last week. It's not the first time it's happened (and in fact, happened twice this week).
There are 3 possible solutions we're looking at. If you can think of others, please leave a comment on this article:
1) Stop allowing forwarders altogether (or stop allowing forwarders that redirect to remote systems). This fixes the problem altogether.
2) Reconfigure our system to scan forwarded emails before they're sent off. This has proven to be very difficult to do and, since no spam filter is perfect, spam could still get through which could give us a bad name.
3) Use an outside email hosting service that allows forwarders. Fixes the problem altogether (or at least makes it someone else's problem).
I think we're heavily considering option #3 at the moment. More specifically, Google Apps. Leverage the power of Google's already established Gmail email service (and their spam filters) with the size and scale of Google to provide a pretty powerful and feature-rich email experience for our clients.
Good luck to any other system admins that are fighting this right now. It's a tough position to be in. Maybe we'd be better off if we hadn't started off offering forwarders to ours users but we did so here we are. Maybe offloading email to an established service like Google for everyone's email (regardless of forwarder use) would be better for everyone and remove us from this equation.
