“Hey, where did all our traffic go?”

Monday, 23. November 2009

Or: “How to execute a flawless migration from one CMS to another and lose more than 98% of your traffic in the process.”

Just over one month ago I migrated our web site from Drupal to WordPress. The migration went flawlessly and everything except the comments transferred without problem. The only content that didn’t migrate was comments from the old site. They were more difficult to migrate cross-CMS than the rest of the content and we decided the benefit did not justify the effort. Following the migration I tested everything (or so I thought), both from inside and outside our network. All indicators were green and everything worked as planned.

GraphThen, a couple of days ago I looked at the site’s Google Analytics reports for the first time since the migration and I said “@#$&” and “$%&^” and “%&$#?@!” too. And I meant every word of it. Why did I suddenly break out in grawlixes? Because I saw the traffic graph on the right. Our web site traffic fell off of a cliff and never recovered after the migration. According to Google Analytics our traffic was down 98.17% from our historical average.

How did that happen? It happened because I failed to check that all of the permalinks matched the old site. I had spot-checked some and they were the same, and so I assumed they would all be the same. That was an enormous mistake because Drupal and WordPress generate permalinks from post titles in subtly different ways. For example, given a title containing the phrase “Ubuntu 9.04”, Drupal would generate a permalink containing “Ubuntu-9-04” and WordPress would generate a permalink containing “Ubuntu-904”. This meant that a large number of our archives had new permalinks that were now one character different from the original. In fact, all of our most popular permalinks were affected. Bookmarks, inbound links and search results were all getting a 404 page, and not a very good one at that.

Much of this traffic is lost forever. After correcting most of the permalinks traffic is up to approximately 20% of our pre-migration levels. I expect more of it to return as search engine crawlers clean up after the mess I caused. But it won’t all return and it will take months to rebuild to pre-migration levels. If our business depended on web site traffic, I would likely be updating my resumé now instead of writing this.

Lessons Learned

I learned these lessons the hard way so you don’t have to.

  • Compare the permalinks from the old and the new CMS. All of them.
  • Look at your web stats reports regularly.
  • Make sure you have a helpful 404 page. One that says “404: Page Not Found” and nothing else is not helpful.
  • Tag and categorize your content. The 404 page for the WordPress theme I’m using uses this information to help a visitor landing on the 404 page navigate to somewhere more useful.
  • Get a real-time report of 404 errors. There are several WordPress plugins that will help. I used the 404 Notifier plugin by Alex King. That plugin sends an email anytime a 404 error occurs, but there are other plugins available that serve a similar purpose. If I had this kind of real-time report during the migration, the problem would have been identified and corrected much sooner and the traffic impact would have been minimized.
  • Use a plugin that tries to locate the missing content. I’m using Smart 404 for WordPress by Michael Tyson. Smart 404 tries to find the missing page and redirects to the correct page. If it can’t locate the correct page then it tries send the visitor to a relevant category or tag. There are similar plugins that take other approaches, such as using a site map and a local Google search to locate either the missing content or relevant content.

Share '“Hey, where did all our traffic go?”' on Delicious Share '“Hey, where did all our traffic go?”' on Facebook Share '“Hey, where did all our traffic go?”' on Google Buzz Share '“Hey, where did all our traffic go?”' on Google Reader Share '“Hey, where did all our traffic go?”' on LinkedIn Share '“Hey, where did all our traffic go?”' on Email

One Response to ““Hey, where did all our traffic go?””

  1. David Hobbs says:

    Thanks for the useful post. Those kinds of drops definitely make your heart sink. You raise lots of good points about how to avoid these issues. I think your second and fifth bullets (regardless of what CMS is being used) are probably the biggest items. This will also catch any inadvertent issues in how statistics are being collected, in addition to problems with the site. One of the biggest advantages of watching the 404s carefully is that you can prioritize what redirects / etc are the most important to put in quickly, since you can react/fix to the urls that are generating the most 404s.