“Hey, where did all our traffic go?”

Monday, 23. November 2009

Or: “How to execute a flawless migration from one CMS to another and lose more than 98% of your traffic in the process.”

Just over one month ago I migrated our web site from Drupal to WordPress. The migration went flawlessly and everything except the comments transferred without problem. The only content that didn’t migrate was comments from the old site. They were more difficult to migrate cross-CMS than the rest of the content and we decided the benefit did not justify the effort. Following the migration I tested everything (or so I thought), both from inside and outside our network. All indicators were green and everything worked as planned.

GraphThen, a couple of days ago I looked at the site’s Google Analytics reports for the first time since the migration and I said “@#$&” and “$%&^” and “%&$#?@!” too. And I meant every word of it. Why did I suddenly break out in grawlixes? Because I saw the traffic graph on the right. Our web site traffic fell off of a cliff and never recovered after the migration. According to Google Analytics our traffic was down 98.17% from our historical average.

How did that happen? It happened because I failed to check that all of the permalinks matched the old site. I had spot-checked some and they were the same, and so I assumed they would all be the same. That was an enormous mistake because Drupal and WordPress generate permalinks from post titles in subtly different ways. For example, given a title containing the phrase “Ubuntu 9.04”, Drupal would generate a permalink containing “Ubuntu-9-04” and WordPress would generate a permalink containing “Ubuntu-904”. This meant that a large number of our archives had new permalinks that were now one character different from the original. In fact, all of our most popular permalinks were affected. Bookmarks, inbound links and search results were all getting a 404 page, and not a very good one at that.

Much of this traffic is lost forever. After correcting most of the permalinks traffic is up to approximately 20% of our pre-migration levels. I expect more of it to return as search engine crawlers clean up after the mess I caused. But it won’t all return and it will take months to rebuild to pre-migration levels. If our business depended on web site traffic, I would likely be updating my resumé now instead of writing this.

Lessons Learned

I learned these lessons the hard way so you don’t have to.

  • Compare the permalinks from the old and the new CMS. All of them.
  • Look at your web stats reports regularly.
  • Make sure you have a helpful 404 page. One that says “404: Page Not Found” and nothing else is not helpful.
  • Tag and categorize your content. The 404 page for the WordPress theme I’m using uses this information to help a visitor landing on the 404 page navigate to somewhere more useful.
  • Get a real-time report of 404 errors. There are several WordPress plugins that will help. I used the 404 Notifier plugin by Alex King. That plugin sends an email anytime a 404 error occurs, but there are other plugins available that serve a similar purpose. If I had this kind of real-time report during the migration, the problem would have been identified and corrected much sooner and the traffic impact would have been minimized.
  • Use a plugin that tries to locate the missing content. I’m using Smart 404 for WordPress by Michael Tyson. Smart 404 tries to find the missing page and redirects to the correct page. If it can’t locate the correct page then it tries send the visitor to a relevant category or tag. There are similar plugins that take other approaches, such as using a site map and a local Google search to locate either the missing content or relevant content.

Share '“Hey, where did all our traffic go?”' on Delicious Share '“Hey, where did all our traffic go?”' on Facebook Share '“Hey, where did all our traffic go?”' on Google Buzz Share '“Hey, where did all our traffic go?”' on Google Reader Share '“Hey, where did all our traffic go?”' on LinkedIn Share '“Hey, where did all our traffic go?”' on Twitter Share '“Hey, where did all our traffic go?”' on Email

Goodby Drupal, Hello WordPress

Sunday, 18. October 2009

As soon as the DNS changes propagate this blog will be hosted on WordPress at bluehost.com. I still like Drupal and I’ll probably keep current on it and continue to use it for some things, but just for blogging – especially as infrequently as I blog – the overhead for managing Drupal is too much.

Share 'Goodby Drupal, Hello WordPress' on Delicious Share 'Goodby Drupal, Hello WordPress' on Facebook Share 'Goodby Drupal, Hello WordPress' on Google Buzz Share 'Goodby Drupal, Hello WordPress' on Google Reader Share 'Goodby Drupal, Hello WordPress' on LinkedIn Share 'Goodby Drupal, Hello WordPress' on Twitter Share 'Goodby Drupal, Hello WordPress' on Email

Drupal 6.10

Saturday, 14. March 2009

I skipped Drupal 6.9 because I was too busy to upgrade. I had a few minutes free this morning so I upgraded to Drupal 6.10 which was released February 25th, 2009. The upgrade was incident free but the blog API module still returns “1” in response to an RSD query and expects “blog” when posting. I've just about given up hope that anyone will fix it and have resigned myself to patching that module in every new release. Daniel Jalkut of Red Sweater Software opened an issue report on this bug over a year ago.

Share 'Drupal 6.10' on Delicious Share 'Drupal 6.10' on Facebook Share 'Drupal 6.10' on Google Buzz Share 'Drupal 6.10' on Google Reader Share 'Drupal 6.10' on LinkedIn Share 'Drupal 6.10' on Twitter Share 'Drupal 6.10' on Email

Drupal 6.8

Thursday, 18. December 2008

Drupal 6.8 was released December 11th, one day after Drupal 6.7 was released. The update was incident free even though I had skipped 6.7. The blog API module still returns “1” in response to an RSD query but expects “blog” when posting. I downloaded the development snapshot of Drupal 7 to check out the forthcoming changes to blogapi.module. There are a few but they are minor and mostly cosmetic and the RSD query response is still broken.

Share 'Drupal 6.8' on Delicious Share 'Drupal 6.8' on Facebook Share 'Drupal 6.8' on Google Buzz Share 'Drupal 6.8' on Google Reader Share 'Drupal 6.8' on LinkedIn Share 'Drupal 6.8' on Twitter Share 'Drupal 6.8' on Email

Comments RSS Feed

Friday, 5. December 2008

I've been getting a lot of comments on this post and I needed a better way to be notified of comments that I need to respond to than by repeatedly visiting the site myself. A search on Drupal's web site turned up the Comment RSS module. It allows you to subscribe to an RSS feed of the comments on any individual post. Just click on the post you are interested it and look for the RSS feed icon in your browser's nav bar or at the very bottom of the page.

Share 'Comments RSS Feed' on Delicious Share 'Comments RSS Feed' on Facebook Share 'Comments RSS Feed' on Google Buzz Share 'Comments RSS Feed' on Google Reader Share 'Comments RSS Feed' on LinkedIn Share 'Comments RSS Feed' on Twitter Share 'Comments RSS Feed' on Email

Drupal 6.5

Tuesday, 14. October 2008

Drupal 6.5 was released October 8th and I realized that I had missed a couple of releases. I installed the update and everything worked flawlessly.


Update: The blog API module still has an impedance mismatch between RSD and posting. It hands out “1” in response to an RSD query but expects “blog” when posting. So once again I had to re-apply the RSD patch for the Blog API module.

Share 'Drupal 6.5' on Delicious Share 'Drupal 6.5' on Facebook Share 'Drupal 6.5' on Google Buzz Share 'Drupal 6.5' on Google Reader Share 'Drupal 6.5' on LinkedIn Share 'Drupal 6.5' on Twitter Share 'Drupal 6.5' on Email

Drupal 6.2

Wednesday, 9. April 2008

Drupal 6.2 was released today. It addresses a security vulnerability that affects Drupal 6.x sites that have multiple users, as well as a variety of other bug fixes. I upgraded from 6.1 to 6.2 without difficulty. However, I did have to re-apply the RSD patch for the Blog API module.

Share 'Drupal 6.2' on Delicious Share 'Drupal 6.2' on Facebook Share 'Drupal 6.2' on Google Buzz Share 'Drupal 6.2' on Google Reader Share 'Drupal 6.2' on LinkedIn Share 'Drupal 6.2' on Twitter Share 'Drupal 6.2' on Email

Blogging to Drupal from Flickr

Wednesday, 9. April 2008


Drupal Modules as of 11/9/07

Originally uploaded by kentbye

If your blog is powered by Drupal you can blog images and videos from Flickr. This blog post was created with Flickr. First I found an image that was relevant, interesting and licensed under Creative Commons. I decided to use the image to the right which displays a graphical snapshot of the available Drupal modules. Then I clicked the BlogThis button that appears above images along with the more familiar controls. That button sent me to a Flickr blog form where I typed the blog title and content. Then I posted the entry directly to this blog. (Once I had posted it I opened it with MarsEdit to polish it and add some additional images.)

If you want to try blogging from Flickr you need to make sure you have both the Blog and Blog API modules enabled and configured in Drupal. You can see what modules are enabled and enable or disable additional modules at Home

Share 'Blogging to Drupal from Flickr' on Delicious Share 'Blogging to Drupal from Flickr' on Facebook Share 'Blogging to Drupal from Flickr' on Google Buzz Share 'Blogging to Drupal from Flickr' on Google Reader Share 'Blogging to Drupal from Flickr' on LinkedIn Share 'Blogging to Drupal from Flickr' on Twitter Share 'Blogging to Drupal from Flickr' on Email

Drupal 6.0 RC4 and MarsEdit

Saturday, 9. February 2008

Drupal 6.0 RC4 was released yesterday and I downloaded it and took it for a spin. The good news is that the problems I reported here and here have been fixed.

The bad news is that after installing it and running the update I could not post a new blog entry with MarsEdit 2. When I tried to post a test message I received this error dialog:

PostError

The problem is related to the “Blog ID” that is configured in MarsEdit's blog settings and the way Drupal handles that blog ID:

BlogId_1

When MarsEdit auto-detects the blog ID it gets a value of “1”. For Drupal 6.0 RC3 and prior the blogapi.module was mapping the blog ID to a content type. A blog ID of “1” mapped to a content type of “blog”. Starting with RC4 that mapping was removed. RC4 treats the blog ID and the content type as synonomous. This was causing the function _blogapi_validate_blogid to fail because it was trying to validate that the current user had permissions to publish a content type of “1” but the content type should actually be “blog” instead.

The fix for this is to change the blog ID in MarsEdit to “blog” as shown here:

BlogId_blog

Changing the blog ID to “blog” allows posting and does not appear to have any negative side affects.


Update: Apparently the change was intentional in Drupal and the issue persists in versions 6.0 and 6.1. All of the functions in blogapi.module that need a blog ID are expecting a blog ID of “blog” now. Daniel Jalkut of Red Sweater Software pointed out the the incorrect blog ID is being returned by Drupal's RSD function. This is causing any blogging software that uses RSD for auto-discovery to get the wrong blog ID which in turn causes the problems described earlier in this post. The blog ID is hardcoded to “1” in blogapi.modules's RSD function. I submitted a patch to change the blog ID returned by the RSD function in blogapi.module to “blog” so that RSD auto-discovery will work correctly with MarsEdit and hopefully other blogging clients as well. You can also view and download the patch here.

Share 'Drupal 6.0 RC4 and MarsEdit' on Delicious Share 'Drupal 6.0 RC4 and MarsEdit' on Facebook Share 'Drupal 6.0 RC4 and MarsEdit' on Google Buzz Share 'Drupal 6.0 RC4 and MarsEdit' on Google Reader Share 'Drupal 6.0 RC4 and MarsEdit' on LinkedIn Share 'Drupal 6.0 RC4 and MarsEdit' on Twitter Share 'Drupal 6.0 RC4 and MarsEdit' on Email

Drupal 6.0 RC3 and MarsEdit 2

Tuesday, 5. February 2008

Drupal 6.0 RC3 was released a few days ago. I was hoping that it fixed the issue I wrote about in this post but it doesn't. I spent some more time working on the problem and this time I tracked it down to the node_submit function in modules/node/node.module. When node_submit is called from modules/blogapi/blogapi.module then $node->teaser_include is not set. When $node->teaser_include is not set the node_submit function cuts the teaser from the body of the post and that causes the problems i described here. When a new post is created from Drupal's create content form $node->teaser_include is set and everything works the way it should.

I haven't managed to track down why $node->teaser_include isn't getting set when posting with modules/blogapi/blogapi.module. But there is a simple workaround, set $node->teaser_include before every call to node_submit in modules/blogapi/blogapi.module using this diff patch.

Share 'Drupal 6.0 RC3 and MarsEdit 2' on Delicious Share 'Drupal 6.0 RC3 and MarsEdit 2' on Facebook Share 'Drupal 6.0 RC3 and MarsEdit 2' on Google Buzz Share 'Drupal 6.0 RC3 and MarsEdit 2' on Google Reader Share 'Drupal 6.0 RC3 and MarsEdit 2' on LinkedIn Share 'Drupal 6.0 RC3 and MarsEdit 2' on Twitter Share 'Drupal 6.0 RC3 and MarsEdit 2' on Email