Tuesday, November 16, 2010

Successful migration from Echo on Blogger to Disqus on Blogger

As I wrote here: http://anarchangel.blogspot.com/2010/11/in-which-known-unethical-and-idiotic.html

JS-KIT has just upped the price of their already only marginally acceptable service (oh how I miss you Haloscan. Yes, you went down a lot, but otherwise you were simple, and you functioned well, with a good interface, and easy to use management tools) to the ridiculous rate of $120 a year, prompting me to finally get off my butt and find a way to replace them.

At the time JS-KIT forced me to migrate to Echo from Haloscan (a bit over a year after they bought Haloscan if I remember properly) there was no way to successfully export comments back to blogger, or to the other major commenting services, Disqus and Intense Debate. So I ponied up the cash they wanted (I think it was just $10) to move to Echo.

Well, as it happens, a couple months later, wordpress, Disqus, and Intense debate did come up with a comment importing engine for Haloscan; but not for the much more complex formatting and content of Echo comments.

I wasn't particularly happy with Echo, but I'd already paid the money, and my comments had migrated successfully, without any loss of content. Doing an import to one of the other services looked like it wasn't worth it.

In the mean time, Echo jacked up their price; and the other commenting systems (and wordpress the other major blog host) have all come up with ways to migrate comments from echo. Unfortunately, they looked rather complicated, and there was some risk of dataloss (especially in the wordperfect model).

So, I was presented with four options:
  1. Migrate entirely to wordpress: Something I don't want to do, that is potentially somewhat irritating, and which has the potential for data loss. To get both comments, and the blog, synchronized; I would need to separately export them both, then set up a local wordpress install and run a python script designed to merge and resync the data into something wordpress could work with. Then I would need to import that data into my local wordpress install, clean it up, and export it to my live wordpress instance.

  2. Migrate comments to intense debate: This also required an export, and running scripts on XML output; and again, carried a risk of data mangling.
  3. Migrate comments to Disqus: In theory, this required an export, rejiggering of the XML import filters, and a reimport; but it seemed to be significantly simpler and more reliable than the solutions offered by Wordpress and Intense Debate.
  4. Migrate back to blogger comments: This should have been easy, just requiring an upgrade to my blog template, then the removal of the Echo comment widget. More on this later.

All of these options required me to "upgrade" my blog template from classic .css to Bloggers "advanced layout" template; something I have resisted doing for four years. Unfortunately, all of the import and sync functions are dependent on being in "advanced layout" format, so staying with the old template just isn't an option anymore.

Initially I decided to try for option four, it being theoretically the simplest and most reliable.

Theoretically.

I of course had not counted on the incompetence of JS-Kit.

Echo comments are supposed to be continuously synchronized with Blogger comments, so that at any time, you can disable Echo, and bloggers comments will take over, without losing any data.

Theoretically.

Except that most of the time, for most users, it doesn't work. The Echo support forums are completely full of instances of people asking support to resync their comments etc...

On Sunday, I was one of them. Even after disabling, and theoretically forcing a resync, none of my data was put back into blogger.

This meant that under all the import functions I knew were working, I was going to lose my comments.


Late last night/Early this morning however, I decided to do a little more research into Disqus's custom comment import filters, and lo and behold I find they've gone and added a standard plugin to import the XML comment file that Echo exports.

So last night I signed up for, and installed Disqus, and starterd importing.

I am happy to report that it mostly works. I have lost some comments; but I'm pretty sure that's Echos fault, not Disqus, because I grepped the XML file for some strings from comments I  know went missing, and they aint there.

Also, I exported my comments file from Echo three different times, and got three different file sizes...

At any rate, there is a specific sequence you want to do this in to avoid difficulty:

  1. Disable comments in Blogger: First thing, shut off comments in Blogger, so while you're working, things don't get even more out of sync; and put up a post letting your readers know comments are going to be disabled, and that the template is going to change.  Also disable comment moderation and email notification.

  2. Export your comments in XML format from Echo: I recommend you export three times, to three different file names. If they are different sizes, you're going to want to import all three (Disqus won't create dupes)

  3. "Upgrade" your Blogger template to the advanced layout: Use Bloggers upgrade funtion to install a new layout, and then add the elements and widgets you want, INCLUDING the Echo install widget (which you need to install from the Echo dashboard). This will allow you to function with the new comment systems, including Echo as it is theoretically supposed to work, Intense Debate, and Disqus. You may need to wait a few hours, if your comments are actually synchronizing between Echo and Blogger, for them to all sync up. Once the syncing is complete, you can remove the Echo widget.

  4. Sign up for Disqus: While you are waiting for your comments to sync up, sign up for Disqus. It takes two seconds. If your comments are done syncing, or just don't work anyway, or you don't give a damn, you can temporarily reenable comments, and install Disqus on your blog (it's dead easy, with step by step instructions on the site). Configure your settings, making sure you disable new comment notification and moderation temporarily (otherwise as you import, your mailbox will be flooded with notification emails).

  5. Reconcile and Import your comments: Ok, at this point, if your comments are fully synced with blogger, or not synced at all, you're good. Go and clean out any spam comments (there WILL be spam comments. Bloggers comment API has a weaknesses that allows spammers to post spam comments directly into the post archive without actually hitting the form. They show up in the admin interface, even if you never saw them in Haloscan/Echo) and then run a comment import from Blogger into Disqus.

    At this point DO NOT enable blogger comment synchronization, or conflicts and duplicates might be created.

    If your comments are NOT fully synced, you need to decide whether to keep what IS there, or to delete it and just go with the XML exported from Echo. If you keep the blogger comments, you might create dupes, but you might be able to keep content that the XML export missed.

    When the blogger import completes, check and see if any comments are missing. There most likely will be. That's why we have the Echo export XML.

    Clean up the comments in the Blogger import, then run an import on the Echo export XML comments. DO NOT RUN THESE IMPORTS SIMULTANEOUSLY. It will created issues.

    If your multiple export files ended up different sizes or on review have different content, import all of them, one at a time. Disqus will not create dupes of these, as each posts unique identifier will be the same across all the files (it may not be between Blogger comments and Echo comments, though it SHOULD be).

    Even after the import is marked as complete, it isn't really. They've loaded all the data up, but it will be several hours before all comments are linked with their posts. I'm going on 16 hours as I write this, and I'm still only about 3/4 done. I'm guessing someone with 5+ years of comments like me is going to have to wait a full day or more. I'd guess that no matter how much comment data you have, if it's not done in 3 days, you've got all you're going to get.

  6. Clean up: Time to clean up your comments. The Disqus interface is pretty good for that. At this point you can decide to sync comments with blogger, or not. Personally, I say not. If you need to migrate again later, there are tools that can deal with Disqus format, a lot better than bloggers; and of course with Disqus you can force a comment resync at any time. By not syncing now, you don't have to worry about inconsistencies between the two. If you use email notification, now is the time to turn it back on. Then fix your template, and re-enable comments through blogger.
I'm a bit miffed that I still seem to have lost SOME comments; but by far the vast majority of my 5+ years of comments are alive and well, and in a format that is better and easier to use.

The only thing I don't like about Disqus, is that I can't make the comments go to a popup window; at least not as far as I can tell.