Well, sometimes you have to learn it the hard way. Initially three days back I was really amazed by the huge increase of traffic I was getting after I had started blogging. I wrote new blog entries and new content and thought there should only be one way for the traffic to go: increase even more. Wrong! Yesterday and today have been far worst than the two days before, which I know thanks to Google Analytics and WordPress.com Stats. So kept myself asking what is going wrong here… And I just found out the answer: I literally pissed off Googlebot with broken links and as a consequece a whole bunch of webpages from my blog was dismissed from the search index.

All this happened because I wanted to optimize my blog for search engines and changed the URL format. Also I was playing around with linking to the blog entry id’s instead of the permalinks… bad idea! If you omit the domain part (absolute path) and work with relative paths the links will work fine from your mainsite of the blog, but won’t once you try to click on one of them from within a blog entry. So I recommend just linking to permalinks and don’t - I mean really NEVER - change your URL structure once it has been set up! My post about installing cakePHP on Mac became quite popular (for my standards of course…) and I’m still losing a lot of traffic because the initial URL all those sites are linking to is gone.

So what had happened? Googlebot started to crawl my site as usual but soon stumbled across a couple of 404 errors (page not found). It seems that at some point this just got too much (17 errors) and Google stopped crawling the site and several pages that still existed were dropped from the index (I checked by querying search terms, where I usually came up pretty high and also used the site: query)! I found this out using the great Google Webmaster Tools which showed me exactly where errors occured. After fixing all the broken links I also used those tools to - hopefully - repair this damage as quick as possible: I used the URL removal tool to mark several directories as outdated and also re-submitted my Sitemap with the working URLs. As I already posted a couple of days ago, Googlebot is fast, so I hope this will be fixed soon.

But as two major learnings from this I will take away: get your links right, have ZERO 404 errors on your blog & get your permalink structure right the first time and NEVER change it! I hope this is useful for some of you (I make the mistakes so you don’t have too…) and if you have other tips on this topic please share by commeting below!

Project WebMoney, Search Engine Optimization 404, error page, Google, Googlebot, indexing

Ok, with all that baking, coding and blogging yesterday I almost forgot about Project WebMoney. Well only almost… Of course I had to check the traffic statistics for my four websites and here are the results:

tobman.com: PR 0, about 120 visitors/day +1200 %

django.at: PR 3, about 20 visitors/day +/-0%

videopodcast.tv: PR 0, about 5 visitors/day +25%

advertbreak.com: PR 0, about 5 visitors/day +25%

If you compare this to the figures just two days ago you will notice one major difference: tobman.com is going through the roof! Two things that are remarkable: First Google’s webcrawler Googlebot is really fast - those new blog entries have been indexed within a couple of hours! How do I know? Because from the referring information of the logs I can already see that people were looking for things such as “setting up cakePHP on Mac OS”. And that article didn’t even exist 24 hours ago… And that’s second: it actually seems that the content I have been providing through my blogging so far seems to be relevant to some people out there. Great. Really great.

What also can be seen that the PageRank hasn’t changed so far on any of the sites. This is not surprising because the public Toolbar PageRank (the one that we can see) is only updated about every 3 months or so. But we see some changes in the traffic of videopodcast.tv & advertbreak.com - although nothing impressive it looks like that some of the readers of my blog actually visited those sites to see what’s on there currently.

The whole project WebMoney hasn’t been started more than 35 hours ago and to be honest those interim results is much more than I expected! Let’s continue the work and see what happens!

Project WebMoney Googlebot, PageRank