Of course you can have a look at your Google Analytics and Wordpress Stats to get a feeling how popular your blog is, but there are other great tools out there that you should use to optimize your content and analyse how popular your blog really is.

The first tool I would like to mention is popuri.us. You just enter your domain name there (one time with www and one time without, because results may vary) and popuri.us shows you several statistics. The two most important ones are your Google PageRank and your Alexa Rank. Google’s public PageRank is a figure between 0 and 10, with ten being the most relevant sites rated by Google. Pagerank for tobman.com is currently … well, 0.

BUT - I checked my popularity today with popuri.us and to my delight I saw that tobman.com for the first time in history has a Alexa Rank! Alexa is web service company that provides a toolbar for browsers. I haven’t used it myself yet, but apparently some million others do this and by doing so provide statistics what webpages they are surfing to. Alexa aggregates this data and calculates a Rank for each site. And now I’m proud to announce that tobman.com is among the TOP 10 Million webpages of the world. Great stuff. I think as soon as you have reached the TOP 100.000 things start to make fun, but that’s a looooong way to go I’m afraid.

Anyway, you can check your Alexa Rank and several other interesting statistics about your website directly here.

Blogging, Search Engine Optimization Alexa, Blog, PageRank, Popularity

A couple of days ago I wrote about how to get your blog indexed by Live Search & Yahoo! Search. While the process itself was quite straightforward the result is really disappointing. At the moment Live Search still hasn’t updated its index although it has downloaded my sitemap already a couple of days ago.

With Yahoo! the situation is a little bit more satisfying. Yahoo Slurp (their crawler) has accessed the site and some pages appear on the index already. The real problem is that I’m really bad ranked for some reason. When you search for tobman (my current Google Nr. 1 position) you find my blog on the 7th result page… That’s really bad. Even worse: if you search for other really high ranked posts (e.g. cakephp mac or cakephp breadcrumbs) the result is not the relevant post but just the general link to tobman.com. Of course this is not appealing to anybody searching for those terms. As a reason I currently get 0 traffic from Live Search and Yahoo! Search.

And then on the other hand there is Google. What a pleasure! My blog currently has 26 posts, 25 of them are indexed on Google! It was really interesting to see that after my Index fiasko Google gained “trust” from day to day and added more of my posts to its index. It now takes roughly one hour for a new post to appear on the Google search engine. I think this is really amazing and just shows from a webmaster’s point of view why Google is just so superior to Live Search & Yahoo! Search.

I will try to find out if there is anything that I can do about improving results with Live Search & Yahoo! Search and will let you know of course! If you have experience yourself please feel free to comment this post!

Search Engine Optimization Google, LiveSearch, Yahoo, Yahoo Slurp

So far I have been only working on getting my site being listed at Google. I had a look at my Google Analytics stats today and saw that 100% of the traffic generated through search engines comes from Google. So I asked myself - maybe there are other search engines out there that should link to my site as well? I personally never use a different search engine than Google so the first difficulty  was to find out, what other search engines I could have a look at.

Of course I knew that there was some search service offered by Microsoft and of course I knew that there was Yahoo out there. But here’s the funny thing: the result pages of Yahoo’s search site look so similiar to Google that in fact I thought they were using a rebranded Google service. But that’s not the case… So first I head a look at whether my site is listed in Yahoo or not. Shock! It doesn’t seem to be… or at least when I tried popular search terms that give me high ranks with Google my site never showed up at the front. Time for submitting my site to Yahoo!.

I clicked on their link where it says Submit Website and was overwhelmed by all the different options. Next surprise: you can pay for your site being listed at Yahoo! Of course I don’t want to do this, so I guess Submit Your Site for Free is the best alternative. I chose to submit a site feed (my SML sitemap) and had to sign up for a Yahoo ID… At least the sign up form was very web 2.0 with a lot of Ajax an fancy stuff.

Ok, once signed up and everything you get to the Yahoo Site Explorer, which is basically the same as Google’s Webmaster Tools just for the Yahoo search engine. To be honest I think the design is a little bit more professional than Google’s, but there is not as much functionality. Here is what I did so far:

• I verified my site (using a Meta-Tag)
• I added my sitemap as feed
• I checked the statistics: currently 0 pages from my blog are indexed by Yahoo
• I deleted some old blog entries using the Delete URL feature

Deleting sites is called “Site actions” and you are restricted to 25 actions per website. I don’t know whether just for the moment, or forever, but currently I only have 21 delete requests left. So much for Yahoo… I then moved on to try the search engine of Microsoft. First I had to find out that this is called Live Search nowadays. There is a link for webmasters on their site as well and it gets you to the Webmaster Center. Again, this is very similiar to Google and Yahoo, but much less functionality and the design is really outdated as well. Here is what I did at Live Search:

• I verified my site using the XML file method - they check the verification every time, so if you don’t want to have the annoying Meta Tag in your HTML code all the time I recommend using the XML file
• I added my sitemap

That’s it. Really nothing more that you can do. One interesting thing is the so called “Page Score”. I assume it’s something similiar to Google’s pagerank, but I ranked really high: 4 out of 5. I don’t know whether this is something to be happy about, or something that makes me wonder, how on earth MS calculates this score…

Search Engine Optimization Google, Live Search, Microsoft, Yahoo

If you have followed my blog lately, you will know that I had some problems with entries of my blog being dropped from the Google Index. This was several days back and I assumed it had to do with a lot of 404ers Googlebot was getting. I tried to fix this with re-submiting the Sitemap and requesting old content to be removed. This took some while, but eventually today it seemed to work. Well, at least partly.

The first removal request with the Google Webmaster Tools took around 2 days. The second request only 10 hours or so and my third request was processed within a couple of hours! I really don’t know why it took longer in the beginning and was super fast at the end, but now the number of pages on tobman.com listed in the Google index dropped from around 120 to 15! Great news! You can check this yourself using this search request:

site:www.tobman.com

Of course you should try to run this query with your domain as well. The goal is just to have relevant content being indexed. While it worked out to get rid of outdated content I still have the problem that not all of my sites are being indexed. It’s much better now than a couple of days back, but currently out of my 22 posts only 12 are indexed. At the Webmaster Tools it even says that 0 out of the 23 links submitted with the sitemap are indexed. It seems that it takes some time for this report to update (although I think I have never seen anything else than 0 so far).

I sat down before and tried to figure out what the reasons for not being indexed could be. I honestly could not find any regularity. I had a look at the number of onsite and offsite links and whether pictures are included in the post or not - this doesn’t seem to affect the indexing behaviour. The only thing that might the case that posts not indexed seem to be a little shorter than others. Maybe Google thinks their content is too short and therefore not worth being listed? I don’t know… I will try to relink to some of the “lost” posts, just as to my great post on why content is the key and see whether this changes anything. If not I will try to lengthen those posts as an alternative approach.

Just as a sidenote: this has been the fourth blog entry on search engine optimization in a row. I hope you find it useful… If not, don’t worry I have some new ideas for cakePHP posts and will try to write them as soon as possible!

Search Engine Optimization Google, Google Webmaster Tools, Index, Search

As you might have read I’m currently struggling with Google to get my current sites indexed and my outdated and deleted pages out of the index. Today some progress was made. First of all the first content removal request was processed! Hooray! I had to wait two days (Google indicated that it takes between 3-5 business days usually … so I’m really asking myself, what are they doing? Are they processing thos requests manually?) and the result: well the pages are still in the index. BUT it just might have been my fault, because I chose to remove the page /tag/ instead of the directory /tag/. The latter one should include all subdirectories, so I posted another removal request today.

But I have also done something else - I updated my robots.txt. It now looks like this:

User-Agent: *
Allow: /
Disallow: /blog/category/
Disallow: /blog/tag/
Disallow: /*?
Disallow: /blog/2008/

Basically the first line says that whoever is reading this (some bot usually) you better take note of what follows. In principle every directory on the host is allowed to be indexed, BUT don’t even try to index categories, tags and the 2008 archive! I also have excluded any page with a question mark - those pages are usually the ones from a search request. I’m doing all this to get rid of the outdated content that is still in the Google index and also to try to minimize the duplicate content, as this is supposedly not good for page ranks.

I have also modified the looks of the blog today a little bit. Tweaked the headers (now on the mainpage the blog name is <h1> and the titles of the entries are <h2> and on every single entry view the blog title is <h3> and the title of the entry is <h1>) and filled some titles with some keywords. We’ll see how all this works out in the coming days I suppose!

Blogging, Search Engine Optimization Google, Robots Exclusion Standard

I posted yesterday how some of my pages have disappeared from Googles index. I also mentioned how I tried to fix this issue (re-submitting sitemap, request broken links to be removed). Now, one day later the situation is as such:

This data from the Google Webmaster Tool actually means that currently not a single of my blog entries is listed in Google. And this in return explains one thing: why currently my site traffic is almost down to ZERO! I think this shows very impressively how dependet website and blog owners are of Google. If you’re not listed in the search engine nobody will find you! Of course this is especially true in the beginning of starting a blog where you don’t have any subscribers, other sites that link to your blog etc.

Now as you might imagine this situation is a little bit frustrating for me. But right now there is actually nothing I can do about it! The requests for content removal (all the broken links) is still pending, I double checked my robots.txt and meta information - all this seems to be correct. So right now the only thing I can do is to wait… I will keep you informed on how things evolve!

Blogging, Search Engine Optimization Google, Google Search, Google Webmaster Tool

Well, sometimes you have to learn it the hard way. Initially three days back I was really amazed by the huge increase of traffic I was getting after I had started blogging. I wrote new blog entries and new content and thought there should only be one way for the traffic to go: increase even more. Wrong! Yesterday and today have been far worst than the two days before, which I know thanks to Google Analytics and WordPress.com Stats. So kept myself asking what is going wrong here… And I just found out the answer: I literally pissed off Googlebot with broken links and as a consequece a whole bunch of webpages from my blog was dismissed from the search index.

All this happened because I wanted to optimize my blog for search engines and changed the URL format. Also I was playing around with linking to the blog entry id’s instead of the permalinks… bad idea! If you omit the domain part (absolute path) and work with relative paths the links will work fine from your mainsite of the blog, but won’t once you try to click on one of them from within a blog entry. So I recommend just linking to permalinks and don’t - I mean really NEVER - change your URL structure once it has been set up! My post about installing cakePHP on Mac became quite popular (for my standards of course…) and I’m still losing a lot of traffic because the initial URL all those sites are linking to is gone.

So what had happened? Googlebot started to crawl my site as usual but soon stumbled across a couple of 404 errors (page not found). It seems that at some point this just got too much (17 errors) and Google stopped crawling the site and several pages that still existed were dropped from the index (I checked by querying search terms, where I usually came up pretty high and also used the site: query)! I found this out using the great Google Webmaster Tools which showed me exactly where errors occured. After fixing all the broken links I also used those tools to - hopefully - repair this damage as quick as possible: I used the URL removal tool to mark several directories as outdated and also re-submitted my Sitemap with the working URLs. As I already posted a couple of days ago, Googlebot is fast, so I hope this will be fixed soon.

But as two major learnings from this I will take away: get your links right, have ZERO 404 errors on your blog & get your permalink structure right the first time and NEVER change it! I hope this is useful for some of you (I make the mistakes so you don’t have too…) and if you have other tips on this topic please share by commeting below!

Project WebMoney, Search Engine Optimization 404, error page, Google, Googlebot, indexing

Just until recently (before starting project WebMoney) I was a huge fan of Webalizer - an online log tool analysis tool written in C. Don’t get me wrong, I still am, but as it turns out my webhoster has been using the same webalizer version for the past 6,5 years. This is not the fault of my webhoster though, there just hasn’t been an update for webalizer until recently. But just the fact that I’m relying on a traffic and visitor analysis tool that is based on technology from several years back made me wonder and I decided to have a look at alternatives.

Of course I had heard of Google Analytics before. Especially this summer there were some reports about Google collecting data of almost 80% of the entire German web traffic and some well visited and famous German websites publicly declared not to use Google Analytics any more. Well, I’m not getting that much traffic yet and also I’m curious about what the guys at Google invented here again so I wanted to give it a try. The signup process at their page is quick and easy. You have to include some Javascript code before your <body> tag on every site (so if you are using a blog or CMS include this in your template files) to start tracking your visitors. Important note: Include the code in your error page template as well! When you have done this Google will check whether your code was insereted correctly and if so it says “Receiving Data” for your website.

When you click on the first report you’ll immediately notice: this isn’t Webalizer anymore - this is much powerful. The dashboard includes the most important reports from three areas. The first one is visitors: visitors to your website, page views, pages per visit, average time on site and two really cool metrics: bounce rate and new visits. The bounce rate tells you how many visitors left your website on the page that was their entry page, i.e. didn’t surf around on any link (which they should if your site is sticky). New visits tells you how much of your traffic is from existing, loyal visitors and how much new traffic you are driving to your website. Just a note: If you are unsure what any of those metrics mean just click “About this Report” on the left side bar and it gives you a short explanation.

The second area of reports is traffic sources: here you’ll find out where your visitors are coming from. Either from search engines, other links on websites or directly. With visitors from search engines the specific search engine and the keywords that were searched for are shown.

The third main area is “Content” which shows you what are the most visited pages on your website. It also has a cool feature called “Site Overlay” where you can view your website and Google shows you the percentage values for every link on your site. This tells you what the most clicked links are and which are not so important.

By the way - those reports I have mentioned are just the beginning. There are other cool features such as benchmarking with other sites, getting details on software and hardware (screen resolution) used for browsing and much more.

One last thing I would like to point out with Google Analytics is the possibilty to define Goals. Goals are certain actions that you want your website visitors to do - e.g. register, order something etc. You can define goals by some meta information or by the URL of the goal page, i.e. that is the page your visitor ends at when he reached the goal. You can also define a funnel path, that allows you to monitor certain steps a user has to take in order to reach a goal (e.g. register page 1, 2, 3 etc.). Goals are excellent to measure your conversion rates and see how your website performs with getting users where you want them to be.

If you don’t want to use just another Google service or don’t need all this wealth of reports there is another alternative: WordPress.com Stats. You can use this if you have a WordPress blog for free. It loads an image (not the small smily on the bottom of this page…) and generates some statistics out of this. It is much simpler than Google Analytics but the basic functionality: tracking the number of visitors, where the come from and what the view is there.

Blogging, Search Engine Optimization Google Analytics, Web traffic

My blog in this form is now a little bit over 40 hours old and I already made several mistakes with regards to our friends: the search engines. There are several good articles and presentations that I would recommend for reading: try this, this and this. So here are some of the basic rules you should follow when creating a blog:

1. Keep the link to the articles simple and nice. With Wordpress this is called Permalink and you can choose different formats. There is only one format that really makes sense: /%postname%/ This is just the title of your post (no dates, categories etc. needed). Important: Once your blog has been online for a certain while never change the link format again!
2. Get your titles optimized! The standard titles of most blogs are something like “My new blog >> Article title”. This is just not good when looking at your article as the result of a search request on Google or some other search engine. If you are using Wordpress you can use this plugin to change the way titles are presented and even specify unique titles for each blog entry.
3. Change the main title of your blog… “Tobman’s Blog” doesn’t tell Googlebot or other spiders much about the content to expect. When you go to my homepage and have a look at the title you will see that I have changed it in order to more clear on what to expect on this blog. You can use the plugin mentioned above for this task as well.
4. Get a Sitemap and submit it to Google! I used this plugin to create an XML sitemap and also to automatically submit the sitemap to Google and MSN.

As a sidenote: I have not followed those rule from the beginning - unfortunately. I thought to have content first is more important and then do the search engine optimization. Well, I would say this was a mistake, as I had to change the paths to my blog entries and this will definitely lead to some 404 errors…

Another thing that really annoyed me was the fact that I had linked from one of my blog entries to some others several times - all those links were broken after changing the permalink layout of course. So I was wondering - is there a better way to link within blog entries? Well, no not really. As described in the docs you can either use peramlinks or linking to the id of the blog entries. When you link to the id’s it does not matter when the permalinks change, but it’s not very pretty:

<a href="index.php?p=123">Post Title</a>

And also you have to look up the id number of the post… So basically I would suggest sticking to permalinks and just not changing them once their structure is set! But if anyone knows a more elegant solution (e.g. a plugin) I’d be really interested to know!

Blogging, Search Engine Optimization Permalink, Search engine optimization, WordPress

I just tested another nice feature of Google I wasn’t aware of yet: Google Webmaster Tools. This is a set of tools developed for us webmasters to monitor and analyse all kind of stuff in relation to the Google search engine. Most relevant features I think are:

• See with what search terms your website appears and on what rank
• See what terms Googleboot thinks are relevant at your site
• Check errors that occured while indexing your site

It’s free to sign up and you can just enter the domains that you own. To access the full features you have to verify that you are the owner of the domains by uploading a randomly named HTML file and have Google download it (beware your .htaccess which might restrict access to it!).

Fore more info check out this nice article: Grow your Studio with Google Webmaster Tools.

If you like the Google Webmaster Tools you should definitely have a look at Google Analytics as well. It’s another great tool for webmasters that can be used for free and that allows you to analyse the traffic that you get to your webpage. I have written a little summary on how to use it and it’s main features, which you can view here.

Search Engine Optimization Google Search, Google Webmaster Tools