After the May Day update many websites reported a significant loss of traffic, while others have seen improved traffic from search engines. After all, Google’s Search index resembles an ecosystem where one site’s loss is another one’s gain.

If you are using Google Analytics there is an easy way to check if and how this update has affected your performance in Google. In order to segment and analyze your long tail traffic you need to set up Advanced Segments using Regular Expressions. The nifty thing with Advanced Segments is that they allow you to analyze historical data, something which is not possible with filters, and to easily share them with others. Below I  have created 4 different segments which will hopefully help you get started with analyzing your long tail traffic. Simply click on any of the links to add them to your own Google Analytics profiles.

>> segment single-keyword queries

>> segment multi-keyword queries (>1)

>> segment two-keyword queries

>> segment three-keyword queries

For those of you interested, here is the basic regular expression used in the examples above. When applied with the Keyword dimension it filters in visits where exactly two keywords were used.

^(\S+)\s(\S+)$

With one of these Advanced Segments activated, you can for instance search for any patterns in your traffic from Google. You can find this report under [Traffic Sources > Search Engines > google].

I would love to hear your feedback, as well as any findings related to this Google update. Happy analyzing! :)

Photo credit: jean-fabien on Flickr

Annotations now seem to be rolled out to all users and here are 8 suggestions on how you can use it in your SEO projects.

In general you should make a habit of creating a note in Google Analytics for the following reasons:

- Technical adjustments including server updates and migrations, change of hosting provider, changes in the code (scripts, CSS, etc), database updates and modifications, etc.

- Structural modifications such as changes to your H-tags, URL structure, meta-robot tags, robots.txt, sitemaps submissions, etc.

- Editorial updates including copywriting, experiments with call to action, page titles, link anchor texts, etc.

- Campaign launches, both paid and organic, as well as offline campaigns. Changes in for instance AdWords budget should also be logged.

- PR work including articles written about you, your company or product, or related stories which could indirectly stimulate interest in your content.

- SEO related announcements including changes in search algorithms or introduction of new features, for instance launch of Google Caffeine, Personalized Search, and Real-Time Search.

- New inbound links to track any positive effect on your rankings and traffic.

- Seasonal trends, for instance start of the summer holiday season, related seasonal keyword trends such as ‘winter tires’ and ‘soccer world cup’, related behavioural trends resulting in higher search volumes in specific keyword clusters, e.g. health and fitness queries after the Christmas holiday season, etc

Feel free to add more suggestions in the comments section. Here is a video explaining how you can create an annotation in Google Analytics:

The principles of the Social Search feature are very simple. As a user I can create a Google Profile and link it with my various public accounts on for instance Twitter, YouTube, and Flickr. Google then uses this information, plus other clues such as my Gmail contacts or blogs in my Google Reader, to generate a social graph of my direct connections, friends of friends and so on. Next time I perform a search, the search algorithms look through the public content published in my social graph to see if there is something relevant available.

Here is an example of how it can look like: when I for instance search for [301 redirect], I get two very relevant results from my social circle blended with other results. One is an excellent blogpost from my friend Jesper Åström about how to implement a .htaccess 301 redirect, and a second blogpost from my former colleague at Google John Muller about 301 redirection issues on Twitter.

So how did Google found these two results? That’s simply because I have linked my Twitter profile from my public Google Profile. Google’s algorithms can then see that I’m following both of these guys on Twitter, and that both have a link to their blogs in their Twitter profiles. The type of content which might pop up in Social Search are for instance blogposts, tweets and status updates on other microblogging platforms, reviews, public Picasa or Flickr images,  and other activity from your friends on places which are linked to via their profiles.

To start using Google Social Search, you need to first create your own Google Profile and add links to your public profile on Twitter, FriendFeed, YouTube, LinkedIn, Flickr, etc. Google’s algorithms will then use these links to outline your relationship graph and to identify relevant content from your online neighbourhood next time you make a search.

If you are already using this feature, head over to this page to get an overview of your current social circle as Google sees it. More details about Social Search here.

(Photo credit: laurelpapworth.com, personalizemedia.com)

Update February 4: Following this post and my email conversations with the folks at Digg, they corrected their robots.txt file on January 29.

I was recently looking through some of the major tech news sites around the web and studying their usage of the robots.txt protocol. I came across Digg.com and found a potential error in their robots file which is costing them a lot of lost search engine traffic, as well as revenue from advertising. Since it looks like an honest mistake, I thought I write about it so that you won’t make the same mistake. For those of you not familiar with robots.txt, it is a way for webmasters to give instructions to search engine crawlers (also called robots and spiders) about what they are allowed to crawl, and which areas of the website should not be accessed. You can read more about the robots.txt protocol here.

To come back to Digg.com, I found a mistake among the instructions which causes a large part of the website (the Gadgets directory) to become inaccessible to search engines. If you take a look at their robots.txt file, you can find the following instruction:

Disallow: /gadget

If you are a SEO geek (like me) you have probably noticed the open ending to this line, and hence the wild card effect it causes in practice. What this line basically says is to block access to all content which have ‘digg.com/gadget’ as part of their URL. Note that this is not the same as exclusively blocking the “gadget” directory, but is far more inclusive. In practice, this instruction also covers Digg’s Gadgets directory, ‘digg.com/gadgets’ (note the ending ‘s’), and hence the problem! The correct implementation includes a forward slash ending the line in order to avoid the wild card effect:

Disallow: /gadget/

Digg has a large amount of pages in its index, for instance a Google site search shows close to 290 thousands pages in the ‘digg.com/apple’ directory alone.  We can therefore only speculate exactly how much search engine traffic Digg.com is shutting out by accidentally blocking the /gadgets/ directory (Yahoo Site Explorer shows 108 thousands pages in /gadgets/). This could be done intentionally, but it seems more likely to be an honest mistake (it’s one sinlge “/” missing after all). According to Compete.com, Digg has around 40mil monthly unique visitors, so I can only speculate around how much money such a small mistake has cost them in advertising revenues during this period.

Finally, the lessons we learn from this are to:

1) generate easy to read robots.txt files so it’s easier to spot such mistakes. There are online tools which help constructing and deciphering robots.txt files.

2) always use web analytics tools and segment your traffic so it’s clear to you what is going on across the various parts of your site.

3) be afraid of the wildcard effect, in fact just completely avoid anything with the word “wild” in it :)

What are other important things to think about when creating a robots.txt file?