Multiple XML sitemaps can be used to help detect how the indexing of the site is progressing or if there are any problems. For example if you have 20,000 pages in the site and only 15,000 are indexed it’s hard to find out which 5,000 pages are not indexed by the search engine.

Google Webmaster Tools feeds back on this information saying how many pages have been indexed from each XML sitemap, best of all you can submit multiple sitemaps to the search engines.

Breaking the sitemap down can pinpoint which sections of your site are not getting indexed. So lets break that 20,000 pages down.

So we can see that the main problem is with the products section not getting indexed with only 44% of pages indexed. This now means you can start to find out the reasons why your product pages are not being indexed.

That then opens a new can of worms as it might be due to duplicate content, poor coding, internal linking, server issues etc. The good point is that you now know what area is the problem.

To link the XML sitemaps together you should have one sitemap called sitemapindex.xml that links to the other sitemaps using the xml protocol. Submit all the sitemaps to Google Webmaster Tools and list all the sitemaps in the robots.txt. E.g.

Sitemap: http://www.domain.com/sitemapindex.xml
Sitemap: http://www.domain.com/categories.xml
Sitemap: http://www.domain.com/sub-categories.xml
Sitemap: http://www.domain.com/prodcuts.xml
Sitemap: http://www.domain.com/blog.xml

So if you think your site has an indexing problem then split your site down logically into different XML sitemaps. It’s not going to solve your problem but will point you in the right direction.

If you’re an avid reader of any UK SEO blogs you should have noticed posts about the recent shift with foreign (predominantly American) sites, outranking many UK sites on certain search terms. The problem has been seen under a wide selection of search phrases from many SEO companies / individuals in the UK. There have also been some indicated of the problem reserved with UK sites ranking well in Australia search results.

So what have we seen?

- 1st / 2nd page can seem unaffected with no movement at all.
- Around 10-30 sites rank from the 2nd / 3rd page thus pushing rankings 2 to 3 pages down.
- Domains are .com and hosting in foreign country or have a country specific TLD e.g. ww.domain.be

What should you do?

- Continue to add relevant unique content to your site be that news or a blog
- Continue to build links to the site, don’t increase this rate rapidly.
- Read SEO blogs to track the changes.
- Clean up any errors showing on Google Webmaster Tools, doesn’t have any effect just something to take your mind of the problem.
- Don’t panic and sit this problem out, Google should* reverse this problem.
* Nothing is guaranteed but with so much talk about this problem in the UK Google has to rectify this problem.

The issue seen by many UK SEO companies is that when Google makes a change such as the one seen you can’t really do much to rectify it. Clients will be annoyed about this stance but you can only sit it out and wait for the Search Engine to update, explain the issue to clients and keep them up to date. Most importantly you can’t really complain about Google, for all we know they could shut down tomorrow and there is nothing we can do about it!

An Idea

I do have one idea that could help thanks to Bing. Collective pressure from UK SEO companies on Google to either publically acknowledge the problem or get a fix in place. The only way this could succeed is by getting the issue in the public domain, on the BBC, ITV, Sky News etc. This then forces Google into doing something to prevent losing users to Bing, which is actually succeeding in being decent alternative to Google for UK public.

If you want to read up more on this or need proof to see that this isn’t an isolated case then here are 11 articles on the problem.

Google American Results Conspiracy – Fresh Egg

US websites ranking in Google.co.uk an example commercial laundry equipment – A UK SEO

Google uk mixing in us based queries – SEOptimise

Big Geo Problems Still Exist in Google UK Serps – PPC Blog

Google UK Serps Com On Down – North South Media

Have you noticed Googles shift from UK to US? – Argo IBC

The mystery of the UK SERPS – updated – Search Cow Boys

Will the Real Google Search Results Please Stand Up – SEO Design Solutions

Is Google’s Geo Targeting Off In Web Search? – Search Engine Round Table

UK Ranking Problems – UK Business Forums

Poor State of UK search today – Andrew Girdwood – Big Mouth Media

If you have any more, add them into a comment below and I’ll add them to the above list. Do you think collective pressure would get a responce from Google?

There has been lots of chatter about a recent shift in ranking by Google with US based sites ranking in Google.co.uk, seoptimise and Search Round Table reported in articles and there was plenty of tweets about the movement. It’s caused a few problems with plenty of UK sites moving down the rankings due to the influx of US based sites.

It’s hard to quantify sometimes what has shifted in the rankings has done as you don’t have before and after data of the top 20, 30, 40 rankings.

Fortunately I have a report from January the 18th 2009 for the top 40 sites in google.co.uk on a search for “commercial laundry equipment”.

I’ve then taken the top 40 websites for the same search query today and the difference is concerning with US sites ranking in the place of many good UK sites. There is also sites from Belgium, India, Spain and China ranking in Google.co.uk.

The list is big so I have added it into a Google Docs Spreadsheet which can be seen by clicking the image.

It would be great to get someone from Google to have a look over this. If anyone has seen something similar with their clients or SERP’s post the details below in a comment. Also link to your own blog with an example of the movement of US sites in the UK rankings.

UPDATE 17/06 : Fresh Egg have an article on a similar problem with the key phrase Pet Surgery. Check out the article which has a great conspiracy theory behind the US sites ranking.

In business not everyone gets along, generally in SEO were a friendly bunch but I’ve been in the middle of arguments between clients and ex-web designers. The disagreements are usually about payments, missed deadlines or lost contracts.

I’ve seen on three occasions web developers get their own back by making little changes to stop sites ranking in Google.

So what can you learn, give all third parties a temporary FTP access to the site, when they finish work delete the account. Check after work your htaccess file and for any Google Webmaster Tools verifications (either a html file or a meta tag in the home page).

The only way to avoid any problem is not to be a dick and annoy web masters or SEO’s!