Make Your Website Search Engine Robot-Friendly
If you're new here, you may want to subscribe to the RSS feed for this blog. Or you can subscribe to a combined news feed for all SMM publications. Thanks for visiting!
Search Engine Robots Read Site Maps Too
In November 2006, all the major search engines for once agreed on new Sitemap standards. Sitemaps.org set out the rules for sitemap files that all the major search engines would follow.
If you use a program such as GSiteCrawler, you can produce a full listing of all the web pages on your website in an XML file: the standard name for this file is sitemap.xml. The search engines do prefer a G-zipped version of this file, usually named sitemap.xml.gz. The GSiteCrawler program produces both versions. Although even Microsoft’s MSN/Live subscribed to this standard, as yet they have not indicated how they wish to implement the standard. The other majors have been more helpful.
A good way to start is via the website for Google’s Webmaster Tools. Once you have loaded your sitemap file to your domain, you can submit this to Google. An advantage of this approach is that Google will then in due course evaluate the sitemap file and indicate any errors therein.
The real news came up last week when Google, Yahoo! and Ask indicated that another route to inform them of the sitemap file is to include a reference to the precise URL for the sitemap file in the robots.txt file. Every domain should have a robots.txt file, even if it is an empty file. Search engine robots (or spiders) will sometimes visit a domain and check only the robots.txt file. This confirms that the domain is live. Without such a file, an error is recorded. Now you can add anywhere in the file, say at the bottom, an additional line that reads as follows:
Sitemap: http://www.yoursite.com/sitemap.xml.gz
The robots.txt file is normally checked often by search engine spiders. By doing the above, you should quickly get the new file picked up. Ask, Google and Yahoo! are all using this robots.txt file approach.
If you have just loaded up a sitemaps file and want to be sure that the sitemap file is picked up ASAP, you can ping the search engines directly. The following hyperlinks are the appropriate way to do this.
Ask:
http://submissions.ask.com/ping? sitemap=http://www.yoursite.com/sitemap.xml.gz
Google:
http://www.google.com/webmasters/sitemaps/ping? sitemap=http://www.yoursite.com/sitemap.xml.gz
Yahoo:
http://search.yahooapis.com/SiteExplorerService/V1/ping? sitemap=http://www.yoursite.com/sitemap.xml.gz
NOTE: The space after ping? should be removed. It is included here to improve the formatting of the blog post.
This should provide all the information you need on the sitemap file and how to alert the search engine robots that you have one. If there are additional points, hopefully someone will add them in the comments.
Related:
What’s new with Sitemaps.org? - Official Google Webmaster Central Blog
Use Your Robots.txt To Publish Your Sitemaps Xml File - Cre8asite Forums Discussion
BPWrap
April 21st, 2007 at 4:30 pm
Make Your Website Search Engine Robot-Friendly
Liked what you just read here ? Vote for it on Blogmemes ! Your website robots.txt file can now signal to the search engines where to find your sitemap file. This will improve the website’s search engine visibility.
April 23rd, 2007 at 2:20 pm
Hi,
Here is a question for anyone who may know the answer…
I have a new website with google ads on it. If I get 10 clicks per day on my ads, does google favor my site over and above others that may be in the same category that don’t run google ads?
I was thinking to some extent they might, so they can make more money.
Do you know anything about this?
Thanks, and I look forward to your reply.
Dean
April 23rd, 2007 at 2:56 pm
That’s a bit off-topic and I think you’d do better to post such a question in a forum such as the Cre8asite Forums, where you’d get a number of opinions.
My own opinion on this is that what you do with Adwords in no way affects the ranking of your web pages in normal keyword searches. It would only make sense for Google to do so if it could tell everyone it was doing so, which would destroy its credibility.
July 23rd, 2007 at 5:32 am
Very Helpful I have a sitemap.xml.gz file and now my robots text file points to it thanks to this article
Regards Bill Bailey
http://www.euphorish.com
July 23rd, 2007 at 6:19 am
By the way submitting the sitemap.xml.gz is not the correct file sitemap.xml is the appropriate file
as I used all those examples you offered and they all failed until i submitted sitemap.xml
Kind regards Still useful
Bill
July 31st, 2007 at 2:02 pm
Thanks for that comment, Bill. The .gz version works for me and is recommended by some experts in the field so I don’t know why you had a problem with it.