How To Use The Robots.txt File To Increase Your Web Ranking

Typically we rank well on one engine for a particular keyphrase and assume that each one search engines can like our pages, and hence we will rank well for that keyphrase on a variety of engines. Unfortunately this can be rarely the case. All the majorsearch engines differ somewhat, so what’s get you ranked high on one engine may really facilitate to lower your ranking on another engine.

It’s for that reason that some folks wish to optimize pages for each particular search engine. Typically these pages would solely be slightly completely different but this slight difference might create all the difference when it comes to ranking high.

However as a result of search engine spiders crawl through sites indexing each page it can find, it may come across your search engine specific optimizes pages and because they are terribly similar, the spider could assume you’re spamming it and will do one amongst two things, ban your site altogether or severely punish you in the form of lower rankings.

The answer is this case is to prevent specific Search Engine spiders from indexing some of your net pages. This is done using a robots.txt file which resides on your webspace.

A Robots.txt file could be a important half of any webmasters battle against getting banned or punished by the search engines if she or he styles different pages for different search engine’s.

The robots.txt file is just a straightforward text file as the file extension suggests. It’s created using a simple text editor like notepad or WordPad, difficult word processors like Microsoft Word will solely corrupt the file.

You’ll be able to insert sure code during this text file to form it work. This can be how it will be done.

User-Agent: (Spider Name)
Disallow: (File Name)

The User-Agent is the name of the search engines spider and Disallow is the name of the file that you do not need that spider to index.

You have got to start a replacement batch of code for each engine, however if you want to list multiply disallow files you’ll one underneath another. For instance

User-Agent: Slurp (Inktomi’s spider)

Disallow: xyz-gg.html
Disallow: xyz-al.html
Disallow: xxyyzz-gg.html
Disallow: xxyyzz-al.html

The on top of code disallows Inktomi to spider two pages optimized for Google (gg) and two pages optimized for AltaVista (al). If Inktomi were allowed to spider these pages as well as the pages specifically created for Inktomi, you’ll run the chance of being banned or penalized. Hence, it’s always a sensible idea to use arobots.txt file.

The robots.txt file resides on your webspace, but where on your webspace? The basis directory! If you upload your file to sub-directories it can not work. If you wanted to disallow all engines from indexing a file, you just use the “*” character where the engines name would sometimes be. However beware that the “*” character will not work on the Disallow line.

Here are the names of a few of the large engines:

Excite – ArchitextSpider
AltaVista – Scooter
Lycos – Lycos_Spider_(T-Rex)
Google – Googlebot
Alltheweb – FAST-WebCrawler

Be positive to test over the file before uploading it, as you’ll have made a simple mistake, that may mean your pages are indexed by engines you do not need to index them, or even worse none of your pages would possibly be indexed.

Are you looking for more information on website seo. Or about website seo. Get pro advice on website seo.

Possibly Related Posts:

• Are You A N00b Webmaster ?
• About Keyword Research Targets And SEO
• What Is Web 3.0?
• Social Media Marketing For Marketers
• The Best SEO Practices