The robots.txt file is a document that tells search engine indexing spiders which parts of a website can be indexed and provides a link to the XML-sitemap.<\/p>\n
Put more simply, a robots.txt file tells search engine crawlers which URLs of a site they can access, but it is in no way a mechanism that prevents the page from being indexed, since many crawlers ignore the robots.txt instructions.<\/p>\n
When crawlers or spiders ignore the indexing block placed on them in the robots.txt (which is none other than noindex), you will need to password-protect the website to prevent them from indexing it.<\/p>\n
In general, the robots.txt file is used to give specific instructions to the crawlers of the different search engines. Let’s look at some more specific functions.<\/p>\n
Although any default installation of a CMS<\/a> such as WordPress<\/a> creates a robots.txt file by default, you can also easily create one yourself: using a plain text editor, create a plain text file in ASCII or UTF-8 format and put the indexing instructions you want inside it.<\/p>\n Just below we will show some of the most common commands or instructions used in robots.txt.<\/p>\n The most commonly used commands in the robots.txt document are:<\/p>\n User-agent:<\/strong> also called the user agent, it is used to tell each search engine robot or spider what you want from it. It is important to note that the instructions for each crawler are given together, that is, a single instance is used for Googlebot (Google search engine) to tell it what it is and isn’t allowed to do.<\/p>\n Its basic syntax is:<\/p>\n Here are some examples of the bots or agents of the main search engines:<\/p>\n Here are some examples of lines that you can place in robots.txt and what each one is for:<\/p>\n3. robots.txt commands<\/h2>\n
\n
4. Examples for robots.txt<\/h2>\n
\n
\n
User-agent: \\\\\\*<\/code><\/p>\nDisallow: \/wp-admin\/<\/code><\/p>\nDisallow: \/temp\/<\/code><\/p>\nDisallow: \/precios.shtml<\/code><\/p>\n\n
User Agent: Googlebot-Image<\/code><\/p>\nDisallow: \/<\/code><\/p>\n\n
User-agent: Googlebot<\/code><\/p>\nDisallow: \/\\\\\\*.pdf$<\/code><\/p>\n\nRelated entries<\/h2>\n\n