1. Home
  2. Marketing Glossary

Crawling

What is crawling and what types of web crawlers are there?

Crawling is the process carried out by a spider, bot, robot or crawler (any of the search engines’ indexing robots) with the aim of locating, reading and analyzing the content of a web page.

Closely linked to this “research” work by crawling spiders is the term crawl budget: the time a spider or bot spends analyzing a website in full. This time is controlled by the search engine’s own applications, which decide which websites to crawl and how much time each one is given.

1. Types of web crawlers

There are different types of web crawlers, and the main difference between them is the purpose they were designed for. These are the main types of web crawlers you will come across:

– Search engine web crawlers. They are the best known and most commonly used ones. Each search engine has its own web crawler to examine, collect and index the content of websites.

Some of the most popular ones are: GoogleBot (Google), Bingbot (Bing), Slurpbot (Yahoo), DuckDuckBot (DuckDuckGo), Baiduspider (Baidu), Yandex Bot (Yandex) and Alexa Crawler (Amazon).

– Commercial web crawlers. These are web crawlers with many features and uses, created by software companies to sell to other companies.

– Desktop web crawlers. These are web crawlers that can run on a PC or laptop. They are usually low cost and have very limited use: they can normally crawl only small amounts of information and websites.

– Cloud web crawlers. These are web crawlers that don’t store data on local servers but in the cloud, and they are usually offered as a service by software companies. Their main advantage is that they are scalable.

– Custom web crawlers. They offer very simple functionality and are used by companies to carry out very specific tasks. One example is crawlers that monitor websites and web services for possible outages.

2. Aspects of crawling: Google

Crawling is a vital part of web ranking and SEO, so it plays a key role in this area. It is known that Google handles more than 200 variables that determine how search results are ranked.

How the algorithm works is, naturally, not known. What is known are some of the criteria taken into account to rank a web page, many of which are obtained through crawling. These include:

  • Domain age.
  • External links that refer to the domain.
  • How easy the site is for the web crawler to crawl.
  • The page structure.
  • The domain’s geographic extension.
  • The quality of the content and updates.
  • Whether or not there are errors in its HTML.
  • Whether it is optimized for mobile devices (smartphones and tablets).
  • The website loading speed.

There are more factors in SEO ranking than the ones Google makes public, and with each update of its algorithm they change or lose relevance, so you need to pay close attention after every update to its ranking algorithm.

Related entries

Discover how Mailrelay can help you with your email marketing strategy.