Bots & Crawlers

What is a web bot 🤖?

A web bot is a computer program that systematically browses websites and creates indexes for them, by tracking the keywords entered on the internet.

These bots can have a difference purpose but they operate more or less in the same manner. Yclas allows you to prevent those bots from accessing your classifieds website or give them back their access with a single switch.

The process of collecting information from online web documents helps to reveal a website’s structure and purpose.

Therefore, the web bots/crawlers return these results to their respective search engines whenever the keywords match the search – for example, Bing, Google or Yahoo.

Why would I want to block the bots?

There are many reasons why webmasters and developers might want to block robots from indexing on their sites, and these include (but are not limited to):

  • The fact that, not all bots are good – some are bad and can crash your system.
  • The fact that, granting bots access can generally leave a site exposed and vulnerable.
  • Privacy, copyrights and legal issues.

However, allowing bots/crawlers to navigate your website has its advantages; particularly with regards to a great SEO (Search Engine Optimization) campaign.

So you shouldn’t block all bots because some are ‘good’.

How to allow/disallow bots and crawlers:

Let's learn how to allow/disallow these 🤖 Follow the steps below. 1. Login to the Admin Panel. 2. Select Settings -> General. 3. Tick the disallows (blocks) bots and crawlers on this website box (to disallow the bots)
OR
Untick/Leave empty the disallows (blocks) bots and crawlers on this website box (to allow the bots).

  1. Click the save button.

✔ The change is successful – general configuration updated.

Now go to your Admin Panel and continue customizing your classifieds website so that it truly serves its purpose.