Blacklisting content scrapers, search engines, web crawlers, & other robots
You can use FortiWeb features to control access by Internet robots such as:
• search engine indexers
• automated tools such as link checkers, web crawlers, and spiders
FortiWeb keeps up-to-date the predefined signatures for malicious robots and source IPs if you have subscribed to FortiGuard Security Service.
To block typically unwanted automated tools, use
Bad Robot.
To control which search engine crawlers are allowed to access your sites, go to
Server Objects > Global > Known Search Engines; also configure
Allow Known Search Engines.
See also