Bot Blocking is a security feature designed to give publishers the power to control which bots and user agents can access their site content. This tool is essential for preventing unauthorized data scraping and ensuring that content is not used to spread misinformation. The feature supports enhanced digital security and privacy by allowing site administrators to specify which bots should be blocked from accessing their sites.
The Bot Blocking feature includes three settings that cater to different levels of bot management needs, as depicted in the provided screenshots:
The Settings Tab within the Bot Blocking Settings showing the bot blocking options. 'Enable BLOX Digital automated bot blocking' is the default setting.
The Blocked User Agents Tab within the Bot Blocking Settings features a list view with 'New' and 'Delete' options, allowing admins to manage their custom list of blocked bots.
Bot Blocking
The Settings Tab within the Bot Blocking Settings showing the bot blocking options. 'Enable BLOX Digital automated bot blocking' is the default setting.
The Blocked User Agents Tab within the Bot Blocking Settings features a list view with 'New' and 'Delete' options, allowing admins to manage their custom list of blocked bots.
Enable BLOX Digital Automated Bot Blocking (Default):
This setting activates a pre-configured list of bots deemed problematic by BLOX Digital. It is maintained and regularly updated to ensure comprehensive protection against unwanted scraping. Note, navigate to https://<domain>/robots.txt to view what is being blocked by default
When this option is enabled, AI2Bot, AI2Bot-Dolma, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, cohere-ai, Diffbot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, ICC-Crawler, ImagesiftBot, Meta-ExternalAgent, OAI-SearchBot, PerplexityBot, PetalBot, Scrapy, Timpibot, VelenPublicWebCrawler, Webzio-Extended, YouBot, iaskspider/2.0, img2dataset, omgili, omgilibot. are automatically blocked based on BLOX Digital's curated list.
Manually Define User Agents:
This setting provides flexibility for users to add specific user agents that they wish to block. It is useful for sites that encounter unique bots not included in the automated list.
Administrators can add or remove user agents via a simple interface, ensuring that only the desired bots are blocked.
Disabled:
Selecting this option will disable bot blocking entirely, allowing all bots and user agents to crawl the site. This setting is suitable for sites that do not require restrictions on data scraping.
How to Access and Configure Bot Blocking Settings
Navigate to 'Settings' under ‘Other’ in the Application Menu
Click 'Bot Blocking’ to open the settings dialog
Select ‘Enable BLOX Digital automated bot blocking’, ‘Manually define user agents’, or ‘Disabled’
If selecting, ‘Manually define user agents’, navigate to the ‘Blocked User Agents’ Tab to add or remove user agents as needed to customize which bots are blocked.
