Discourse Page Views & crawlers

I got excited the other day seeing we had exceeded the free discourse hosting page views limit

Activity by humans is going up, :grin:
...but we have also had a jump in the number of crawlers hitting the site.

As per the advice from Controlling Web Crawlers For a Site - admins - Discourse Meta I am going to rate limit the worst ones - but would appreciate any advice community members care to give.

Best regards

user agent pageviews 6-June-23 to 7-July-23
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36 2045
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com) 1227
Tiny Tiny RSS/22.10 (Unsupported) (https://tt-rss.org/) 935
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) 835
hackney/1.18.1 760
Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/) 724
Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com) 547
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 416
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.5735.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 406
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0 334

Controlling Web Crawlers For a Site - admins - Discourse Meta