I got excited the other day seeing we had exceeded the free discourse hosting page views limit
Activity by humans is going up, 
...but we have also had a jump in the number of crawlers hitting the site.
As per the advice from Controlling Web Crawlers For a Site - admins - Discourse Meta I am going to rate limit the worst ones - but would appreciate any advice community members care to give.
Best regards
Stephen
| user agent | pageviews 6-June-23 to 7-July-23 |
|---|---|
| Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36 | 2045 |
| Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com) | 1227 |
| Tiny Tiny RSS/22.10 (Unsupported) (https://tt-rss.org/) | 935 |
| Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | 835 |
| hackney/1.18.1 | 760 |
| Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/) | 724 |
| Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com) | 547 |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | 416 |
| Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.5735.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | 406 |
| Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0 | 334 |
Controlling Web Crawlers For a Site - admins - Discourse Meta