# Discourse Page Views & crawlers

**URL:** <https://racket.discourse.group/t/discourse-page-views-crawlers/2081>\
**Category:** Site Feedback\
**Created:** [July 7, 2023, 8:49am UTC](https://racket.discourse.group/t/discourse-page-views-crawlers/2081 "2023-07-07T08:49:37Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![spdegabrielle](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/spdegabrielle/32/95_2.png) [@spdegabrielle](https://racket.discourse.group/u/spdegabrielle)\
**Post date:** [July 7, 2023, 8:49am UTC](https://racket.discourse.group/t/discourse-page-views-crawlers/2081/1 "2023-07-07T08:49:37Z")

</div>

I got excited the other day seeing we had exceeded the free discourse hosting page views limit

Activity by humans is going up, 😁  
...but we have also had a jump in the number of crawlers hitting the site.

As per the advice from [Controlling Web Crawlers For a Site - admins - Discourse Meta](https://meta.discourse.org/t/controlling-web-crawlers-for-a-site/244925) I am going to rate limit the worst ones - but would appreciate any advice community members care to give.

Best regards  
Stephen

| user agent | pageviews 6-June-23 to 7-July-23 |
| --- | --- |
| Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36 | 2045 |
| Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [spider-feedback@bytedance.com](mailto:spider-feedback@bytedance.com)) | 1227 |
| Tiny Tiny RSS/22.10 (Unsupported) ([https://tt-rss.org/](https://tt-rss.org/)) | 935 |
| Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | 835 |
| hackney/1.18.1 | 760 |
| Mozilla/5.0 (compatible; MJ12bot/v1.4.8; [http://mj12bot.com/](http://mj12bot.com/)) | 724 |
| Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; [help@moz.com](mailto:help@moz.com)) | 547 |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | 416 |
| Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.5735.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | 406 |
| Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_11\_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0 | 334 |

 ![image](https://global.discourse-cdn.com/free1/uploads/racket/original/2X/d/d27d128b07fb8bd88f2ea037f1c10a43654da992.png)

[Controlling Web Crawlers For a Site - admins - Discourse Meta](https://meta.discourse.org/t/controlling-web-crawlers-for-a-site/244925)