Mitigation Measures for Self-Hosted Gitea Instances Overwhelmed by Crawler Traffic

Nite included in Tutorials

2026-08-02 About 500 words 3 minutes

Contents

Symptoms

A self-hosted Gitea started consuming abnormal bandwidth. The provider panel showed the traffic allowance draining at an alarming rate, with the monthly quota running out fast. Aggregating access logs by response size, the git domain accounted for the vast majority of traffic, dominated by /owner/repo/compare/ version-comparison pages with single responses up to 10MB.

The crawler pattern was obvious: Amazonbot, Lightpanda, and a rotating pool of IPs faking old Chrome/Firefox user agents, repeatedly fetching blame and compare pages. Most unique IPs made only one or two requests — classic distributed scraping.

Causes

Crawler user agents vary widely, from well-known bots (Amazonbot, Lightpanda) to rotating IPs faking browser UAs.
Heavy pages are large — compare pages exceed 10MB uncompressed — so every crawl burns real bandwidth.
Per-IP rate limiting does nothing against a rotating IP pool, where most IPs make one or two requests and move on.

Fixes

1. Caddy config

The UA blacklist (@block_bots returns 403 on match) and the heavy-page rate limit (@heavy, 2 requests per IP per 10s) live in the same site block and run in order. The UA list covers common crawlers and scanners; extend or trim it as needed:

git.example.com {
	# Logging: per-domain access log
	import log git.example.com

	# 1. UA blacklist: known crawlers get 403
	@block_bots header_regexp User-Agent "(?i)(SemrushBot|Lightpanda|GPTBot|ClaudeBot|Bytespider|AhrefsBot|MJ12bot|DataForSeoBot|Amazonbot|DotBot|CCBot|cohere-ai|Diffbot|Google-Extended|Meta-ExternalAgent|Meta-ExternalFetcher|Applebot-Extended|Timpibot|ImagesiftBot|Crawl4AI|Scrapy|ChatGLM-Spider|DeepSeekBot|cohere-training-data-crawler|AI2Bot|TikTokSpider|omgili|BLEXBot|Exabot|360Spider|80legs|MegaIndex|Barkrowler|DataCha0s|Nikto|Nmap|Nessus|OpenVAS|Sqlmap|WPScan|Nuclei|Masscan|Shodan|Acunetix|Dirbuster|Whatweb|Havij|Fimap|Jbrofuzz|Zgrab|Censys)"
	handle @block_bots {
		respond 403
	}

	# 2. Rate limit heavy pages (compare/blame etc.) per IP
	@heavy path_regexp heavy_path ^/[^/]+/[^/]+/(?:compare|blame|commit|commits|archive|pulls|issues)(?:/|$)
	handle @heavy {
		rate_limit {
			zone dynamic_zone {
				key {client_ip}
				events 2
				window 10s
			}
			log_key
		}
		reverse_proxy 100.64.0.10:3000
	}

	# 3. robots.txt: cooperative crawlers read the rules first
	handle /robots.txt {
		respond `User-agent: *
Disallow: /blame/
Disallow: /compare/
Disallow: /commit/
Disallow: /commits/
Disallow: /archive/
Disallow: /pulls/
Disallow: /issues/`
	}

	# 4. Everything else goes to Gitea
	reverse_proxy 100.64.0.10:3000
}

The @heavy path pattern covers compare, blame, commit, and other heavy page types; an IP that exceeds 2 requests in 10 seconds gets a 429. With log_key enabled, rate-limited requests are logged with their IP, which helps tune the threshold.

2. Enable origin gzip in Gitea

Add ENABLE_GZIP = true to the [server] section of app.ini, then restart Gitea:

[server]
ENABLE_GZIP = true

ENABLE_GZIP compresses runtime-generated content only; static assets are unaffected. Enable it at the origin rather than at the reverse proxy because the path is origin → proxy → visitor, and under bidirectional billing origin compression saves on both legs, while proxy-only compression saves on one. Caddy’s reverse_proxy adds Accept-Encoding: gzip for clients that omit it and passes the upstream’s compressed response through untouched. No encode directive is needed on the Caddy side.