The 400bc crawler

You probably arrived here from a name in your server log. This is what that crawler does, how slowly it does it, and how to stop it.

What it calls itself

Every request carries exactly this User-Agent:

400bc/0.1 (+https://400bc.com/ne/crawler)

robots.txt

Each host's robots.txt is read first and obeyed. It is remembered for six hours and then read again, so a change you make takes effect within six hours at the outside.

How slowly it crawls

At least two seconds pass between two requests to the same host, and one batch never takes more than a single address from any one host. A Crawl-delay in robots.txt is honoured, up to thirty seconds — a host asking for longer than that is one not worth crawling at the rate it wants.

What it reads

Only hosts added to the crawl list by hand are crawled; it does not go looking for new sites on its own. It reads HTML and plain text only, and abandons anything oversized part-way. Only the extracted text is kept — never the raw HTML.

How to block it

This in your robots.txt is enough:

User-agent: 400bc
Disallow: /

Contact

If blocking does not work, if it seems to be crawling too fast, or if you want to ask something, write to [email protected]. Binod Tamang reads it.