ProofBot

ProofBot is the crawler behind Appcited. It collects public business information to build small, dated data samples.

How to recognise it

It sends this user agent: ProofBot/0.1 (+https://appcited.io/bot)

What it fetches

• Public pages and public endpoints only: for example a store's homepage, its public product listing and its contact page.
• Public registries such as certificate transparency logs and domain registration (RDAP) records.
• Never pages behind a login, and never personal data. It keeps only business details a site publishes itself, such as a support@ or hello@ address.

How it behaves

• It reads and obeys robots.txt before fetching anything else from a site.
• At most one request per second to any one host, and every response is cached so pages aren't fetched twice.
• It only reads. It never submits forms, places orders or writes to any site.
• If a site blocks it, it stops. It doesn't use proxies or other ways around blocks.

How to opt out

Add this to your robots.txt:

User-agent: ProofBot
Disallow: /

Or email [email protected] and we'll stop crawling your site and delete what we hold.