SignalkeenBot

SignalkeenBot is the crawler of Signalkeen, a research service run by Oak Mountain Digital LLC. It reads what companies and public bodies publish about themselves, so that sellers can tell when an organisation may be ready to buy a product category.

What it reads

• An organisation’s own public pages: its home page, and the pages its sitemap and links point to, such as about, careers, press and trust pages.
• Its public DNS records.
• A company’s own job board, through the board’s public posting API (Greenhouse, Lever, Ashby, Workable, Personio).
• Common Crawl’s public archive, only to date when something first appeared on an organisation’s own page.

What it never does

• It collects no personal data. Names, email addresses, phone numbers and profile links are not extracted, stored or passed on. A page is kept whole only as proof of what it said.
• It never reads LinkedIn.
• It does not log in, solve challenges or CAPTCHAs, disguise itself, or route around a block. A page that refuses it is recorded as refused.

How it behaves

• It identifies itself in every request as SignalkeenBot/0.3 with a contact address.
• It reads robots.txt and follows it as the standard (RFC 9309) describes. A group addressed to SignalkeenBot applies before *, and so does a group written for an earlier name of this crawler.
• It waits at least half a second between requests to one site, and honours a Crawl-delay of up to 60 seconds.

To stop it

Either of these works. Neither needs a reason.

Email us. Write to crawler@signalkeen.com and name the domain. The mailbox is read by a person. Within five working days we will:

• stop all requests to that domain and its subdomains, which takes effect at once for any run in progress;
• delete the pages we saved from it;
• stop passing on anything we learned about it;
• reply to confirm.

Writing from an address at the domain helps us act without a second message. We will act on a request from elsewhere too, and ask the sender to confirm.

Or use robots.txt. Add:

User-agent: SignalkeenBot
Disallow: /

It takes effect at the crawler’s next visit.

For anything else, write to the same address.