For publishers
About our crawler
If you found this page in your server logs, this is what we were doing and how to make us stop.
How to recognise us
We identify as TodaysHeadline/0.1 (+https://todaysheadline.app/about) and never present ourselves as a browser or as anyone else.
What we request
- Your RSS or Atom feed, roughly every five minutes.
- Conditional requests. We send If-None-Match and If-Modified-Since, so an unchanged feed costs you a 304 and almost no bandwidth.
- Occasionally an article page — only when your feed carries no image, and we read <meta> tags in the <head>. We stop reading at </head> and never take article text.
We honour robots.txt, respect Crawl-delay and Retry-After, and back off after errors.
What we publish
Your headline, your own excerpt capped at 40 words, your image and your name — with every headline linking to your site. We never take article text and never write a summary of your reporting. Reading happens on your page, and we never place our advertising on or around it.
How to stop us
Fastest is to tell us: request removal, or email support@todaysheadline.app. We stop reading your feed and delete what we have stored.
You can also block us in robots.txt, which we obey:
User-agent: TodaysHeadline Disallow: /