Matt Foster @ Thoughtworks: “The Content Signals proposal adds a consent layer. Publishers can insert three signals: ...

Matt Foster @ Thoughtworks: “The Content Signals proposal adds a consent layer. Publishers can insert three signals: search, ai‑input and ai‑train into robots.txt comments to declare whether their content may be indexed, used as real‑time AI input or included in model training . A 'yes' allows a use, 'no' forbids it, and absence expresses no preference. Cloudflare acknowledges that the signals are merely preferences, not enforceable rules, and notes that its Markdown responses currently include Content‑Signal: ai‑train=yes, search=yes, ai‑input=yes by default . The company says many customers have already deployed managed robots.txt files that permit search but disallow training, signaling a desire for fine‑grained control.”

https://www.infoq.com/news/2026/03/cloudflare-crawler/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=global

Loading...