[gtld-tech] [Regops] Search Engines Indexing RDAP Server Content
John R Levine
johnl at taugh.com
Fri Jan 29 17:30:15 UTC 2016
>> It also begs the question of the need for a BCP describing operational
>> practices for server operators. There are ways for web servers to influence
>> or restrict crawler behavior, but what's appropriate in this context?
> It would be good to keep in mind that respecting mechanisms such as
> robots.txt is entirely voluntary on the crawler’s side.
Right -- the only reasonable assumption is that if casual users can find
it with http, evil search engines (of which there are plenty) will too.
The search engines that most people use such as Google and Bing all obey
robots.txt so it will keep info out of casual searches. But I'd want to
understand what the threat model is before inventing solutions.
John Levine, johnl at taugh.com, Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail.
More information about the gtld-tech