[gtld-tech] Search Engines Indexing RDAP Server Content
shollenbeck at verisign.com
Fri Jan 29 13:34:35 UTC 2016
(cross-posting to multiple lists - sorry if that's inconvenient)
So I saw a tweet from Gavin Brown (@GavinBrown) that describes how one particular search engine has indexed the RDAP server of a gTLD registry operator:
This is all the more reason to work on a client authentication specification that includes support for varying responses based on client identity and authorization. I've been working on such a specification and welcome feedback on the approach:
It also begs the question of the need for a BCP describing operational practices for server operators. There are ways for web servers to influence or restrict crawler behavior, but what's appropriate in this context?
More information about the gtld-tech