[gtld-tech] Search Engines Indexing RDAP Server Content

Hollenbeck, Scott shollenbeck at verisign.com
Fri Jan 29 13:34:35 UTC 2016

(cross-posting to multiple lists - sorry if that's inconvenient)

So I saw a tweet from Gavin Brown (@GavinBrown) that describes how one particular search engine has indexed the RDAP server of a gTLD registry operator:


This is all the more reason to work on a client authentication specification that includes support for varying responses based on client identity and authorization. I've been working on such a specification and welcome feedback on the approach:


It also begs the question of the need for a BCP describing operational practices for server operators. There are ways for web servers to influence or restrict crawler behavior, but what's appropriate in this context?


More information about the gtld-tech mailing list