[gtld-tech] Search Engines Indexing RDAP Server Content

[Hollenbeck, Scott]

(cross-posting to multiple lists - sorry if that's inconvenient)

So I saw a tweet from Gavin Brown (@GavinBrown) that describes how one particular search engine has indexed the RDAP server of a gTLD registry operator:

https://twitter.com/GavinBrown/status/692718904058191872

This is all the more reason to work on a client authentication specification that includes support for varying responses based on client identity and authorization. I've been working on such a specification and welcome feedback on the approach:

https://datatracker.ietf.org/doc/draft-hollenbeck-weirds-rdap-openid/

It also begs the question of the need for a BCP describing operational practices for server operators. There are ways for web servers to influence or restrict crawler behavior, but what's appropriate in this context?

Scott