[gtld-tech] [weirds] Search Engines Indexing RDAP Server Content

[Stephane Bortzmeyer]

On Wed, Feb 03, 2016 at 12:23:42AM +0000,
 Francisco Arias <francisco.arias at icann.org> wrote 
 a message of 77 lines which said:

> The search page
> (https://www.google.co.uk/search?q=site:rdg.afilias.info) appears to
> be the result of crawling links from the first link that appears
> there (http://rdg.afilias.info/rdap/help). The help page contains
> links to search and lookup examples that return several objects with
> their directly-related objects, which are in turn shown in the
> search results. This could have happened in web-Whois if someone
> were to publish a page containing example queries.

It seems to me that having a robots.txt at the root of the RDAP server
would solve the problem (if you regard it as a problem). Is there
anything in the tons of ICANN rules and requirments that prevent us to
install such as robots.txt?

User-agent: *
Disallow: /