[gtld-tech] [weirds] Search Engines Indexing RDAP Server Content

[Francisco Arias]

On 2/3/16, 2:05 AM, "Stephane Bortzmeyer" <bortzmeyer at nic.fr> wrote:


>On Wed, Feb 03, 2016 at 12:23:42AM +0000,
> Francisco Arias <francisco.arias at icann.org> wrote 
>
>> The search page
>> (https://www.google.co.uk/search?q=site:rdg.afilias.info) appears to
>> be the result of crawling links from the first link that appears
>> there (http://rdg.afilias.info/rdap/help). The help page contains
>> links to search and lookup examples that return several objects with
>> their directly-related objects, which are in turn shown in the
>> search results. This could have happened in web-Whois if someone
>> were to publish a page containing example queries.
>
>It seems to me that having a robots.txt at the root of the RDAP server
>would solve the problem (if you regard it as a problem). Is there
>anything in the tons of ICANN rules and requirments that prevent us to
>install such as robots.txt?
>
>User-agent: *
>Disallow: /

There is nothing in the draft RDAP profile or any other rule in ICANN that I’m aware that would prohibit a registry/registrar to do what you suggest for both web-Whois and RDAP, Stephane.

Regards,

-- 
Francisco