[gtld-tech] [weirds] Search Engines Indexing RDAP Server Content
Michele Neylon - Blacknight
michele at blacknight.com
Wed Feb 3 12:38:05 UTC 2016
On 03/02/2016, 12:06, "gtld-tech-bounces at icann.org on behalf of Hollenbeck, Scott" <gtld-tech-bounces at icann.org on behalf of shollenbeck at verisign.com> wrote:
>>From: gtld-tech-bounces at icann.org [mailto:gtld-tech-bounces at icann.org]
>>On Behalf Of Stephane Bortzmeyer
>>Sent: Wednesday, February 03, 2016 5:05 AM
>>To: Francisco Arias
>>Cc: gtld-tech at icann.org
>>Subject: Re: [gtld-tech] [weirds] Search Engines Indexing RDAP Server
>>On Wed, Feb 03, 2016 at 12:23:42AM +0000,
>> Francisco Arias <francisco.arias at icann.org> wrote
>> a message of 77 lines which said:
>>> The search page
>>> (https://www.google.co.uk/search?q=site:rdg.afilias.info) appears to
>>> be the result of crawling links from the first link that appears
>>> there (http://rdg.afilias.info/rdap/help). The help page contains
>>> links to search and lookup examples that return several objects with
>>> their directly-related objects, which are in turn shown in the
>>> search results. This could have happened in web-Whois if someone
>>> were to publish a page containing example queries.
>>It seems to me that having a robots.txt at the root of the RDAP server
>>would solve the problem (if you regard it as a problem).
>That will only work if a crawler reads robots.txt and respects the published directive(s). Not all do.
The nastier bots ignore the robots.txt directives ..
As Scott and others have pointed out, unauthenticated access *is* a problem
Trying to draw parallels between current whois (web or otherwise) and RDAP might work with less technical types, but with this audience it simply won’t fly.
RDAP’s entire “power” lies in the way that you can traverse the database in multiple ways
You cannot do that with “normal” whois and this is both a security and a privacy issue
Repeatedly telling us you don’t think it is doesn’t change the fact that it is
Mr Michele Neylon
Hosting, Colocation & Domains
Intl. +353 (0) 59 9183072
Direct Dial: +353 (0)59 9183090
Blacknight Internet Solutions Ltd, Unit 12A,Barrowside Business Park,Sleaty
Road,Graiguecullen,Carlow,Ireland Company No.: 370845
More information about the gtld-tech