[gtld-tech] [weirds] Search Engines Indexing RDAP Server Content

[Francisco Arias]

I talked with Andrew about the email below and I think we clarified things. I thought I’ll share with the list the assessment that Gustavo and I did on the issue. Andrew, please feel free to correct me.

Gustavo and I double checked the draft RDAP profile and do not see any element in there that is leading to expose more data than what the current Whois is, e.g., a domain name links to a few entities (e.g., registrant, registrar, admin, and tech contacts), a registrar, and zero or more name servers.

The search page (https://www.google.co.uk/search?q=site:rdg.afilias.info) appears to be the result of crawling links from the first link that appears there (http://rdg.afilias.info/rdap/help). The help page contains links to search and lookup examples that return several objects with their directly-related objects, which are in turn shown in the search results. This could have happened in web-Whois if someone were to publish a page containing example queries.

In other words, the alluded behavior is not something enabled by RDAP or the profile.

Please let me know if we are missing something.


Regards,

-- 
Francisco





On 1/29/16, 6:49 PM, "gtld-tech-bounces at icann.org on behalf of Andrew Sullivan" <gtld-tech-bounces at icann.org on behalf of asullivan at dyn.com> wrote:

>On Fri, Jan 29, 2016 at 10:39:29PM +0000, Francisco Arias wrote:
>> 
>> The behavior described as vulnerability has the same potential to
>> appear in the so-called web-Whois that has been there for years and
>> it is not being proposed to disappear in neither gTLD registries nor
>> registrars.
>
>Poppycock.  The RDAP provides, on purpose, links among the objects in
>its responses.  Web whois basically provides a terminal-scrape of what
>people would get if they still knew how to type whois at a command
>line.  Since crawlers respond automatically to the very
>machine-readable markup that RDAP was precisely designed to emit, this
>means that crawlers that were never intending to catalogue the entire
>whois will now do so as a matter of course.
>
>> "Beauty is in the eye of the beholder”. What you call a
>> vulnerability others may call it a feature. 
>
>Yes.  And when my customers are giving me their information and I am
>forced by contractual terms with ICANN to deploy that in a way that
>causes a whole new class of people to suck all that up into
>widely-searchable machine-readable archives, that seems to me to be a
>new [feature|vulnerability] that I was never in a position to warn
>people about and to which they didn't agree.  
>
>> The fact of the matter is that gTLD contracts state that all
>> information must be shown in RDDS services, period. If we don’t like
>> it, there is the RDS policy development process that is tasked,
>> among other things, to revisit differentiated access. 
>
>With respect, what you are claiming is that the procedure is being
>followed and therefore this is ok.  I am claiming that Scott has
>uncovered a new consequence of the policy that seems to have
>consequences for the implementation, and that needs to be taken into
>consideration.  I'm reasonably willing to believe that, if it turned
>out using RDAP caused you accidentally to forego your first-born
>child, we'd be having a different discussion about the
>implementation.  So where, exactly, does the line fall here?
>
>> With the exception of Scott, I don’t see any of the people that have
>> complained about the lack of differentiated access in RDDS in the
>> RDS list at
>> https://community.icann.org/pages/viewpage.action?pageId=56986659. If
>> you care about this issue, please participate in RDS.
>
>I have submitted my name, but I have to admit that part of the
>difficulty in getting permission to spend yet more time on this is the
>absurd way that ICANN develops policies around things affecting the
>Internet: anyone who wants to be a "participant" has to promise to
>join inconveniently-timed phone calls (well, ok, Internet-carried
>phone calls), fly to far away places for face to face meetings, and so
>on.  If one could actually participate in Internet policy discussions
>using, you know, the Internet, it might be somewhat easier to justify
>participation.
>
>Best regards,
>
>A
>
>-- 
>Andrew Sullivan
>Dyn
>asullivan at dyn.com