[gtld-tech] [weirds] Search Engines Indexing RDAP Server Content

[John Levine]

>That will only work if a crawler reads robots.txt and respects the
>published directive(s). Not all do.

All of the search engines used by consumers do.

I'm still having trouble understanding what the problem is here.  The
specific set of records that Scott noticed are in fact just some
examples linked from a public web page, and I see no reason to think
that it'd be hard to keep RDAP info out of the usual search engines if
that's what you want to do.  For a very long time, Domaintools and
others have scraped WHOIS info and provide a little of it for free and
more for pay.  RDAP doesn't change that.

If you want to redact information beyond what's in WHOIS, that's a
reasonable discussion to have, but it's exactly the same for WHOIS or
RDAP.

R's,
John