[gtld-tech] [Regops] Search Engines Indexing RDAP Server Content

Luis E. Muñoz lem at uniregistry.link
Fri Jan 29 16:49:36 UTC 2016

On 29 Jan 2016, at 5:34, Hollenbeck, Scott wrote:

> It also begs the question of the need for a BCP describing operational 
> practices for server operators. There are ways for web servers to 
> influence or restrict crawler behavior, but what's appropriate in this 
> context?

It would be good to keep in mind that respecting mechanisms such as 
robots.txt is entirely voluntary on the crawler’s side.

Luis Muñoz
Director, Registry Operations

2161 San Joaquin Hills Road
Newport Beach, CA 92660

Office +1 949 706 2300 x 4242
lem at uniregistry.link
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mm.icann.org/pipermail/gtld-tech/attachments/20160129/b971dab6/attachment.html>

More information about the gtld-tech mailing list