[gtld-tech] [Regops] Search Engines Indexing RDAP Server Content

[Luis E. Muñoz]

On 29 Jan 2016, at 5:34, Hollenbeck, Scott wrote:

> It also begs the question of the need for a BCP describing operational 
> practices for server operators. There are ways for web servers to 
> influence or restrict crawler behavior, but what's appropriate in this 
> context?

It would be good to keep in mind that respecting mechanisms such as 
robots.txt is entirely voluntary on the crawler’s side.

Luis Muñoz
Director, Registry Operations
____________________________

http://www.uniregistry.link/
2161 San Joaquin Hills Road
Newport Beach, CA 92660

Office +1 949 706 2300 x 4242
lem at uniregistry.link
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mm.icann.org/pipermail/gtld-tech/attachments/20160129/b971dab6/attachment.html>