<p dir="auto">On 29 Jan 2016, at 5:34, Hollenbeck, Scott wrote:
</p>
<blockquote>
<p dir="auto">It also begs the question of the need for a BCP describing operational practices for server operators. There are ways for web servers to influence or restrict crawler behavior, but what's appropriate in this context?
</p>
</blockquote>
<p dir="auto">It would be good to keep in mind that respecting mechanisms such as robots.txt is entirely voluntary on the crawler’s side.
</p>
<p dir="auto"><body style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><p style="font-family: 'Proxima Nova', Helvetica, Arial, sans-serif; line-height: 12px;"><span class="address-sep break" style="display: inline;"><b>Luis Muñoz</b><br></span><font color="#1e1e1e">Director, Registry Operations</font><br><span class="txt signature_jobtitle-input sig-hide" style="color: rgb(130, 130, 130); display: inline;">____________________________</span></p><p style="color: rgb(0, 0, 0); font-family: 'Proxima Nova', Helvetica, Arial, sans-serif; font-size: 10px; line-height: 14px;"><a href="http://www.uniregistry.link" class="clink sig-hide logo-container"><img src="http://static.uniregistry.net/assets/img/ur-logo@2x.png" alt="Uniregistry" class="sig-logo" height="40" border="0" width="165"></a></p><p style="color: rgb(0, 0, 0); font-family: 'Proxima Nova', Helvetica, Arial, sans-serif; font-size: 11px; line-height: 14px;"><span class="website-sep break" style="display: inline;">2161 San Joaquin Hills Road<br>Newport Beach, CA 92660</span></p><p style="color: rgb(0, 0, 0); font-family: 'Proxima Nova', Helvetica, Arial, sans-serif; font-size: 11px; line-height: 14px;"><span class="txt signature_officephone-input sig-hide" style="display: inline;">Office +1 949 706 2300 x 4242<br>lem@uniregistry.link</span></p></div></div></div></body>
</p>