[UA-discuss] Web Crawlers behavior on UA

Dessalegn Yehuala mequanint.yehuala at gmail.com
Wed Jan 8 07:26:16 UTC 2020

Hi All,

The robot.txt file is encoded in UTF-8 format and I am just curious if
there has been any study or analysis done on  web crawlers behavior on UA .

Kind regards,

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mm.icann.org/pipermail/ua-discuss/attachments/20200108/39298141/attachment.html>

More information about the UA-discuss mailing list