[UA-discuss] Web Crawlers behavior on UA

Edmon edmon at registry.asia
Wed Jan 8 07:35:35 UTC 2020

This is somewhat related to search engines I suppose… I believe there were some that were looked at before.

But crawler tools themselves, I am not sure.  But yes, a good point of analysis.




From: UA-discuss [mailto:ua-discuss-bounces at icann.org] On Behalf Of Dessalegn Yehuala
Sent: Wednesday, January 8, 2020 3:26 PM
To: UA-discuss <UA-discuss at icann.org>
Subject: [UA-discuss] Web Crawlers behavior on UA


Hi All,

The robot.txt file is encoded in UTF-8 format and I am just curious if there has been any study or analysis done on  web crawlers behavior on UA .


Kind regards,


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mm.icann.org/pipermail/ua-discuss/attachments/20200108/c6cdc724/attachment.html>

More information about the UA-discuss mailing list