[UA-discuss] Web Crawlers behavior on UA

[Edmon]

This is somewhat related to search engines I suppose… I believe there were some that were looked at before.

But crawler tools themselves, I am not sure.  But yes, a good point of analysis.

Edmon

 

 

From: UA-discuss [mailto:ua-discuss-bounces at icann.org] On Behalf Of Dessalegn Yehuala
Sent: Wednesday, January 8, 2020 3:26 PM
To: UA-discuss <UA-discuss at icann.org>
Subject: [UA-discuss] Web Crawlers behavior on UA

 

Hi All,

The robot.txt file is encoded in UTF-8 format and I am just curious if there has been any study or analysis done on  web crawlers behavior on UA .

 

Kind regards,

Dessalegn

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mm.icann.org/pipermail/ua-discuss/attachments/20200108/c6cdc724/attachment.html>