[UA-discuss] Web Crawlers behavior on UA

Marc Blanchet marc.blanchet at viagenie.ca
Wed Jan 8 14:34:17 UTC 2020

On 8 Jan 2020, at 2:26, Dessalegn Yehuala wrote:

> Hi All,
> The robot.txt file is encoded in UTF-8 format and I am just curious if
> there has been any study or analysis done on  web crawlers behavior on 
> UA .

good point. also the sitemap are to be considered.


> Kind regards,
> Dessalegn
> _______________________________________________
> By submitting your personal data, you consent to the processing of 
> your personal data for purposes of subscribing to this mailing list 
> accordance with the ICANN Privacy Policy 
> (https://www.icann.org/privacy/policy) and the website Terms of 
> Service (https://www.icann.org/privacy/tos). You can visit the Mailman 
> link above to change your membership status or configuration, 
> including unsubscribing, setting digest-style delivery or disabling 
> delivery altogether (e.g., for a vacation), and so on.

More information about the UA-discuss mailing list