[UA-discuss] Web Crawlers behavior on UA

[Marc Blanchet]

On 8 Jan 2020, at 2:26, Dessalegn Yehuala wrote:

> Hi All,
>
> The robot.txt file is encoded in UTF-8 format and I am just curious if
> there has been any study or analysis done on  web crawlers behavior on 
> UA .

good point. also the sitemap are to be considered.

Marc.

>
>
> Kind regards,
>
> Dessalegn
>
> _______________________________________________
> By submitting your personal data, you consent to the processing of 
> your personal data for purposes of subscribing to this mailing list 
> accordance with the ICANN Privacy Policy 
> (https://www.icann.org/privacy/policy) and the website Terms of 
> Service (https://www.icann.org/privacy/tos). You can visit the Mailman 
> link above to change your membership status or configuration, 
> including unsubscribing, setting digest-style delivery or disabling 
> delivery altogether (e.g., for a vacation), and so on.