[UA-EAI] listserv question

Michael Casadevall michael at casadevall.pro
Thu Jul 25 18:50:00 UTC 2019 

I'm honestly not sure without a more through examination of the source
code. Mailman historically had problems with UTF-8 email. A second look
at the source code shows that the codebase has specific handling for
various codepage emails; from Mailmain/Defaults.py.in:

add_language('he',    _('Hebrew'),              'utf-8',       'rtl')
add_language('hr',    _('Croatian'),            'iso-8859-2',  'ltr')
add_language('hu',    _('Hungarian'),           'iso-8859-2',  'ltr')
add_language('ia',    _('Interlingua'),         'iso-8859-15', 'ltr')
add_language('it',    _('Italian'),             'iso-8859-1',  'ltr')
add_language('ja',    _('Japanese'),            'euc-jp',      'ltr')
add_language('ko',    _('Korean'),              'euc-kr',      'ltr')

The version of Mailman shipped by Ubuntu and Debian have a local patch
to better support UTF-8 out of the box for email content. This patch is
not in the upstream code as far as I know (in the Debian sources, its in
debian/patches/91_utf8.patch).

I don't know what, if any research, has been done in determining the
codepage/MIME settings of email clients to see if they're sending UTF-8
emails are not. I know on USENET/NetNews/NNTP (which shares many of the
same characteristics as standard email), iso-2022-jp is still very
common on Japanese newsgroups. I took a closer look at my sent emails,
and the initial reply to this email I sent has an encoding of "en-US",
which suggests it was sending ASCII/7-bit email and not UTF-8. That
might change if I used non-ASCII characters.

To clarify a point, listserv is here is the underlying technology of
creating a mailing list (the name comes from the LISTSERV software used
on CSNET (before the Internet). The implementation we're using for this
list and all ICANN lists is GNU mailman. It shoudld be noted that
LISTSERV still exists as a commerical product
(https://en.wikipedia.org/wiki/LISTSERV).

The fundamental technology of a mailing list is at least theoretically
UA-agonistic, but because it send emails on behalf of other others, it
is possible for said emails to go fail because they're being sent to
users who don't support EAI/IDNs. For example, Ubuntu 18.04 ships
Thunderbird 60 (and what I'm using); while it understands EAIs, it
throws an error if I try to reply to them saying EAI email addresses are
unsupported.

For a listserv/Mailman to work properly with EAIs, the following must
all be true.
 * The listserv must use a mailserver that supports EAI/IDNs
 * The software itself must support SMTPUTF8, and IDN encoding
 * The backend storage must handle UTF-8 (or another unicode encoding
correctly)
 * Emails should be normalized from codepage encoding to Unicode.
 * All users of said mailing list must be at least EAI Phase 1 compliant.

As such, the issue of UA in regards to mailing lists is "it varies, and
everyone must be on the same page).
Michael

On 7/25/19 10:07 AM, Tan Tanaka, Dennis wrote:
> Michael, thank you. Follow up question, would it be fair to say (based on the explanation below) that listserv is UA-agnostic because it depends on other components to function properly?
> 
> Dennis
> 
> On 7/24/19, 4:10 PM, "Michael Casadevall" <michael at casadevall.pro> wrote:
> 
>     It depends on what's implementing the listserv. The UA-EAI list (and all
>     ICANN lists to my knowledge) use GNU Mailman to implement listserv, and
>     the short answer is that it appears not. It *also* depends on the
>     version of Python installed on the system.
>     
>     There are known issues with EAIs with Python's built-in core libraries
>     (specifically the SMTP module trips an exception). According to the
>     documentation, Python 3.5 smtplib is supposed to support SMTPUTF8
>     (https://docs.python.org/3/library/smtplib.html), however, the core
>     libraries can't properly handle a IDNA2008 domain so this may fall over
>     in places. I haven't extensively tested but I could set up some EAI
>     mailboxes and IDN subdomains on casadevall.pro if people want me to test it.
>     
>     I downloaded the source for mailman and did a quick grep and couldn't
>     find any references to EAIs so there's probably other issues here in the
>     Archiver and List Mananger.
>     
>     I'm unaware of any alternatives that support EAI. Mailman is pretty much
>     the de facto standard
>     Michael
>     
>     On 7/24/19 3:57 PM, Tan Tanaka, Dennis via UA-EAI wrote:
>     > Hello all,
>     > 
>     >  
>     > 
>     > Does anyone know if listserv is UA-ready? If not, are there alternatives?
>     > 
>     >  
>     > 
>     > Thanks,
>     > Dennis
>     > 
>     > 
>     > _______________________________________________
>     > UA-EAI mailing list
>     > UA-EAI at icann.org
>     > https://mm.icann.org/mailman/listinfo/ua-eai
>     > 
>     > _______________________________________________
>     > By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
>     > 
>     
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pEpkey.asc
Type: application/pgp-keys
Size: 2468 bytes
Desc: not available
URL: <https://mm.icann.org/mailman/private/ua-eai/attachments/20190725/3a70d569/pEpkey.asc>

More information about the UA-EAI mailing list