[gnso-rpm-wg] Actions & Notes: RPM PDP WG 30 May 2018

[Justine Chew]

Prof Tushnet,

Many thanks for your reply.

You are right -- I have used the term "statistical significance"
inappropriately in this case since I forgot that we are dealing with the
universe and not a sampling of the data. I should have merely said
"patterns of significance", my apologies for that.

Thanks also for pointing to the "Fields and Methodology" worksheet. I will
take a closer look at that for my purposes.

Regards,

Justine
-----


On Thu, 31 May 2018, 10:02 Tushnet, Rebecca, <rtushnet at law.harvard.edu>
wrote:

> Thanks, Justine.  First, we haven't made any claims about statistical
> significance--that's not usually the kind of claim one makes when studying
> a universe and not testing a hypothesis (e.g., what might have happened had
> we provided half the universe/sample with free access to counsel).  The
> percentages that can be seen from the data are the percentages from the
> universe.  One might well argue that we have enough cases now to draw some
> conclusions about the functioning of the system, but it's not a matter of
> sampling.
>
>
> For other decisions made during coding, you can see them under the tab for
> "Fields and Methodology,"
>
>
> As you note, AB is a matter of determining what's in the decision and
> lacks a discretionary component.
>
>
> I don't think that V, W, and X require much if anything in the way of
> judgment: exact string matching (V) is pretty self-explanatory, and W and X
> sort among the remaining domain names.  We haven't done any percentage
> analysis relying on V, W, and X, but if you're concerned about
> categorization, collapsing W and X into "non-identical string" could remove
> any discretion.  Final note: In the page on methodology, you can see that
> spaces and hyphens were disregarded in determining placement into these
> categories.
>
>
> Y, Z, and AA: I'd be the last person to claim that there's much objective
> about sorting terms between descriptive and suggestive!  That's got to be a
> matter of judgment.
>
>
>
> Rebecca Tushnet
> Frank Stanton Professor of First Amendment Law, Harvard Law School
> 703 593 6759
> ------------------------------
> *From:* Justine Chew <justine.chew at gmail.com>
> *Sent:* Wednesday, May 30, 2018 9:27:05 PM
> *To:* Tushnet, Rebecca
> *Cc:* gnso-rpm-wg at icann.org
> *Subject:* Re: [gnso-rpm-wg] Actions & Notes: RPM PDP WG 30 May 2018
>
> Prof. Tushnet,
>
> I had a couple of questions regarding the research you/your team
> undertook:-
>
> 1. Looking at the MASTER -- Enter Data worksheet, would I be correct in
> concluding that only the TM Analysis columns (i.e. columns V to AB, *well
> maybe excluding column AB*) called for a subjective judgment on the part
> of your data entry person / RA? In other words, all data in all columns
> were based on actual "matches" or "presence of matching or relevant text"
> *except* for the following columns:
>
> V: Domain name (string) identical to TM
> W: Domain name contains TM + generic or related term
> X: Domain name similar to TM
> Y: TM is fanciful
> Z: TM is arbitrary/suggestive for category
> AA: TM is descriptive for category
> *AB: TM owner in TMCH (mentioned in decision)*
>
> 2. Apologies if I had missed coming across it in earlier emails/calls, is
> there a compendium report to the research/data that informs on assumptions
> taken and any resulting patterns of statistical significance?
>
> ​Many thanks
> ,
>
> Justine Chew
> -----
>
> On 31 May 2018 at 05:39, Tushnet, Rebecca <rtushnet at law.harvard.edu>
> wrote:
>
> Thanks all.  My deepest apologies for missing the meeting--I had conflated
> it with the later meeting today on my calendar.
>
>
> If there are specific questions I can answer, I'd be happy to do so.  For
> example, there's some mention in the notes of the "assumptions" of the
> coding--I am unclear what exactly is at issue and I'd like to know so we
> can discuss in the group.
>
>
>
> Rebecca Tushnet
> Frank Stanton Professor of First Amendment Law, Harvard Law School
> 703 593 6759
> ------------------------------
> *From:* gnso-rpm-wg <gnso-rpm-wg-bounces at icann.org> on behalf of Julie
> Hedlund <julie.hedlund at icann.org>
> *Sent:* Wednesday, May 30, 2018 5:12:12 PM
> *To:* gnso-rpm-wg at icann.org
> *Subject:* [gnso-rpm-wg] Actions & Notes: RPM PDP WG 30 May 2018
>
>
> Dear All,
>
>
>
> Please see below the action items and brief notes captured by staff from
> the Working Group call held on 30 May 2018 (1200 UTC).  Staff have posted
> to the wiki space the action items and notes.  *Please note that these
> will be high-level notes and are not meant as a substitute for the
> transcript or recording.*  The recording, transcript, WebEx chat, and
> attendance records are posted on the wiki at:
> https://community.icann.org/display/RARPMRIAGPWG/2018-05-30+Sub+Team+for+Data
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__community.icann.org_display_RARPMRIAGPWG_2018-2D05-2D30-2BSub-2BTeam-2Bfor-2BData&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=E-M4OQvQBo8UWqE1LwEiDR3PcWlfM0I-0jiI1c4ous0&m=Fz5mz32nned9r4I18xXJQUzQMbagMGoa3kI6hmEPJMI&s=JKd6pXw7OFfZqDQjtB7OEgFZ16KYACFfJVhoTs1Sd0s&e=>.
>
>
>
>
> Best Regards,
>
> Julie
>
> Julie Hedlund, Policy Director
>
>
>
> *Action Items:*
>
>
>
>    1. Co-Chairs will consider how to proceed with the data from Prof.
>    Tushnet and send a proposal to the WG.
>    2. Co-Chairs and staff will develop programs for the ICANN62 sessions:
>       1. Session 1: Wednesday, 27 June, 10:30-12:00
>       2. Session 2: Thursday, 28 June, 09:00-10:15, and
>       3. Session 3: Thursday, 28 June, 10:30 to 12:00
>
>
>
> *Notes:*
>
>
>
> 1. Discuss what quantitative data from Rebecca’s research can be used to
> address Documents Sub Team questions about the 250 “response received” and
> 58 “respondent prevailed” cases,
>
>
>
> Introduction -- Berry Cobb:
>
> -- The intent for today’s agenda was to understand what the WG’s or Sub
> Teams next steps are for Rebecca Tushnet’s research.  Within the Documents
> Sub Team that began with the foundation document with the charter questions
> created an additional column on what types of data sets would help to
> answer those questions.
>
> -- When the Document Sub Team was wrapping up it work it was about the
> time Rebecca sent the result of her coding exercise.
>
> -- WG/Sub teams can review the data to see how it might answer the charter
> questions, how this data could be used – such as put back in the sub Teams
> – or considered by the full WG.
>
> -- Rebecca’s coding data set there were 4-5 potential areas where the data
> might help the Providers Sub Team answer some questions.
>
> -- As it relates to the agenda item, staff has already done some analysis
> (staff is still reviewing the results of the data) the responses we were
> able to look at those a little bit more in depth based on the coding
> exercise.  Was the response in 14 days and if not in 14 days was there a
> response in 6 months and was there a request for an extension.
>
> -- In the 58 cases where the respondent prevailed just about half of them
> there was no response, which creates a smaller subset of cases we could
> review.  That would be around the area as it relates to the standard of
> proof defenses in 5.7 and 5.8.
>
> -- As it relates to Rebecca’s research there is one particular part of
> that analysis that talks about the domain’s disposition.  What happened to
> it after the determination had been made.  Did the domain get suspended and
> what happened after it was suspended.  The WHOIS data I had provided was
> not a current query.
>
> -- I have done a fresh query against WHOIS and got 1400 from automated and
> manually fetched other 400, but this new data set will tell us whether the
> name is registered or not.  Then I have looked at whether the name is under
> suspension.  Of those that aren’t suspended to categorize them as to
> whether they are under brand protection, DPML, or whether the original
> registrant is using the name.  Cleaning up that data.
>
> -- Having done that exercise there are two things to note: As one example
> (preliminary results) I only found one instance where domain that the
> complainant prevailed, went through suspension, became available for
> registration, and the original registrant re-registered the name.  In doing
> this I noted that there were duplicate domains that had two URS cases filed
> against them.  Some of which where the case that was first filed was either
> withdrawn or the claim was denied and the trademark owner resubmitted the
> claim.  Another primary use case where the trademark owner prevailed in the
> original case, the domain was deleted, then it was registered by a
> different registrant and a different URS case was filed against the same
> name.  It looks like it might be another useful data set.
>
> -- Trying to confirm and validate some of the pivot table.
>
> -- The question is how to proceed with the use of this data set at the
> full WG or Sub Team level.
>
>
>
> Discussion:
>
> -- There were 827 URS cases filed to date.  Out of those there were
> responses in about 250.  In 58 half of those came in cases where there was
> no response.  The 58 is a subset of all the cases.  What conclusions from
> that high level data from that research could help to draw conclusions
> about the operation of the URS – the response time, objectiveness of the
> determinations, etc.  So far as whether conclusions to be drawn by the full
> WG or by the Sub Team, we don’t have enough people on the call to draw that
> conclusion.
>
> -- In the questions from the Practitioners or Providers Sub Team could
> some of those be addressed by the data? Answer: Don’t think there were
> potential data sets where the data might be useful for the Practitioners,
> but there were 4 or 5 for the providers since there were some pretty
> detailed questions and they would need to review the cases to respond.  In
> the 4-5 instances where this could be helpful the providers could try to
> use the data.
>
> -- Note that it will take time for some analysis to find the appropriate
> coding columns and create a pivot table.  There are several summary tabs
> that may be able to answer those questions, but likely need some manual
> analysis such as via pivot tables, to aid the WG or the Sub Team.
>
> -- Noticed that there are other questions that we can answer with the data
> using the cases by respondent tab.  See some non-English speaking
> countries.  If we look at the default rate for non-English speaking
> countries that could help answer some of the language related questions,
> even the success rate when the do respond.
>
> -- While this data is interesting, how are we trying to use it?  Make sure
> that we don’t jump to conclusions based on raw data.  We don’t have things
> we are trying to prove or disprove.  Worried about taking months to look at
> data without context.
>
> -- We do have a lot of comprehensive data.  There are a certain number of
> withdrawals as well.  We have a lot of questions and a lot of data.  We
> need to see how we can use it.
>
> -- It seems that individual members have theories that they would like to
> prove using this data.  We need to be careful with the inferences that we
> draw from the data and extrapolating.  For example, whether the default
> rate for non-English speakers can tell us anything.  We don’t know.
>
> -- We need to be very careful about only drawing conclusions that flow
> naturally from the data and not pull things out that aren’t clearly
> supported.
>
> -- Sub Teams could perhaps review the data and decide which of their
> questions might be addressed.
>
> -- Question: Noted that there are a series of language cases, but not sure
> what that meant.  Also, in the summary tables there are references to
> default, default/final, and then final.  Wonder how those distinctions were
> made.  Are there only 15 cases where there was a final decision in the
> instance of a default.  Or there 566 cases where no final decision was
> rendered.
>
> -- What staff observed: The one tab that related to the country is only
> from the registrants location and don’t think it has more detailed
> information.  The only two providers where cases were in a language other
> than English were in MFSD and ADNDRC.  Haven’t come across one case from
> FORUM that wasn’t in English.  The same columns of data are similar from
> the original data set that staff scraped from the providers and that is how
> they are displayed on the providers site.  They will provide two kinds of
> status at it relates to the cases. What is the ultimate outcome – either
> claim was denied, domain was suspended, case was withdrawn.  Then you had
> the determination and anywhere were it lists default there was no response,
> no request for an extension, and it was a default outcome that resulted in
> the domain being suspended, and/or the claim denied.  In cases where there
> is a final it meant there was a response to that case that could have
> happened in the 14 days or the 6-month period.  There are a few that are
> default/final, and an additional column that highlights appeals.  When you
> are looking at Prof. Tushnet’s sheet the determination is what staff had
> scraped from the providers’ web sites.
>
> -- Regarding the use of an appeal, is there a place where that type of
> case has been highlighted?  Answer: The Documents Sub Team did an analysis
> that was part of the summary report that the Sub Team produced.  There were
> 14 appeals that we analyze, compare the case to the appeal, understand the
> responses, but didn’t dive into the aspects of the elements to prove within
> those cases other than highlighting what it was and the ultimate
> determination.
>
> -- The appeals are highlighted in the research in column AP – either coded
> as not applicable, successful, or unsuccessful and column AQ will have a
> link the appeal.
>
>
>
> Phil reading from the chat:
>
> Agree with Jeff. I think just looking at data by itself does not help
> without a context. In particular, the data has some assumptions and coding
> built into it that may not be accurate . So if we are going to look at the
> data we need to consider Rebecca's underlying assumptions as well so we
> have a better sense of what the data might or might not show.
>
> 05/30/2018 07:32:30 AM from Mary Wong to All Participants:
>
> As Berry noted, it is for the WG to decide which parts of the data, and
> how, is relevant to the scope of its work. Rebecca's RA has done a lot of
> work, but it is entirely her research that she has kindly provided to us in
> full. That doesn't mean it is all directly relevant to the WG's work, so
> that's why some of it may be very useful from the data side but perhaps not
> all.
>
> 05/30/2018 07:34:02 AM from Mary Wong to All Participants:
>
> For example, some of it seems very relevant to the Documents Sub Team
> (which highlighted the Respondent Prevailed and Response Filed cases as the
> two other categories it considered may be helpful to look at).
>
> 05/30/2018 07:35:59 AM from Mary Wong to All Participants:
>
> The other two Sub Teams may wish to see if any of the data can supplement,
> as quantitative measures, responses received from the Providers and/or
> Practitioners - since, collectively, these three Sub Teams scoped out the
> URS more or less.
>
>
>
> -- Suggestion: We have very extensive research by Prof. Tushnet that is
> not a WG document that is submitted to the WG to consider whether it could
> help to answer our revised charter questions relating to the URS.  Some of
> it will be relevant to the Sub Teams.  Suggest that the Co-Chairs will
> discuss this about the best way to proceed, which may be to invite members
> from the three URS Sub Teams to engage with the Professor and report back
> to the WG by ICANN62 as to the relevance of the data with respect to the
> URS charter questions.  We hope to have the answers back from the
> practitioners and providers surveys by then.  Co-Chairs will consider this
> and come back to the WG with a proposal for how to proceed.
>
>
>
> 2. ICANN62 Planning:
>
>
>
> Three sessions:
>
> Session 1: Wednesday, 27 June, 10:30-12:00
>
> Session 2: Thursday, 28 June, 09:00-10:15, and
>
> Session 3: Thursday, 28 June, 10:30 to 12:00
>
>
>
> -- Staff will work with the Co-Chairs to develop programs for each of the
> sessions.
>
> -- One will be talking about procedural issues leading up the publication
> of our initial report.
>
>
>
> _______________________________________________
> gnso-rpm-wg mailing list
> gnso-rpm-wg at icann.org
> https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__mm.icann.org_mailman_listinfo_gnso-2Drpm-2Dwg&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=E-M4OQvQBo8UWqE1LwEiDR3PcWlfM0I-0jiI1c4ous0&m=Fz5mz32nned9r4I18xXJQUzQMbagMGoa3kI6hmEPJMI&s=qDWDazORF6UAy5MvOTuwpAlRY6LySPSbvNanC-v80JY&e=>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mm.icann.org/pipermail/gnso-rpm-wg/attachments/20180531/70f57645/attachment-0001.html>