[gnso-rpm-wg] Critique of INTA survey

[Kurt Pritz]

George (et.al.):

Despite an ever-mounting sense of regret for my previous intervention, I wish to make a couple points in response to George’ email below and the discussion generally.

My sense of frustration is that parties advocating a position will author or refer to a flawed study or paper - and because that paper cites figures or “facts,” people will tend to believe it. Here are three examples:

1) George refers to the EFF and in past emails has particularly referred to the EFF white paper, "Which Internet registries offer the best protection for domain owners?” as justification for eliminated the Sunrise RPM. In my opinion, that paper contains platitudes, no independent data, and contains half-truths and, also in my opinion, seems more like a guide for those who wish to conduct illicit behavior with the least possible interference from governments or well-intentioned watchdog organizations. Yet, because it is “published," this is the source cited often on this list as authority for eliminating Sunrise. The EFF does good work and I support many of its positions but not in this case.

2) The ICANN study into the Clearinghouse operations said that Clams notices resulted in a 93% abandonment rate. Somewhere else, in relatively tiny font, the authors admitted that the figure was unreliable. Subsequently, the number published has been debunked. Yet that number is still bandied about as true, at great detriment to objective discussion

3) The recent INTA study, it seems to me, did not work out as hoped for the authors. As Kristine said, the anecdotal comments have value but maybe the report should have just focused on that. The risk is that some of the unsubstantiated claims listed in the reports findings will become tomorrow’s headlines and incorrectly shed a bad light on the multi-stakeholder model and the new gTLD program. It matters little that disclaimers were published on another page of the report, the prose will be gainsaid by any advocate for any side of a discussion. 

My comment the other day was out of frustrations resulting form the fact that authors of studies and papers go ahead and publish data when the better course would be to withhold it. I know that is a tough course to take when a study has been funded but does not turn out as hoped. 

As a result, we see hard numbers with soft disclaimers, which can lead in incorrect conclusions. 

Not to make a big deal of it but my first business school class in statistics started with a full session about ethics. We were going to be trained in the use of numbers and it was our duty not to use them inappropriately to make a point. As I said, it wasn’t a big deal but I remember it to this day. 

I understand papers and studies cannot be perfect and that good can come from an imperfect study — and I am coming to understand that all of us are capable of filtering out unsupported arguments made by those papers without me being pedantic about it.

I am sorry for this mess of an email chain to which my initial comment contributed and the rabbit hole down which we went.

Kurt






> On Aug 31, 2017, at 5:23 AM, George Kirikos <icann at leap.com> wrote:
> 
> Hi Greg,
> 
> On Wed, Aug 30, 2017 at 10:54 PM, Greg Shatan <gregshatanipc at gmail.com> wrote:
> 
>> Random samples are preferable as a starting point, but even there there are
>> various types of biases. One is "non-response bias" -- that members of the
>> selected sample don't respond to the survey.  [George seems to ignore the
>> distinction between the initial sample and the responses, but that wouldn't
>> serve his purposes....]
> 
> The report made not attempt to adjust for any of those biases, like
> self-selection bias, or that it was unrepresentative of the larger
> (1000+ members) group of INTA members.
> 
> I didn't "ignore" anything. To anticipate every argument in the first
> email I sent, and make more in depth would have distracted from the
> key points. They'd have been lost in a 50 page email.
> 
>> This is an important point -- the initial sample is really the 1000+ members
>> of INTA, which is in turn a subset of all trademark owners.  The group of
>> respondents is a subset of that sample.
> 
> How is that an "important point"? We have 33 observations. Is that in
> dispute?  We know those 33 are unrepresentative of the 1000+ members
> of INTA. And we know from the metrics of INTA that they're
> unrepresentative of all TM holders (i.e. INTA's membership is skewed
> large, and the 33 were skewed even larger).
> 
> What attempts were made to "correct" (if it's even possible) for that?
> None. It's a lost cause, because there just wasn't enough data to
> begin with to make those assertions about the larger groups in a
> statistically robust manner.
> 
> If ICA or EFF had submitted 33 self-selected survey responses that
> were skewed in a similar manner, and tried to make assertions about a
> large population, I'd be equally hard on them, as I've shown in the
> past through my balanced criticisms. My focus is on the pure math,
> regardless of where it came from or what the results said.
> 
> If/when Lori/Nielsen ever comes back to us with the margins of error
> that were requested in the other thread, those will help demonstrate
> things further (although Kurt already took an initial stab at things
> with his calculations; they'd be different for each slide, though).
> 
> Sincerely,
> 
> George Kirikos
> 416-588-0269
> http://www.leap.com/
> _______________________________________________
> gnso-rpm-wg mailing list
> gnso-rpm-wg at icann.org
> https://mm.icann.org/mailman/listinfo/gnso-rpm-wg