[CWG-RFP3] Quantitative answers?

[Gomes, Chuck]

All I can say Steve is that we are not planning to do that, at least not as far as I understand.  I have been in a lot of WGs and it is common to try to get a sense of where the group is to find out where our ongoing focus needs to be.  In my opinion, that is what we are doing here.  A survey like this is usually not used for this but in this case we are looking at a very large number of public suggestions and a very large number of WG participants so it seemed quicker and more efficient to do it this way in this case.

I would argue against using the data we get in the way you think it might be used.  When we get a proposal for which the chairs think we may have strong agreement they will likely do a consensus call; we are not close to that point now.

Like I responded earlier, I think it is best to see the results we get and evaluate how we use them.  A good view of that should be possible after next weekend.

Chuck

From: Steve Crocker [mailto:steve.crocker at icann.org]
Sent: Saturday, January 03, 2015 5:42 PM
To: Gomes, Chuck
Cc: Steve Crocker; Greg Shatan; RFP3
Subject: Quantitative answers?

Chuck and Greg,

Aside from the embedded assumptions, the questions also ask for degree of agreement.  What's to be done with the answers?  Presumably they will be assembled into some form of histogram to show the number of people responding with strong agreement, agreement, neutral, disagreement, strong disagreement and no response.  Check, you wrote:

[T]he purpose is not to produce statistically significant results and it is not intended to assess consensus.  Instead it is intended to get a high level sense of where CWG participants are with respect to specific suggestions made in the public comment period.

I'm not sure I understand the distinction you're making between "high level sense of where CWG participants are" versus "consensus."  Assembling the results in a quantitative fashion seems to me exactly in the direction of mining a statistically valid result from the results.

Steve



On Jan 3, 2015, at 3:52 PM, Steve Crocker <steve.crocker at icann.org<mailto:steve.crocker at icann.org>> wrote:


Chuck,

Thanks.  Some specific comments in line below.

Steve

On Jan 3, 2015, at 3:27 PM, Gomes, Chuck <cgomes at verisign.com<mailto:cgomes at verisign.com>> wrote:


Thanks for the quick reply Steve.  It is very much appreciated.  Please see my responses below.

Chuck

From: Steve Crocker [mailto:steve.crocker at icann.org]
Sent: Saturday, January 03, 2015 2:13 PM
To: Gomes, Chuck
Cc: Steve Crocker; Greg Shatan; RFP3
Subject: Re: [CWG-RFP3] Revised Survey

Chuck,

Thanks for your note.  Both the Board as a whole and I personally have been sensitive to the concern about undue influence from the Board.  On that account we stayed silent for a long time during the initial deliberations.  Part of the feedback we got during that period is that people wanted to hear what our position was.  We submitted our position to this group directly and via the public comments.
[Chuck Gomes] I recognize that and appreciate it.

The idea that there is fundamental tension between the Board and a bottom-up multi-stakeholder process is incorrect.  ICANN as an organization and the Board in particular is not only committed to the bottom-up multi-stakeholder process, it is directly charged with nurturing, supporting and protecting it.  We are an integral part of the process.  They idea that we're conflicted with respect to the multi-stakeholder process is equivalent to saying ICANN is failing as a multi-stakeholder organization.  If that's what you or others believe, the right thing to do is to engage in that discussion directly, and, preferably, to fix ICANN.  In the meantime, ICANN has an incredible range of reporting mechanisms to ensure transparency, and more than a few accountability mechanisms in place.
[Chuck Gomes] The conflict I was referring to was not a conflict with the multistakeholder process but rather a conflict with the outcome of the multistakeholder process with regard to the IANA stewardship transition.  It is no secret that ICANN as an organization wants to continue providing the IANA services; I personally think that is a justifiable desire.  I want that to happen as well.

[Steve Crocker] An important distinction that appears to be lost in this discussion is that NTIA did not ask whether ICANN should continue to provide the function.  That's not really in question.


 My comments on the survey today are intended as constructive input to flag both the defects in the methodology and the potential difficulties this group will encounter as it proceeds down this course.  With respect to the methodology of the survey, it could be improved quite markedly if the survey included questions asking respondents whether they agree or disagree with the assumptions that the various pieces of the model are needed and whether the roles envisioned for them are appropriate.  The questions in the current draft of the survey are relevant only for those people who accept the assumptions embedded in them and could be directed to just those respondents who answer affirmatively regarding the assumptions.
[Chuck Gomes] I am the first to recognize that this survey is very flawed as a statistically valid survey.  There is no way that such a survey could be designed in the very short period of time that it was created.  Nor do I think that was necessary because the purpose is not to produce statistically significant results and it is not intended to assess consensus.  Instead it is intended to get a high level sense of where CWG participants are with respect to specific suggestions made in the public comment period.  I hope that will help us demonstrate the fact that we are diligently considering the public comments that were made and also give us more direction as to where to focus our efforts in the coming weeks.  You are right that the questions in the current draft are only  relevant to the people who accept certain assumptions but the reality is that there are people in the community who accept those assumptions.  It is now the responsibility of the CWG to sort the valid suggestions (and hence the valid assumptions) from the invalid ones and hopefully explain why we think some are valid and some are not; then we need to focus on the valid ones (including those that oppose the currently proposed approach) and work together to develop a proposal that the majority of the community can support.  Before passing final judgment on the survey, I suggest waiting to see what the results are and more importantly how the CWG uses those results.  Some statements will show strong agreement among CWG participants and some will show strong disagreement; those will be the easy ones that will likely provide clear direction in most cases.  Other statements will show mixed results and we will have to do further investigation, try to understand the basis of the different opinions (assumptions) and get creative to find ways to develop a proposal that mitigates the differences.

[Steve Crocker] The issue is not whether the survey is statistically valid but whether the questions are logically meaningful.  Statiscal validity has to do with whether an appropriate sample of people have been polled.  For example, taking a poll outside of a hot dog stand asking, "Do you like hot dogs," is unlikely to give a valid estimate of the number of people in the overall population who like hot dogs.

The problem with this survey is deeper because the questions contain assumptions.  To borrow from your profession, the questions are comparable to asking a witness when he stopped beating his wife.  The detailed questions on how to populate the CSC and MRT and how the CSC and MRT coordinate at the margins all strongly depend on acceptance of the basic model.  If the survey were explicitly limited to just those who accept this model, then I wouldn't be raising this objection.  But the survey does not attempt to find out whether the respondent accepts or rejects the model.  This is a logical flaw that undermines the credibility of the survey on an individual basis, well before statistical validity comes into play.



 My comment about the difficulties that will be encountered if this group proceeds down the current path are also intended to help the group look ahead to the discrepancy between a Contract Co + CSC + MRT + IAP model and the views of other parties.  It is thus a choice whether to deal with this difficulty now or defer it to later.
[Chuck Gomes] I think we are on the same page here.  We do need to deal with discrepancies now; I believe that is what we are trying to do; part of that is diligently considering the public comments, even those that some of us may disagree with.  In my experience within the ICANN community, people not only want to have an opportunity to comment but they also want to be assured that their comments were heard even if they are not accepted; I know you understand that as well as I do.  The comments have been submitted.  This survey will hopefully allow us to do a very quick assessment of how CWG participants view particular suggestions from the comments.  And I am sure you noticed that the statements in the survey include positions on all sides of the issues.  Please keep in mind that the survey statements were primarily developed from the public comments, not from any positions supported by the CWG or some subset of the CWG.

[Steve Crocker] I agree completely that people wanting to be assured their comments were heard.  It seems to me this survey falls short in that regard because people who do not agree with the assumption of the Contract Co + CSC + MRT + IAP model are effectively ignored.



 It would be a disservice to this group and to the community as a whole for the Board not to contribute to this discussion and provide its best advice.
[Chuck Gomes] Totally agree.  But I also think that the Board needs to give community members some credit.  It may not be overly apparent right now, but I personally think the process is working.  Will it happen as quickly as we hoped?  I seriously doubt it, but I am confident that it will happen and the result will be sound.

Steve




On Jan 3, 2015, at 1:51 PM, Gomes, Chuck <cgomes at verisign.com<mailto:cgomes at verisign.com>> wrote:



Steve,

With all due respect, I find it very interesting that the Chairman of the Board is trying to influence a bottom-up multi-stakeholder process in such a strong way.  It speaks volumes to the commitment to the multi-stakeholder model especially when ICANN the organization is highly conflicted with regard to the outcome.

With regard to the survey, it is important to understand its design and purpose.  It is not intended to be a scientific survey but rather simply a quick test of how CWG members and participants feel about suggestions made in the public comments.

I am curious as to what direction you think the CWG is headed based on the survey and would appreciate you explaining how you determined that.

Chuck

From: cwg-rfp3-bounces at icann.org<mailto:cwg-rfp3-bounces at icann.org> [mailto:cwg-rfp3-bounces at icann.org] On Behalf Of Steve Crocker
Sent: Saturday, January 03, 2015 12:31 PM
To: Greg Shatan
Cc: RFP3
Subject: Re: [CWG-RFP3] Revised Survey

Greg,

I appreciate your hard work on this.  However, as I have explained in various comments and submissions, we think this is really headed in the wrong direction.  The methodology in the survey is flawed because the assumption of some form of a Contract Co + CSC + MRT + IAP is deeply embedded in the survey.  I say this in forceful terms because this issue will need to be resolved, ideally within the CWG, but if not, then within the ICG or, if not resolved there, then after the ICG submits its proposal.

I have added comments and candidate answers to the survey to make this as clear as I can.

Steve


On Jan 3, 2015, at 11:08 AM, Greg Shatan <gregshatanipc at gmail.com<mailto:gregshatanipc at gmail.com>> wrote:




I like Milton's suggestion.  The attached final draft "accepts" all of the redlines in the previous version, cleans up a few typos, and adds Milton's suggestion (in redline to show the change).  I'm going to suggest that this should be finalized and put up as as online survey as soon as possible.
Greg

On Sat, Jan 3, 2015 at 10:45 AM, Milton L Mueller <mueller at syr.edu<mailto:mueller at syr.edu>> wrote:

Greg:



The survey has been improved very much. I can live with it as is.

I have only one suggestion. I was very happy to see that the text now clarifies that "Neither Agree nor Disagree should be used for statements that you believe you (or your group) can live with." Why not just describe the option that way to begin with? (i.e., make "can live with it" the middle option rather than describing it as "Neither Agree nor Disagree" and then clarifying what it means?)



From: cwg-rfp3-bounces at icann.org<mailto:cwg-rfp3-bounces at icann.org> [mailto:cwg-rfp3-bounces at icann.org<mailto:cwg-rfp3-bounces at icann.org>] On Behalf Of Greg Shatan
Sent: Saturday, January 3, 2015 2:44 AM
To: RFP3
Subject: [CWG-RFP3] Revised Survey



With apologies for the delay, here's a revised version of the survey we discussed on the most recent call.  Please provide comments ASAP.

Greg

<Final Draft IANA CWG Survey Questions for the CSC  MRT clean with MM Suggestion.docx>_______________________________________________
Cwg-rfp3 mailing list
Cwg-rfp3 at icann.org<mailto:Cwg-rfp3 at icann.org>
https://mm.icann.org/mailman/listinfo/cwg-rfp3



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mm.icann.org/pipermail/cwg-rfp3/attachments/20150103/131f3e86/attachment-0001.html>