[council] Re: [gnso-wpm-dt] WPM Preliminary Status: Step 2

[Alan Greenberg]

Let me add a few thoughts based on having gone 
through the exercise initially by myself and then 
with several ALAC colleagues. I prefix it with 
the disclaimer that I did not participate in the 
group that created the process - if I had, 
perhaps things would have been clearer.

I had a great difficulty assigning a "priority" 
to a number of the projects because it was not 
clear what the effect of the priority would be. 
In particular, several of the projects were very 
near completion. Does that mean they can be low 
priority, because there is very little to do, or 
high priority so that we get them completed and 
off the table. The other aspect is what resources 
does it actually use up? If there is only two 
GNSO people working on a project, and it only 
meets (on the average) monthly, the impact is far 
different from the VI project with a 
near-infinite number of people meeting twice a week.

Alan

At 10/06/2010 04:47 AM, Stéphane Van Gelder wrote:
>Hi Ken,
>
>Thanks for providing such a clear and concise response.
>
>I would venture to suggest that the fundamental 
>question we should now be asking ourselves is 
>whether the system, as currently proposed, would 
>allow the GNSO Council to decide that projects 
>should be stricken from the to-do list because 
>they are rated too low in priority?
>
>A secondary question may be, should the GNSO 
>Council determine that there is a finite number 
>of projects that can be undertaken at the same 
>time, considering the available resources (# of 
>Councillors, # of support Staff, etc.)?
>
>I am copying the Council because this may be a 
>point we want to discuss during our teleconference tonight.
>
>I guess what I am saying is that whilst 
>prioritization is a good first step, to be truly 
>efficient, the GNSO may want to decide that it 
>can only work on 15 projects in one given time 
>period (e.g. a year) and that more projects can 
>only be added when one of the 15 is finished 
>(the "15" is arbitrary here, I only use it to 
>illustrate my idea, I am not suggesting that 
>should be the number)? I freely admit to being 
>inspired by Staff's batching idea for new gTLDs in this regard ;)
>
>Stéphane
>
>Le 9 juin 2010 à 22:26, Ken Bour a écrit :
>
>>Stéphane:
>>
>>I will try

>>
>>You wrote:  “I wonder if the reason we are not 
>>getting any projects with a lower score than 2 
>>is that Councilors are not sufficiently aware 
>>that they can strike projects all together 
>>should they wish to when they rate them?”
>>
>>Shorter response:
>>1)      Councilors are rating projects lower 
>>than 2 in Value; lots of “1s” have been registered.
>>2)      My original comment about “>2” dealt 
>>with the Range statistic, not the individual Councilor project ratings.
>>
>>I apologize for using your inquiry to 
>>springboard into a broader discussion of 
>>central tendency measures, but I wanted to 
>>convey those thoughts to the WPM-DT while they 
>>were fresh on my mind.   Sorry, if my explanation was overly confusing

>>
>>Ken
>>
>>From: Stéphane Van Gelder [mailto:stephane.vangelder at indom.com]
>>Sent: Wednesday, June 09, 2010 3:56 PM
>>To: Ken Bour
>>Cc: <mailto:gnso-wpm-dt at icann.org>gnso-wpm-dt at icann.org
>>Subject: Re: [gnso-wpm-dt] WPM Preliminary Status: Step 2
>>
>>Hi Ken,
>>
>>Well if I was confused before, I am now 
>>completely flummoxed after trying to read your latest email :)
>>
>>If there's any way you can explain it to me in 
>>a couple of sentences, I would be grateful. If 
>>not, please just ignore my previous comment.
>>
>>Stéphane
>>
>>Le 9 juin 2010 à 15:18, Ken Bour a écrit :
>>
>>
>>Stephane:
>>
>>I think you might have misunderstood my 
>>comment.   Every project’s Range statistic is > 
>>2, which means that, when I subtract the 
>>highest rating from the lowest rating, that 
>>difference is larger than 2 for every 
>>project.   On the surface, that statistic 
>>indicates a wide spread among certain 
>>Councilors as to their perception of the each 
>>project’s relative Value.   It also means that 
>>there are no projects that can be automatically 
>>removed from the Brussels discussion based upon 
>>the individual ratings step, which required a Range of 2 or less.
>>
>>One of the possible improvement steps, going 
>>forward, may be to change the way that central 
>>tendency is measured after the individual 
>>ratings round.   When we only had 5 testers, 
>>the Range looked like the best indicator; it is 
>>easy to calculate and understand.   Now that I 
>>am seeing over a dozen values, the Standard 
>>Deviation might be a more useful statistic for 
>>determining agreement since the population size 
>>is so much larger.   For example, out of 12 
>>ratings for one particular project, we have the 
>>following ratings:   5, 3, 4, 3, 6, 4, 4, 4, 5, 
>>4, 4, 4.   Even though the Range is 3 (6-3), 
>>the Mode, the Median, and the Mean are all 
>>equal to 4 and the Standard Deviation is less 
>>than 1.0 (actually .8), which indicates a very 
>>tight spread not only statistically, but 
>>visually as well.   For our current exercise, 
>>we will not be excluding this project from 
>>discussion; but, in the future, it could be a 
>>candidate for pre-determining agreement (in 
>>this case: Rating=4) on the basis of its low Standard Deviation.
>>
>>Here is one more example that is 
>>interesting.  The 12 ratings thus far are:  6, 
>>5, 7, 6, 5, 5, 7, 7, 4, 7, 7, 7.   The Range=3, 
>>the Mode=7, Median=7, Mean=6, and Std 
>>Dev=1.04.   I think a strong case could be made 
>>for accepting 7 as the group Value Rating 
>>although, if the rule were written such that 
>>Std Dev had to be < 1.00, it would fail the test.
>>
>>I expect to have a more fully thought-out 
>>recommendation once this part of the process 
>>has completed and I have more time to analyze the results.
>>
>>Lastly, I just want to be clear there are tons 
>>of individual project ratings that are “1” and, 
>>for that matter, “7”.   From what I can discern 
>>in examining each Councilor’s spreadsheet, it 
>>doesn’t appear that anyone misunderstood the 
>>directions.   Any value from 1-7 could be 
>>selected for any cell and, while some 
>>Councilors used the entire range and others did 
>>not, I don’t think there is anything to be 
>>concluded other than that is how they perceived 
>>Value across the range of Eligible Projects.
>>
>>Ken
>>
>>From: Stéphane Van Gelder [mailto:stephane.vangelder at indom.com]
>>Sent: Wednesday, June 09, 2010 6:30 AM
>>To: Ken Bour
>>Cc: <mailto:gnso-wpm-dt at icann.org>gnso-wpm-dt at icann.org
>>Subject: Re: [gnso-wpm-dt] WPM Preliminary Status: Step 2
>>
>>Thanks Ken for that update. I wonder if the 
>>reason we are not getting any projects with a 
>>lower score than 2 is that Councillors are note 
>>sufficiently aware that they can strike 
>>projects all together should they wish to when they rate them?
>>
>>
>>Stéphane
>>Le 8 juin 2010 à 21:08, Ken Bour a écrit :
>>
>>
>>
>>WPM-DT Members:
>>
>>I thought you might appreciate receiving a 
>>brief status report concerning Step 2-Individual Councilor Ratings

>>
>>As of this afternoon, 8 June, I have received 
>>12 Councilor ratings spreadsheets.   The 
>>deadline, as you may know, has been extended to 
>>9 June (tomorrow).   Happily, other than a few 
>>names/dates being left off (I am saving the 
>>emails and renaming the attachments so that I 
>>can positively ID each one), the data 
>>aggregation process is going as planned and 
>>tested.   No one, thus far, has failed to 
>>provide a 1-7 rating for each of the Eligible Projects.
>>
>>You may be interested, if not surprised, to 
>>learn that not a single project can be excluded 
>>from discussion after the individual rating 
>>step.   Every project’s Range is already > 2 
>>and, of course, it cannot get any tighter as 
>>more results are received.   Of the 15 Eligible Projects:
>>·         11 or 73% have a Range >= 5
>>·         7 or 46% have a Range = 6 (max)
>>
>>I have developed a consolidation spreadsheet, 
>>which is automatically color-coded to reveal 
>>the top/bottom ratings and the most prevalent 
>>answer (or Mode).   Fortunately, several 
>>projects have pretty stable Mode/Median/Mean 
>>results meaning that, while we might have a 
>>couple of 7’s and 1’s (thus Range=6), most 
>>participants rated the project similarly.   In 
>>a few cases, the Mode, Median, and Mean are the 
>>identical value indicating strong central 
>>tendency (so far)!   In those instances, at 
>>least theoretically, it should be possible to 
>>influence the small number of outliers to move 
>>closer to the group’s most common 
>>rating.   Even if that is not possible, after 
>>discussion, it will be somewhat comforting to 
>>know that there was reasonably strong agreement statistically.
>>
>>For Brussels, I estimate that we will have 
>>about 105 minutes net (if we can hold 
>>preliminaries to 15), which leaves an average 
>>of 7 minutes per project for discussion and polling.
>>
>>I am currently drafting a letter that I plan to 
>>send out early next week (14th or 15th) 
>>addressing as many preliminaries as possible so 
>>that the Brussels meeting (on Saturday morning) 
>>can be quickly focused on the group ratings 
>>discussions.   This letter will cover such 
>>topics as:   Councilor Preparation, Meeting 
>>Setup, Guiding Principles, and Process Flow 
>>(briefly).   I will be encouraging participants 
>>to arrive a few minutes early so that we can 
>>speed up the routine process of settling in

>>
>>If WPM-DT members would like to preview the 
>>letter before it goes out, please let me 
>>know.   Although I recognize that you are all 
>>very busy, I would appreciate another set of 
>>eyes on this next communication...
>>
>>Regards,
>>
>>Ken Bour
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mm.icann.org/pipermail/council/attachments/20100610/b2ebb958/attachment.html>