<html>
<body>
Let me add a few thoughts based on having gone through the exercise
initially by myself and then with several ALAC colleagues. I prefix it
with the disclaimer that I did not participate in the group that created
the process - if I had, perhaps things would have been clearer.<br><br>
I had a great difficulty assigning a "priority" to a number of
the projects because it was not clear what the effect of the priority
would be. In particular, several of the projects were very near
completion. Does that mean they can be low priority, because there is
very little to do, or high priority so that we get them completed and off
the table. The other aspect is what resources does it actually use up? If
there is only two GNSO people working on a project, and it only meets (on
the average) monthly, the impact is far different from the VI project
with a near-infinite number of people meeting twice a week.<br><br>
Alan<br><br>
At 10/06/2010 04:47 AM, Stéphane Van Gelder wrote:<br>
<blockquote type=cite class=cite cite="">Hi Ken,<br><br>
Thanks for providing such a clear and concise response.<br><br>
I would venture to suggest that the fundamental question we should now be
asking ourselves is whether the system, as currently proposed, would
allow the GNSO Council to decide that projects should be stricken from
the to-do list because they are rated too low in priority?<br><br>
A secondary question may be, should the GNSO Council determine that there
is a finite number of projects that can be undertaken at the same time,
considering the available resources (# of Councillors, # of support
Staff, etc.)?<br><br>
I am copying the Council because this may be a point we want to discuss
during our teleconference tonight.<br><br>
I guess what I am saying is that whilst prioritization is a good first
step, to be truly efficient, the GNSO may want to decide that it can only
work on 15 projects in one given time period (e.g. a year) and that more
projects can only be added when one of the 15 is finished (the
"15" is arbitrary here, I only use it to illustrate my idea, I
am not suggesting that should be the number)? I freely admit to being
inspired by Staff's batching idea for new gTLDs in this regard
;)<br><br>
Stéphane<br><br>
Le 9 juin 2010 à 22:26, Ken Bour a écrit :<br><br>
<blockquote type=cite class=cite cite="">Stéphane:<br>
<br>
I will try… <br>
<br>
You wrote: “I wonder if the reason we are not getting any projects
with a lower score than 2 is that Councilors are not sufficiently aware
that they can strike projects all together should they wish to when they
rate them?”<br>
<br>
Shorter response: <br>
1) Councilors <u>are</u> rating projects
lower than 2 in Value; lots of “1s” have been registered. <br>
2) My original comment about “>2” dealt
with the Range statistic, not the individual Councilor project ratings.
<br>
<br>
I apologize for using your inquiry to springboard into a broader
discussion of central tendency measures, but I wanted to convey those
thoughts to the WPM-DT while they were fresh on my mind.
Sorry, if my explanation was overly confusing…<br>
<br>
Ken<br>
<a name="_MailEndCompose"></a> <br>
<b>From:</b> Stéphane Van Gelder
[<a href="mailto:stephane.vangelder@indom.com" eudora="autourl">
mailto:stephane.vangelder@indom.com</a>] <br>
<b>Sent:</b> Wednesday, June 09, 2010 3:56 PM<br>
<b>To:</b> Ken Bour<br>
<b>Cc:</b>
<a href="mailto:gnso-wpm-dt@icann.org">gnso-wpm-dt@icann.org</a><br>
<b>Subject:</b> Re: [gnso-wpm-dt] WPM Preliminary Status: Step 2<br>
<br>
Hi Ken,<br>
<br>
Well if I was confused before, I am now completely flummoxed after trying
to read your latest email :)<br>
<br>
If there's any way you can explain it to me in a couple of sentences, I
would be grateful. If not, please just ignore my previous comment.<br>
<br>
Stéphane<br>
<br>
Le 9 juin 2010 à 15:18, Ken Bour a écrit :<br><br>
<br>
Stephane:<br>
<br>
I think you might have misunderstood my comment. Every
project’s <u>Range</u> statistic is > 2, which means that, when I
subtract the highest rating from the lowest rating, that difference is
larger than 2 for every project. On the surface, that
statistic indicates a wide spread among certain Councilors as to their
perception of the each project’s relative Value. It also
means that there are no projects that can be automatically removed from
the Brussels discussion based upon the individual ratings step, which
required a Range of 2 or less. <br>
<br>
One of the possible improvement steps, going forward, may be to change
the way that central tendency is measured after the individual ratings
round. When we only had 5 testers, the Range looked like the
best indicator; it is easy to calculate and understand. Now
that I am seeing over a dozen values, the Standard Deviation might be a
more useful statistic for determining agreement since the population size
is so much larger. For example, out of 12 ratings for one
particular project, we have the following ratings: 5, 3, 4,
3, 6, 4, 4, 4, 5, 4, 4, 4. Even though the Range is 3 (6-3),
the Mode, the Median, and the Mean are all equal to 4 and the Standard
Deviation is less than 1.0 (actually .8), which indicates a very tight
spread not only statistically, but visually as well. For our
current exercise, we will not be excluding this project from discussion;
but, in the future, it could be a candidate for pre-determining agreement
(in this case: Rating=4) on the basis of its low Standard Deviation.
<br>
<br>
Here is one more example that is interesting. The 12 ratings thus
far are: 6, 5, 7, 6, 5, 5, 7, 7, 4, 7, 7, 7. The
Range=3, the Mode=7, Median=7, Mean=6, and Std Dev=1.04. I
think a strong case could be made for accepting 7 as the group Value
Rating although, if the rule were written such that Std Dev had to be
< 1.00, it would fail the test. <br>
<br>
I expect to have a more fully thought-out recommendation once this part
of the process has completed and I have more time to analyze the results.
<br>
<br>
Lastly, I just want to be clear there are tons of individual project
ratings that are “1” and, for that matter, “7”. From what I
can discern in examining each Councilor’s spreadsheet, it doesn’t appear
that anyone misunderstood the directions. Any value from 1-7
could be selected for any cell and, while some Councilors used the entire
range and others did not, I don’t think there is anything to be concluded
other than that is how they perceived Value across the range of Eligible
Projects. <br>
<br>
Ken<br>
<br>
<b>From:</b> Stéphane Van Gelder
[<a href="mailto:stephane.vangelder@indom.com" eudora="autourl">
mailto:stephane.vangelder@indom.com</a>] <br>
<b>Sent:</b> Wednesday, June 09, 2010 6:30 AM<br>
<b>To:</b> Ken Bour<br>
<b>Cc:</b>
<a href="mailto:gnso-wpm-dt@icann.org">gnso-wpm-dt@icann.org</a><br>
<b>Subject:</b> Re: [gnso-wpm-dt] WPM Preliminary Status: Step 2<br>
<br>
Thanks Ken for that update. I wonder if the reason we are not getting any
projects with a lower score than 2 is that Councillors are note
sufficiently aware that they can strike projects all together should they
wish to when they rate them?<br>
<br><br>
Stéphane<br>
Le 8 juin 2010 à 21:08, Ken Bour a écrit :<br><br>
<br><br>
WPM-DT Members:<br>
<br>
I thought you might appreciate receiving a brief status report concerning
Step 2-Individual Councilor Ratings…<br>
<br>
As of this afternoon, 8 June, I have received 12 Councilor ratings
spreadsheets. The deadline, as you may know, has been
extended to 9 June (tomorrow). Happily, other than a few
names/dates being left off (I am saving the emails and renaming the
attachments so that I can positively ID each one), the data aggregation
process is going as planned and tested. No one, thus far, has
failed to provide a 1-7 rating for each of the Eligible
Projects. <br>
<br>
You may be interested, if not surprised, to learn that not a single
project can be excluded from discussion after the individual rating
step. Every project’s Range is already > 2 and, of course,
it cannot get any tighter as more results are received. Of
the 15 Eligible Projects: <br>
· 11 or 73% have a Range
>= 5<br>
· 7 or 46% have a Range =
6 (max)<br>
<br>
I have developed a consolidation spreadsheet, which is automatically
color-coded to reveal the top/bottom ratings and the most prevalent
answer (or Mode). Fortunately, several projects have pretty
stable Mode/Median/Mean results meaning that, while we might have a
couple of 7’s and 1’s (thus Range=6), most participants rated the project
similarly. In a few cases, the Mode, Median, and Mean are the
identical value indicating strong central tendency (so far)!
In those instances, at least theoretically, it should be possible to
influence the small number of outliers to move closer to the group’s most
common rating. Even if that is not possible, after
discussion, it will be somewhat comforting to know that there was
reasonably strong agreement statistically. <br>
<br>
For Brussels, I estimate that we will have about 105 minutes net (if we
can hold preliminaries to 15), which leaves an average of 7 minutes per
project for discussion and polling. <br>
<br>
I am currently drafting a letter that I plan to send out early next week
(14<sup>th</sup> or 15<sup>th</sup>) addressing as many preliminaries as
possible so that the Brussels meeting (on Saturday morning) can be
quickly focused on the group ratings discussions. This letter
will cover such topics as: Councilor Preparation, Meeting
Setup, Guiding Principles, and Process Flow (briefly). I will
be encouraging participants to arrive a few minutes early so that we can
speed up the routine process of settling in…<br>
<br>
If WPM-DT members would like to preview the letter before it goes out,
please let me know. Although I recognize that you are all
very busy, I would appreciate another set of eyes on this next
communication... <br>
<br>
Regards,<br>
<br>
Ken Bour<br>
<br>
<br>
</blockquote></blockquote></body>
</html>