Fwd: [gnso-improvem-impl-sc] Draft WG Self-Assessment Design, Objectives, and Questionnaire

[Mike O'Connor]

hi Ken,

"blazing fast" or "lost in the pile of overdue email" -- those are your choices.  :-)


On Jun 10, 2013, at 6:30 PM, "Ken Bour" <ken.bour at verizon.net> wrote:

> Mikey:
>  
> Thanks for another blazingly fast response…
>  
> RE: PDP Framework
> You mentioned this item several times in your initial comments, but I’ll address it once here. This is an interesting question as to whether the PDP framework itself is so intricately linked to the WG process that it could, on its own merits, influence the overall effectiveness of the WG. If I were to rephrase your inquiry, would this be close: “If the PDP process is cumbersome and difficult to follow, a WG’s Mission could be challenged (if not inhibited) from the outset. If we are intending to evaluate a WG’s effectiveness, one of the issues may be the very subject/content (and attendant processes) it is required to follow, e.g., the PDP framework.” If I’m near the mark, I had not thought about that linkage until you raised it. Most of the WGs I have supported were not PDP-related, so it escaped my attention. I will be interested to see what others have to add on this matter. Another option would be to develop a separate survey-like instrument to evaluate the effectiveness of the PDP Framework itself although, as you point out, it is the WG construct that is asked to navigate those waters. Hmmm…

really close on rephrasing my question -- but where i'm coming from is that *this* gang (the SCI) is really charged with evaluating the effectiveness of the process that was put in place as a result of all those committees.  so it seems like we need to gather some "is that process effective?" type data in addition to the "was the WG effective?" stuff.  so one way to rephrase it would be something like "was there anything in the rules/structure of the PDP that got in the way?"  could be something like you described -- where the WG was put at a disadvantage from the start.  or it could be something in the middle or end of the PDP rules that caused a problem.

this is the "rules committee" type function that the SCI is charged with -- if the rules that our predecessors cooked up are flawed, we're the clean up crew that's supposed to figure out ways to hammer down the last remaining sharp edges.   does that help clarify?

>  
> RE: Individual vs. Collective
> For consistency (analytical, historical, design), I think we should have one defined process vs. having each WG roll its own. That design constraint would not preclude identifying a couple of overarching questions that we want each WG to address every time (via separate format). Furthermore, it is not inconceivable to have a couple of questions that are addressed both collectively and individually. Then, we could compare both sets of answers to see whether the WG answered similarly or differently from the individuals that made up the team (very intriguing and reminiscent of the Abilene Paradox puzzle).
>  
> RE: Demographics
> I agree that private vs. public is complex and there are pros/cons to be weighed. I tend to favor the collection of personal identifying info for the reasons provided, but safeguarding that info in order to promote honesty and candor (still requires trust). Even in a largely volunteer organization, there are some people who will not be comfortable offering what may be perceived as negative commentary unless they can do so under the cover of anonymity. I think it might be cybernetics that advises: in dynamic systems, we learn primarily from error and, if failures go unreported and undiscovered, then those improvement opportunities are forever lost.

yep, absolutely right -- i'm pretty likely to be the first recipient of this type of feedback (it seems like i'm chairing all but a few of the current PDPs).  and i really want people to feel comfortable with offering candid feedback.  so i think i retract that idea of not using anonymous data.  swing and a miss.  sorry about that.

>  
> RE: Size, Length, Complexity
> Hopefully, providing several open-ended text boxes will be generally sufficient for those who have a “lot to say.” I’m not sure that any online survey instrument could be developed that would accommodate the exhaustive and insightful analysis you authored on your experience with the Fast Flux WG. From my recollection, that was and probably will always be remembered as a ONE-OF-A-KIND! J

that was indeed a hum-dinger.  and you're right, i think this could be addressed with a few open text-boxes.  maybe accompanied with an invitation to submit a separate document if it would be easier.  back to the anonymity issue too -- i was happy to pen and post a long rant and put my name on it, but some may not.

>  
> RE: Learning Objectives
> You are exactly correct. I started by extracting LOs from the WG Guidelines and Charter material; then, I built the questionnaire. Once I got into it, I began seeing that I could fold a couple of topics together (similar content) and ended up with fewer questions.

very cool.  nicely done.

>  
> Thanks for the input! I am energized by the responses thus far and am looking forward to continuing this process…

i think this might be a HUGE long-term help to the process.  let's keep at it.  

thanks,

mikey

>  
> Ken
>  
> From: owner-gnso-improvem-impl-sc at icann.org [mailto:owner-gnso-improvem-impl-sc at icann.org] On Behalf Of Mike O'Connor
> Sent: Monday, June 10, 2013 5:27 PM
> To: Ken Bour
> Cc: gnso-improvem-impl-sc at icann.org; Hoggarth, Robert (ICANN)
> Subject: Re: [gnso-improvem-impl-sc] Draft WG Self-Assessment Design, Objectives, and Questionnaire
>  
> hi Ken,
>  
> i really like the work you've done.  i've sprinkled reactions inline, but overall this is a great start.
>  
> thanks,
>  
> mikey
>  
>  
> On Jun 10, 2013, at 3:43 PM, "Ken Bour" <ken.bour at verizon.net> wrote:
> 
> 
> SCI Team Members:
>  
> Since our last teleconference, I have been working on the development of a WG Self-Assessment instrument. Rather than inundate the SCI email list with multiple drafts and iterations during the incubation period, I thought it would be more helpful to post my work products to the ICANN Community Wiki. I have now reached a point where I think the material is ready for your review.
>  
> In particular, you will find the following pages under the GNSO tab (category) and Working Group Resources space. Below is the hierarchy including links:
>  
> WG Self-Assessments: https://community.icann.org/x/2Cp-Ag
>  
> really good introduction.  on second reading i realized that one thing that might need expanding in the "lower-level" chunks of work is the LOs and questions that try to tease out the effectiveness of the formally-documented processes (the newly-changed PDP, developed by all the WGs that came before us).  not a lot more, but just a bit.  i think one of the reactions i had to the first questionnaire was that it was too focused on the formal-process stuff.  so there's some balancing to be found.  i could certainly live with the balance you've struck in this first pass -- but if there was a graceful way to add just a little more review of the PDP framework, i'd like that too.  not too much though.  we're going for "just right."  :-)
> 
> 
> ·         Design Considerations: https://community.icann.org/x/USt-Ag
>  
> individual vs collective -- i like where you're going in the highlighted box -- some kind of hybrid between the two, maybe up to the WG.  if they get really stuck arriving at consensus, just let the discussion serve as documentation rather than trying for consensus.  i agree, it would be a shame to run a working group through another tough consensus process at the very end.  especially tough if they weren't at consensus on their findings and were reopening old wounds.
>  
> demographic information -- all good -- wisecracks about the NSA notwithstanding.  although another approach to consider might be to have the information be public.  i don't know -- tradeoffs.  something to puzzle about.
>  
> size, complexity, length -- yep, this is where i came into the discussion -- the previous questionnaire was really long/hard to fill out.  but it would be interesting to see if there's a way to build in "optional" sections if people had a lot to say.  my post-mortem on my experience chairing the Fast Flux WG took several days to write.  if somebody's got a will to do something like that, we should provide a way.
>  
> methodology -- no strong opinions either way here.  
>  
> 
> 
> ·         Learning Objectives: https://community.icann.org/x/hit-Ag
>  
> first pass -- WOW!  i like these a lot.  there may be more, but this is a really good start.  good for: starting off the WG to set norms and expectations, periodically referring back to "mid flight" to offer course-corrections to the process as its's going, and reflection on opportunities for improvement at the end.
>  
> since we're a PDP-process focused group, i'm wondering if there's a way to work a couple of LOs in about the framework itself.  the goal being to see if there WG members noticed flaws in the process that got in their way.  a good source of ideas might be the questionnaire i was so cranky about.  i was mostly cranky because it was hard to answer, but maybe there's a way to rework some of those into LO's that fit in here?
>  
> 
> 
> ·         Questionnaire (Draft v1): https://community.icann.org/x/EC1-Ag
>  
> this looks like it flows pretty directly out of the LO's -- so maybe we focus on those first?  i really like how the number of LOs is longer than the number of questions -- this looks like a questionnaire that could be filled out pretty quickly.  and maybe it's just distributed as a fill-in-blanks PDF or spreadsheet (to make scraping the answers easier)?  
>  
> same reactions on the "policy-process" part being a little light, perhaps.  not a bunch more questions, but maybe one or two for those reflective types who want a place to note something in the PDP that got in the WG's way?
>  
> 
> 
>  
> If you want to follow my developmental logic path, I suggest starting at the top and proceeding in the sequence shown. If you want to skip to the actual draft questionnaire, it is contained in the bottom linked page. I included a few notes which you should bear in mind before perusing the actual questions.
>  
> At this stage, I would like to receive your feedback before proceeding further to be sure that I am on a productive track. You will certainly notice that this instrument is no longer focused on the WG Guidelines documentation (the original survey scope); instead, on the effectiveness of the WG’s operations, norms, logistics, decision-making, and outputs. I included my rationale for that shift at the top of the Design Considerations page.
>  
> Feel free to comment on the email list or add comments to the individual Wiki pages. I will attend to both sources over the next week or so as you have time to review and evaluate my recommended approach to the WG Self-Assessment.
>  
> Regards,
>  
> Ken Bour
>  
>  
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mm.icann.org/pipermail/gnso-improvem-impl-sc/attachments/20130610/7ad30fab/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3630 bytes
Desc: not available
URL: <http://mm.icann.org/pipermail/gnso-improvem-impl-sc/attachments/20130610/7ad30fab/smime.p7s>