[DTA - SLE] Questions and Answers - Revised

Tue Jun 23 11:36:49 UTC 2015

Thanks Jay for doing the Q&A.

Excellent.

Best

Paul

Quoting Jay Daley <jay at nzrs.net.nz>:

> I had some feedback on my initial Q&As so Iâ€™ve updated them below.  The
> changes are about a) The IANA system development; b) The plan for more data
> capture to inform the performance targets; c) the next steps.  Now that
> Iâ€™ve had some feedback, I was planning to share these later today with a
> few people who are involved in CWG-Stewardship and who are concerned about
> our progress - please let me know if this is an issue.
> 
> Jay
> 
> 
> Q:  Why has there been so little open discussion by DT-A on the mailing list?
>  It all looks very secretive?
> 
> A:  We began work some time before a DT-A mailing list was established and by
> the time it was, a lot of work on drafting had already taken place.  Soon
> after the list was established the issue of how IANA would be involved was
> resolved with the full engagement from the IANA staff, and it was felt that
> direct conversations with IANA were the best way to take advantage of that
> rather than slow conversations on the list. 
> 
> In order to engage fully IANA had to get assemble a series of documents that
> detail their internal processes and then get permission from the NTIA to
> release them.  Altogether this process took a long time.
> 
> 
> Q:  What involvement has ICANN/IANA had and how committed are they to this?
> 
> A:  Initially it appeared to us that ICANN was only going to engage at a
> senior executive level to insist that the SLA stays unchanged from that in
> place currently.  This was not acceptable to the DT-A team as we regard the
> current NTIA approved SLA as well below the minimum SLA required for a
> service of such global importance.  Fortunately IANA staff are now involved
> and they are fully committed to a proper SLA that serves the need of
> customers and drives internal improvements.  
> 
> It is useful to note that an SLE (Service Level Expectation) is generally set
> by customers alone, whereas with this full engagement from IANA it is
> possible for us to agree an SLA (Service Level Agreement) which is equally
> supported by both customers and the providers.
> 
> 
> Q:  Does the DT-A team have an agenda here, such as forcing through the
> automation of IANA?
> 
> A:  No. The team has taken the view from the outset that it is not our role
> to change the processes in any way.  IANA have already started a process of
> automation of some elements of their service, as presented at various ICANN
> meetings, and we are trying to keep up with that but we are not introducing
> any agenda of our own.
> 
> 
> Q:  What is the general plan for the SLA?
> 
> A:  Conceptually it breaks down into three parts:
> 
> 1.  A full service definition with each service broken down into constituent
> processes and stages within those processes that require individual
> measurement.
> 2.  An initial set of performance targets.
> 3.  An initial set of breach thresholds - a percentage of requests that must
> meet the performance target or IANA is in breach of the SLA.
> 
> A set of principles has been developed to guide the work on these three
> parts.
> 
> 
> Q:  What is being measured?
> 
> A:  Both time to complete a request (broken down by stage) and the accuracy
> of the work completed.  For some processes, such as nameserver changes, the
> accuracy needs to be 100% but for others a lower standard is acceptable. 
> 
> 
> Q:  Is all that measurement necessary?
> 
> A:  It is the clear consensus view of DT-A that without this level of
> measurement it is not possible to define an SLA nor is it possible to ensure
> that an SLA meets the needs of the customer.  The breakdown into stages of
> processes has been chosen to reflect the different responsibilities for those
> stages and the different nature of those stages.  For example, if IANA ask
> for more information about a root change then the clock needs to stop and the
> responsibility for progressing that request transfers from IANA to the
> requestor.  
> 
> This level of measurement is vital to resolve, once and for all, two
> different perceptions that people have of IANA performance:
> -  that IANA operates a two-tier SLA with contracted TLDs receiving a faster
> service than non-contracted TLDs while both are still within the overall
> SLA.
> -  that IANA is the primary cause of delays.  By specifying stages where the
> clock stops when responsibility transfers, this ensures an accurate
> measurement of IANA performance.
> 
> 
> Q:  How were the initial performance targets set?  
> 
> A:  It is important to note that no targets have yet been set.  Some targets
> were initially proposed based on analysis of previous IANA performance (the
> full data set used for the analysis is available) but we have agreed with
> IANA that we need more data and are hoping for 2-3 months of data capture
> from IANA.
> 
> The key takeaway is that the performance targets in current NTIA SLA are far
> looser than we as customers believe they should be and IANA as the operator
> clearly recognises this because it is over-performing by so much.
> 
> 
> Q:  Has work been done to ensure that these targets match the importance of
> the processes and are not arbitrary?  Isnâ€™t there a problem that if we use
> current performance then IANA needs to establish a base line even higher to
> ensure that they never breach those targets?
> 
> A:  The answers are yes the work has been done and no there isnâ€™t a
> problem.  These issues are addressed by the use of breach levels, which work
> in conjunction with the performance targets.  For example a performance
> target might be set for a specific type of customers request at 5 days based
> on evidential analysis, but the breach threshold might be set at 90%, which
> means that only 90% of requests of this type need to meet the 5 day target. 
> This breach threshold serves a dual purpose:
> 
> -  It provides headroom for IANA above the target by allowing them to ignore
> the worst performing requests.
> -  This level of headroom given reflects the customer priority of this type
> of request.
> 
> 
> Q:  Once these targets are set are they fixed in stone?
> 
> A:  No, these are an initial set of targets and it is expected that they will
> develop over time in two ways:
> 
> -  ongoing process improvement by IANA leading to improvements in
> performance
> -  better consultation with customers to understand our priorities better,
> leading to adjustments in targets to reflect that
> 
> 
> Q:  How important is it that these elements are in place before transition?
> 
> A:  The view of the DT-A team is that a full SLE/SLA needs to be in place by
> the start of the transition however IANA have noted that there are some
> pre-requisites that must be met before they can implement it.  These are:
> 
> 1.  The current IT system needs to have features added to extract the
> required measurements.   This development can only begin once there is
> certainty that this transition is going ahead on this basis and IANA can only
> turn those changes on after transition given the nature of their agreement
> with the NTIA.  It should also be remembered that IANA will need to make a
> case for development resources to the ICANN executive team as IANA has no
> development resources of its own.
> 2.  IANA staff will need time to adapt to the changes in their working
> practices to implement this measurement.
> 
> 
> Q:  When will the DT-A be finished?
> 
> A:  We are close to completing the SLA except the actual performance targets.
>  These will be added later after a data collection period and discussion with
> IANA.  
> 
> -- 
> Jay Daley
> Chief Executive
> NZRS Ltd
> desk: +64 4 931 6977
> mobile: +64 21 678840
> linkedin: www.linkedin.com/in/jaydaley
> 
> _______________________________________________
> dt1 mailing list
> dt1 at icann.org
> https://mm.icann.org/mailman/listinfo/dt1
> 
> 