[CWG-Stewardship] Fwd: [DTA - SLE] Questions and Answers - Revised

[Paul M Kane - CWG]

Dear CWG members,

Jay Daley (.NZ) has prepared some excellent Q&A to update the community on the
work of the SLE Working Group which may be of interest.

Rest assured we are trying to progress the drafting of the SLE which should be
concluded in the next few weeks so ICANN / IANA has a framework for monitoring
their systems as part of the Implementation Phase. 

Kind regards

Paul

----- Forwarded message from Jay Daley <jay at nzrs.net.nz> -----
    Date: Mon, 22 Jun 2015 15:29:26 -0300
    From: Jay Daley <jay at nzrs.net.nz>
Reply-To: Jay Daley <jay at nzrs.net.nz>
 Subject: [DTA - SLE] Questions and Answers - Revised
      To: dt1 at icann.org

I had some feedback on my initial Q&As so I’ve updated them below.  The
changes are about a) The IANA system development; b) The plan for more data
capture to inform the performance targets; c) the next steps.  Now that I’ve
had some feedback, I was planning to share these later today with a few people
who are involved in CWG-Stewardship and who are concerned about our progress -
please let me know if this is an issue.

Jay


Q:  Why has there been so little open discussion by DT-A on the mailing list? 
It all looks very secretive?

A:  We began work some time before a DT-A mailing list was established and by
the time it was, a lot of work on drafting had already taken place.  Soon after
the list was established the issue of how IANA would be involved was resolved
with the full engagement from the IANA staff, and it was felt that direct
conversations with IANA were the best way to take advantage of that rather than
slow conversations on the list. 

In order to engage fully IANA had to get assemble a series of documents that
detail their internal processes and then get permission from the NTIA to release
them.  Altogether this process took a long time.


Q:  What involvement has ICANN/IANA had and how committed are they to this?

A:  Initially it appeared to us that ICANN was only going to engage at a senior
executive level to insist that the SLA stays unchanged from that in place
currently.  This was not acceptable to the DT-A team as we regard the current
NTIA approved SLA as well below the minimum SLA required for a service of such
global importance.  Fortunately IANA staff are now involved and they are fully
committed to a proper SLA that serves the need of customers and drives internal
improvements.  

It is useful to note that an SLE (Service Level Expectation) is generally set by
customers alone, whereas with this full engagement from IANA it is possible for
us to agree an SLA (Service Level Agreement) which is equally supported by both
customers and the providers.


Q:  Does the DT-A team have an agenda here, such as forcing through the
automation of IANA?

A:  No. The team has taken the view from the outset that it is not our role to
change the processes in any way.  IANA have already started a process of
automation of some elements of their service, as presented at various ICANN
meetings, and we are trying to keep up with that but we are not introducing any
agenda of our own.


Q:  What is the general plan for the SLA?

A:  Conceptually it breaks down into three parts:

1.  A full service definition with each service broken down into constituent
processes and stages within those processes that require individual measurement.
2.  An initial set of performance targets.
3.  An initial set of breach thresholds - a percentage of requests that must
meet the performance target or IANA is in breach of the SLA.

A set of principles has been developed to guide the work on these three parts.


Q:  What is being measured?

A:  Both time to complete a request (broken down by stage) and the accuracy of
the work completed.  For some processes, such as nameserver changes, the
accuracy needs to be 100% but for others a lower standard is acceptable. 


Q:  Is all that measurement necessary?

A:  It is the clear consensus view of DT-A that without this level of
measurement it is not possible to define an SLA nor is it possible to ensure
that an SLA meets the needs of the customer.  The breakdown into stages of
processes has been chosen to reflect the different responsibilities for those
stages and the different nature of those stages.  For example, if IANA ask for
more information about a root change then the clock needs to stop and the
responsibility for progressing that request transfers from IANA to the requestor.  

This level of measurement is vital to resolve, once and for all, two different
perceptions that people have of IANA performance:
-  that IANA operates a two-tier SLA with contracted TLDs receiving a faster
service than non-contracted TLDs while both are still within the overall SLA.
-  that IANA is the primary cause of delays.  By specifying stages where the
clock stops when responsibility transfers, this ensures an accurate measurement
of IANA performance.


Q:  How were the initial performance targets set?  

A:  It is important to note that no targets have yet been set.  Some targets
were initially proposed based on analysis of previous IANA performance (the full
data set used for the analysis is available) but we have agreed with IANA that
we need more data and are hoping for 2-3 months of data capture from IANA.

The key takeaway is that the performance targets in current NTIA SLA are far
looser than we as customers believe they should be and IANA as the operator
clearly recognises this because it is over-performing by so much.


Q:  Has work been done to ensure that these targets match the importance of the
processes and are not arbitrary?  Isn’t there a problem that if we use current
performance then IANA needs to establish a base line even higher to ensure that
they never breach those targets?

A:  The answers are yes the work has been done and no there isn’t a problem. 
These issues are addressed by the use of breach levels, which work in
conjunction with the performance targets.  For example a performance target
might be set for a specific type of customers request at 5 days based on
evidential analysis, but the breach threshold might be set at 90%, which means
that only 90% of requests of this type need to meet the 5 day target.  This
breach threshold serves a dual purpose:

-  It provides headroom for IANA above the target by allowing them to ignore the
worst performing requests.
-  This level of headroom given reflects the customer priority of this type of
request.


Q:  Once these targets are set are they fixed in stone?

A:  No, these are an initial set of targets and it is expected that they will
develop over time in two ways:

-  ongoing process improvement by IANA leading to improvements in performance
-  better consultation with customers to understand our priorities better,
leading to adjustments in targets to reflect that


Q:  How important is it that these elements are in place before transition?

A:  The view of the DT-A team is that a full SLE/SLA needs to be in place by the
start of the transition however IANA have noted that there are some
pre-requisites that must be met before they can implement it.  These are:

1.  The current IT system needs to have features added to extract the required
measurements.   This development can only begin once there is certainty that
this transition is going ahead on this basis and IANA can only turn those
changes on after transition given the nature of their agreement with the NTIA. 
It should also be remembered that IANA will need to make a case for development
resources to the ICANN executive team as IANA has no development resources of
its own.
2.  IANA staff will need time to adapt to the changes in their working practices
to implement this measurement.


Q:  When will the DT-A be finished?

A:  We are close to completing the SLA except the actual performance targets. 
These will be added later after a data collection period and discussion with IANA.  

-- 
Jay Daley
Chief Executive
NZRS Ltd
desk: +64 4 931 6977
mobile: +64 21 678840
linkedin: www.linkedin.com/in/jaydaley

_______________________________________________
dt1 mailing list
dt1 at icann.org
https://mm.icann.org/mailman/listinfo/dt1


----- End forwarded message -----