[Gnso-newgtld-wg] SLA Monitoring (SLAM) Statistics

Mon Sep 16 14:06:50 UTC 2019

Following up on the conversation held at the end of the last call.

Jeff raised some very good points as to why having this data is important.

GDD’s position that “its hard to get” just doesn’t make sense.  1) GDD clearly has this data as they created the charts in the slides.  Noting that some of the charts have issues, having the raw data behind those would yield some better insight.  2) its in ICANN’s benefit to explain this with some context as opposed to throwing out raw number without any narrative.

My continued concern is that it’s clear that some existing registries (and by extension their back end service providers) still are having ongoing issues with hitting SLAs.  And clearly there was a major event in the not too distant past that we have no context about.

I simply don’t know how ICANN can designate someone as “pre-approved” if these operational challenges are evident and ongoing.

From: Jim Prendergast
Sent: Friday, September 06, 2019 11:01 AM
To: gnso-newgtld-wg at icann.org
Subject: FW: SLA Monitoring (SLAM) Statistics

It appears that my previous message was never circulated via the list as I had an incorrect email.  This is take 2.

While the topic did not come up on the aforementioned call, it is still important information for the WG to have as it considers other parts of the program including pre-approval programs.

thanks

From: Jim Prendergast
Sent: Thursday, August 29, 2019 5:34 PM
To: Austin, Donna <Donna.Austin at team.neustar<mailto:Donna.Austin at team.neustar>>; Steve Chan <steve.chan at icann.org<mailto:steve.chan at icann.org>>; Gnso-newgtld-wg <gnso-newgtld-wg-bounces at icann.org<mailto:gnso-newgtld-wg-bounces at icann.org>>
Subject: RE: SLA Monitoring (SLAM) Statistics

As just referenced on the call (in my best impression of Mickey Mouse voice) see my note below as well as the additions here as I’ve had more time to look at the slides.

We really need the raw data and some narrative.  If this will be covered on the call on Tuesday – I move that we table that discussion until we have this information and have had a chance to review and digest.  Sending it out late tomorrow before a holiday weekend for many of the regular participants simply would not provide enough prep time.

As I looked at the slide more a few thoughts

  *   Even the areas that are outside the ‘significant spikes’ are interesting as they represent non-trivial levels of failure.
  *   The DNS had 10+ failures in 23 of 58 quarters (~40%).  And scale on the RDDS graph is such that the lowest line is 200 failures per quarter and the line is above that for all but two quarters over nearly 5 years.
  *   And, of course, all of this data is less informative because it’s an aggregated industry view.  The reader can’t tell which RSP is involved for any of these failures.  If I was shopping for an ICANN “pre-approved” vendor – I would certainly want access to that data to ensure I factored that into my decision making
  *   Slide 2 is missing May 2019, which could or could not be significant, given the performance of June and July.

Thanks

From: Jim Prendergast
Sent: Monday, August 26, 2019 11:19 PM
To: Austin, Donna <Donna.Austin at team.neustar<mailto:Donna.Austin at team.neustar>>; Steve Chan <steve.chan at icann.org<mailto:steve.chan at icann.org>>; Gnso-newgtld-wg <gnso-newgtld-wg-bounces at icann.org<mailto:gnso-newgtld-wg-bounces at icann.org>>
Subject: RE: SLA Monitoring (SLAM) Statistics

Yes.  Add my thanks for this as well.

And I agree with Donna on the need for more detail.

I think having the raw data behind the slides would be beneficial as when I look at the chart on slide 4, the timeline seems to be out of sorts, especially as we get to 2019.

It would also be useful to know if there is any explanation behind the spikes in DNS failures earlier this year and RDDS in 2018.  The DNS failures are really something else.  More in June and July than in the entire cumulative history of the program.

Thanks

From: Gnso-newgtld-wg <gnso-newgtld-wg-bounces at icann.org<mailto:gnso-newgtld-wg-bounces at icann.org>> On Behalf Of Austin, Donna via Gnso-newgtld-wg
Sent: Monday, August 26, 2019 7:16 PM
To: Steve Chan <steve.chan at icann.org<mailto:steve.chan at icann.org>>; gnso-newgtld-wg at icann.org<mailto:gnso-newgtld-wg at icann.org>
Subject: Re: [Gnso-newgtld-wg] SLA Monitoring (SLAM) Statistics

Steve, thanks for providing this update.

The information provided is not as ‘complete’ as that provided to the group in 2017. I’ve attached a document from an RySG Discussion Group that I believe was provided to this group early last year that summarises the level of detail that was available at that time—I think more detail would be helpful for the purposes of this group. Generalisations are not helpful either—

It’s also disappointing that only ‘anecdotal’ information is available about PDT, which I personally don’t find particularly helpful to our discussion because the source doesn’t include any insight from those that were subject to PDT. For this information to be meaningful, more detailed statistics and analysis is necessary.

Donna

From: Gnso-newgtld-wg [mailto:gnso-newgtld-wg-bounces at icann.org] On Behalf Of Steve Chan
Sent: Friday, August 23, 2019 2:26 PM
To: gnso-newgtld-wg at icann.org<mailto:gnso-newgtld-wg at icann.org>
Subject: [Gnso-newgtld-wg] SLA Monitoring (SLAM) Statistics

Dear WG Members,

During the course of discussions, particularly as it relates to RSP pre-approval and most recently, registrant protections, WG members have asked for updated statistics on the Service Level Agreement (SLA) Monitoring program within ICANN. Attached, please find the latest report. A few contextual notes that may help your digestion of this information:

  *   The statistics are on a per TLD basis for each month.
  *   In some months, there are spikes in the number of DNS and RDDS failures. Usually, these spikes are a caused by a failure in an RSP that serves a large number of TLDs.
  *   On the 4ht slide, you will see several failures that reached EBERO thresholds but in the vast majority of cases, the failures did not trigger an EBERO event. As has been shared previously, this is likely because ICANN Org worked with the registry to resolve the event and/or there were minimal domains under management.

In addition, members asked for information around Pre-Delegation Testing (PDT), which I believe is now an element of the broader Registry System Testing (RST). Please see the below response from GDD:

“Additionally, on the PDT/RST side - ICANN org has neither compiled nor published summary data regarding Pre-Delegation Test results from the 2012 round.  Testing data was tracked and managed by IIS, the pre-delegation testing vendor.   Such testing data, was not provided to ICANN org as part of the service delivery efforts.  However, based on ICANN org management of the Pre-Delegation testing processing since 2012, we can share anecdotal information based on the collective recollection of org and vendor staff.  These anecdotal results includes:

1.       Most TLDs received follow-up questions regarding their PDT testing submission (similar to initial evaluation Clarifying Questions)
2.       Approximately one quarter of TLDs tested required extended testing beyond the standard 2 to 3 week testing cycle.  Some of these tests extended up to 12 weeks.
3.       Based on the flexibility of extended testing and the assistance provided by the PDT vendor to TLD operators, fewer than 10 TLDs required a second full PDT testing cycle.“

Hopefully you all find this information helpful. If you have questions or concerns, especially for GDD, please let us know.

Best,
Steve

Steven Chan

Policy Director, GNSO Support

Internet Corporation for Assigned Names and Numbers (ICANN)
12025 Waterfront Drive, Suite 300
Los Angeles, CA 90094-2536

Email: steve.chan at icann.org<mailto:steve.chan at icann.org>
Skype: steve.chan55
Mobile: +1.310.339.4410

Find out more about the GNSO by visiting: https://learn.icann.org/<https://urldefense.proofpoint.com/v2/url?u=https-3A__learn.icann.org_&d=DwMGaQ&c=FmY1u3PJp6wrcrwll3mSVzgfkbPSS6sJms7xcl4I5cM&r=DRa2dXAvSFpCIgmkXhFzL7ar9Qfqa0AIgn-H4xR2EBk&m=jLNFXvpu9gNdUeHi-G6sjWNCF9w4_AwhzzUDFZy2elE&s=o7Auz997kA-HPv9PHJCjFVZw7Pgo8krw4MxfqCwBrIU&e=>
Follow @GNSO on Twitter: https://twitter.com/ICANN_GNSO<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_ICANN-5FGNSO&d=DwMGaQ&c=FmY1u3PJp6wrcrwll3mSVzgfkbPSS6sJms7xcl4I5cM&r=DRa2dXAvSFpCIgmkXhFzL7ar9Qfqa0AIgn-H4xR2EBk&m=jLNFXvpu9gNdUeHi-G6sjWNCF9w4_AwhzzUDFZy2elE&s=kWw4fQPNjw2lVKy1UjTxS2F0BmjEAzaDFWNmsYywbmE&e=>
Transcripts and recordings of GNSO Working Group and Council events are located on the GNSO Master Calendar <https://urldefense.proofpoint.com/v2/url?u=https-3A__gnso.icann.org_en_group-2Dactivities_calendar&d=DwMGaQ&c=FmY1u3PJp6wrcrwll3mSVzgfkbPSS6sJms7xcl4I5cM&r=DRa2dXAvSFpCIgmkXhFzL7ar9Qfqa0AIgn-H4xR2EBk&m=jLNFXvpu9gNdUeHi-G6sjWNCF9w4_AwhzzUDFZy2elE&s=-L6chFfv0OperrXHHpTF722WnH3FZIutn4cS16IvpOg&e=>
See All SO and AC events on the ICANN Global calendar [features.icann.org]<https://urldefense.proofpoint.com/v2/url?u=https-3A__features.icann.org_calendar&d=DwMGaQ&c=FmY1u3PJp6wrcrwll3mSVzgfkbPSS6sJms7xcl4I5cM&r=UAy6fqdE7uFkRCc7uzN4yui8bwTtqofadZHiQEIO1vw&m=-3j0Rpfrf9CXzHr5zFQOxH1P0b4MMztS94_2eyzkY8M&s=9-TBAhNihyEZfKWVWBt0IcfR5bI6SIzBo1ddDmXfB9Q&e=>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mm.icann.org/pipermail/gnso-newgtld-wg/attachments/20190916/345e023d/attachment-0001.html>