[ICANN-CSC] [Ext] RE: Explanation of qualification of monthly PTI performance as included in CSC Reports
bart.boswinkel at icann.org
Thu Jan 31 15:34:47 UTC 2019
To be clear, my understanding was this action item resulted from a question from Brett, regarding the meaning of the qualification (Excellent, Satisfactory, and Need Improvement)
As they refer to a percentage of defined metrics met, I looked if there was any explanation. This was included in the July 2017 report.
That being said, I think the point you raise is interesting in itself. I’ll defer to Naela to explain the metrics in the PTI performance overview.
From: Allan MacGillivray <allan.macgillivray at cira.ca>
Date: Thursday, 31 January 2019 at 16:19
To: Bart Boswinkel <bart.boswinkel at icann.org>, "ICANN-CSC at icann.org" <ICANN-CSC at icann.org>
Subject: [Ext] RE: Explanation of qualification of monthly PTI performance as included in CSC Reports
Bart – can we push this a bit further, perhaps by using an actual example, as I’m not sure that I completely get it. Let’s look at the November PTI Report which gave rise to this question in the first place. The Summary of Performance (pages 2&3) indicates that the ‘Percentage of SLA Thresholds met of those defined’ was 98.4%. There are a total of 72 metrics e.g. 10 under ‘Submission’, 12 under ‘Technical Checks’ etc. Only one threshold was missed - Technical Check (Supplemental) - Routine (Technical). So 71 out of 72 were met, for a total of 98.6% (71/72), not the 98.4% reported. So, what am I missing?
Naela – maybe you are better placed to help me.
From: ICANN-CSC <icann-csc-bounces at icann.org> On Behalf Of Bart Boswinkel
Sent: Wednesday, January 30, 2019 9:37 AM
To: ICANN-CSC at icann.org
Subject: [ICANN-CSC] Explanation of qualification of monthly PTI performance as included in CSC Reports
One of the open actions (Action 01 26 2019: Secretariat to complete overview of CSC qualifications of PTI performance) was to look-up how the CSC has defined the qualifications of its findings.
In November 2016, before publishing their first Findings of PTI performance, the CSC agreed on the need for a limited set of qualifications and these were defined as follows:
Excellent - PTI met all service level agreements (100 %) for the month of [Month Year].
Satisfactory - PTI met the service level agreement for [%] of defined metrics. Missed service level agreements were satisfactorily explained and the CSC has determined that these exceptions were no cause for concern. No persistent problems were identified and no further action is needed.
Need Improvement - Performance needs improvement due to a) severe degradation in meeting SLAs from previous months, b) a trend in complaints that indicate a persistent issue to be resolved, and c) a negative trend in compliance to SLAs over several months.
Further, until June 2017, the CSC Findings also included a description of how the % compliance with defined metrics SLA was arrived at. However in the July 2017 report ( On the PTI performance over June 2017) the CSC noted:
The overall SLA compliance score is correctly identified in the PTI report as "Percentage of SLA thresholds met of those defined." However, prior to this report (BB: CSC Findings June 2017) the calculation has not followed this definition and has instead been based on the compliance of each individual request made, as documented in CSC reports. This calculation has changed for this report and subsequent reports to correctly meet the requirements of the SLA specification. The CSC has considered the reissuing of previous reports with the correct calculation but decided against it because the change would not be significant.
Ever since the description of how the % was arrived at is not included anymore, but follows the PTI report.
-------------- next part --------------
An HTML attachment was scrubbed...
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 4583 bytes
Desc: not available
More information about the ICANN-CSC