[ICANN-CSC] Explanation of qualification of monthly PTI performance as included in CSC Reports

[Allan MacGillivray]

Bart – can we push this a bit further, perhaps by using an actual example, as I’m not sure that I completely get it.  Let’s look at the November PTI Report which gave rise to this question in the first place.  The Summary of Performance (pages 2&3) indicates that the ‘Percentage of SLA Thresholds met of those defined’ was 98.4%.  There are a total of 72 metrics e.g. 10 under ‘Submission’, 12 under ‘Technical Checks’ etc.  Only one threshold was missed - Technical Check (Supplemental) - Routine (Technical).  So 71 out of 72 were met, for a total of 98.6% (71/72), not the 98.4% reported.  So, what am I missing?
Naela – maybe you are better placed to help me.

Thanks

Allan


From: ICANN-CSC <icann-csc-bounces at icann.org> On Behalf Of Bart Boswinkel
Sent: Wednesday, January 30, 2019 9:37 AM
To: ICANN-CSC at icann.org
Subject: [ICANN-CSC] Explanation of qualification of monthly PTI performance as included in CSC Reports

Dear all,
One of the open actions  (Action 01 26 2019: Secretariat to complete overview of CSC qualifications of PTI performance) was to look-up how the CSC has defined the qualifications of its findings.
In November 2016, before publishing their first Findings of PTI performance, the CSC agreed on the need for a limited set of qualifications and these were defined as follows:

Excellent - PTI met all service level agreements (100 %) for the month of [Month Year].

Satisfactory - PTI met the service level agreement for [%] of defined metrics. Missed service level agreements were satisfactorily explained and the CSC has determined that these exceptions were no cause for concern. No persistent problems were identified and no further action is needed.

Need Improvement - Performance needs improvement due to a) severe degradation in meeting SLAs from previous months, b) a trend in complaints that indicate a persistent issue to be resolved, and c) a negative trend in compliance to SLAs over several months.

Further, until June 2017, the CSC Findings also included a description of how the % compliance with defined metrics SLA was arrived at. However in the July 2017 report ( On the PTI performance over June 2017) the CSC noted:
The overall SLA compliance score is correctly identified in the PTI report as "Percentage of SLA thresholds met of those defined."  However, prior to this report (BB: CSC Findings June 2017)  the calculation has not followed this definition and has instead been based on the compliance of each individual request made, as documented in CSC reports. This calculation has changed for this report and subsequent reports to correctly meet the requirements of the SLA specification. The CSC has considered the reissuing of previous reports with the correct calculation but decided against it because the change would not be significant.

Ever since the description of how the % was arrived at is not included anymore, but follows the PTI report.

Kind regards,
Bart


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mm.icann.org/pipermail/icann-csc/attachments/20190131/f7f92429/attachment-0001.html>