[ICANN-CSC] [Ext] RE: Proposed standard language qualification PTI Performance

Tue May 12 13:44:36 UTC 2020

Dear Nigel,

Thank you for your comment. 

Going over the proposed language again to understand your point, I think it can be resolved fairly easily:

Satisfactory- PTI’s performance over [MONTH] was [insert %]. PTI met [insert number] of the currently defined [INSERT TOTAL NUMBER] thresholds.  Missing the threshold(s) [INSERT NAME OF THRESHOLD(S)] was satisfactorily explained and not an indication of a performance issue. [ADDITIONAL NOTE, Explanation]. The current list of thresholds is included in PTI’s performance report.

Applying it to the April 2018 findings: 

PTI’s Perfomance over April 2018 was 95,2%. PTI met 62 of the current defined 65 thresholds. Missing the thresholds i. Technical Check, ii. Technical Check (Supplemental) and iii  Validation and Reviews (ccTLD Creation/Transfer) was satisfactorily explained and not an indication of a performance issue. 

The first two missed service levels are subject to a CSC recommendation that would re-categorize this month’s performance for these metrics as ‘met’. The third metric would require a year’s worth of data under the recommended change, which is not yet available. On the evidence so far, the CSC does not regard this as a cause for concern. Please refer to the Exceptions and Narrative for Reporting Period section of the March 2018 PTI performance report for a more detailed explanation of the missed SLAs. 

The current list of thresholds is included in PTI’s Performance report.

Original April 2018 Findings

Satisfactory- PTI met the service level agreement at 95.3% for the month of March 2018. Missed service level agreements that were satisfactorily explained and not an indication of a performance issue: 
a)  Technical Check (Retest) 
b)  Technical Check (Supplemental) 
c)  Validation and Reviews (ccTLD Creation/Transfer) 
The first two missed service levels are subject to a CSC recommendation that would re-categorize this month’s performance for these metrics as ‘met’. The third metric would require a year’s worth of data under the recommended change, which is not yet available. 

On the evidence so far, the CSC does not regard this as a cause for concern. 

Please refer to the Exceptions and Narrative for Reporting Period section of the March 2018 PTI performance report for a more detailed explanation of the missed SLAs. 

Kind regards,

Bart

From: Nigel Cassimire <Nigel.Cassimire at ctu.int>
Date: Monday, 11 May 2020 at 22:02
To: Bart Boswinkel <bart.boswinkel at icann.org>, "ICANN-CSC at icann.org" <icann-csc at icann.org>
Subject: [Ext] RE: Proposed standard language qualification PTI Performance

Dear Bart,

Thanks for this. I’m generally satisfied with the proposed standardised language which I think is superior to the existing. I have one nagging doubt though as regards the “Satisfactory” text which does not seem to anticipate or cater for more than one missed threshold. I know it’s been since April 2018 that we have had more than one but maybe the standardised language could still cater for this possibility. What about trying the April 2018 report as an example?

Nigel Cassimire | Telecommunications Specialist
[ctu.int] Caribbean Telecommunications Union [facebook.com] [twitter.com] [youtube.com]
4 Mary Street, St. Clair | Port of Spain | Trinidad and Tobago Tel: (+1 868) 628-0281 ext. 235 | Fax: (+1 868) 622-6523 Website: http://ctu.int [ctu.int] 

From: ICANN-CSC [mailto:icann-csc-bounces at icann.org] On Behalf Of Bart Boswinkel
Sent: Monday, May 11, 2020 12:06 PM
To: ICANN-CSC at icann.org
Subject: [ICANN-CSC] Proposed standard language qualification PTI Performance

Proposed Update CSC Findings Qualifications “Excellent” and “Satisfactory”

In 2016 the CSC and PTI agreed on standardized language to qualify the performance of PTI. Recently the CSC requested to clarify the qualification of PTI’s performance. The CSC and PTI recently agreed to additionally include a reference in PTI’s monthly report to the number of Thresholds/SLAs PTI has met out of total number of Thresholds/SLA.  At its April meeting the CSC agreed to include a reference in their monthly Findings of PTIs Performance and change the standard language to that effect. Going forward, starting in May 2020 following generic language is proposed to designate the findings of the CSC of PTI’s performance. The changes are marked and the Original language is included  below
Excellent- PTI’s performance over [MONTH] was 100%. PTI met all [insert number] of the currently defined thresholds over [MONTH]. The current list of thresholds is included in PTI’s performance report.

Satisfactory- PTI’s performance over [MONTH] was [insert %]. PTI met [insert number] of the currently defined [INSERT TOTAL NUMBER] thresholds.  Missing the threshold [INSERT NAME OF THRESHOLD] was satisfactorily explained and not an indication of a performance issue. [ADDITIONAL NOTE, Explaining]. The current list of thresholds is included in PTI’s performance report. 

Needs Improvement - Performance needs improvement due to a) severe degradation in meeting the number of thresholds from previous months, b) a trend in complaints that indicate a persistent issue to be resolved, or c) a negative trend in compliance to thresholds over several months. The current list of thresholds is included in PTI’s performance report.

Original Language
Excellent - PTI met all service level agreements (100 %) for the month.
Satisfactory - PTI met the service level agreement for [less than 100%] of defined metrics. Missed service level agreements were satisfactorily explained and the CSC has determined that these exceptions were no cause for concern. No persistent problems were identified and no further action is needed.
Needs Improvement - Performance needs improvement due to a) severe degradation in meeting SLAs from previous months, b) a trend in complaints that indicate a persistent issue to be resolved, and c) a negative trend in compliance to SLAs over several months.

Applying the proposed qualification Using the proposed language in recent CSC Findings report, they would:
Excellent- PTI’s performance over March 2020 was 100%. PTI met all [65] of the currently defined thresholds over March 2020. The current list of thresholds is included in PTI’s performance report.

Satisfactory- PTI’s performance over January 2020 was 98.5%. PTI met 64 of the currently defined 65 thresholds. Missing the threshold Validation and Reviews (ccTLD Creation/Transfer) was satisfactory explained and not an indication of a performance issue. This threshold is subject to a CSC recommendation to introduce new measurements to create a more meaningful performace measurement for this process. The current list of thresholds is included in PTI’s performance report.

Examples of Recent qualifications in CSC Findings of PTI Performance
Findings February 2020: Excellent- PTI met the service level agreement at 100% for the month of February 2020.

Findings January 2020: Satisfactory- PTI met the service level agreement at 98.5% for the month of January 2020.
Validation and Reviews (ccTLD Creation/Transfer): This missed service level is subject to a CSC recommendation to introduce new measurements to create a more meaningful performace measurement for this process.

Needs Improvement – No example. Has never been used to date.  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mm.icann.org/pipermail/icann-csc/attachments/20200512/1377cba1/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 2327 bytes
Desc: not available
URL: <http://mm.icann.org/pipermail/icann-csc/attachments/20200512/1377cba1/image001-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 803 bytes
Desc: not available
URL: <http://mm.icann.org/pipermail/icann-csc/attachments/20200512/1377cba1/image002-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.jpg
Type: image/jpeg
Size: 815 bytes
Desc: not available
URL: <http://mm.icann.org/pipermail/icann-csc/attachments/20200512/1377cba1/image003-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.jpg
Type: image/jpeg
Size: 849 bytes
Desc: not available
URL: <http://mm.icann.org/pipermail/icann-csc/attachments/20200512/1377cba1/image004-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4583 bytes
Desc: not available
URL: <http://mm.icann.org/pipermail/icann-csc/attachments/20200512/1377cba1/smime-0001.p7s>