<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal><span style='font-size:14.0pt'>Hi Allan,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:14.0pt'>To be clear, my understanding was this action item resulted from a question from Brett, regarding the meaning of the qualification (Excellent, Satisfactory, and Need Improvement)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:14.0pt'>As they refer to a percentage of defined metrics met, I looked if there was any explanation. This was included in the July 2017 report.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:14.0pt'>That being said, I think the point you raise is interesting in itself. I’ll defer to Naela to explain the metrics in the PTI performance overview.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:14.0pt'>Kind regards,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:14.0pt'>Bart<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:14.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:14.0pt'><o:p> </o:p></span></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>Allan MacGillivray <allan.macgillivray@cira.ca><br><b>Date: </b>Thursday, 31 January 2019 at 16:19<br><b>To: </b>Bart Boswinkel <bart.boswinkel@icann.org>, "ICANN-CSC@icann.org" <ICANN-CSC@icann.org><br><b>Subject: </b>[Ext] RE: Explanation of qualification of monthly PTI performance as included in CSC Reports<o:p></o:p></span></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><p class=MsoNormal>Bart – can we push this a bit further, perhaps by using an actual example, as I’m not sure that I completely get it. Let’s look at the November PTI Report which gave rise to this question in the first place. The Summary of Performance (pages 2&3) indicates that the ‘Percentage of SLA Thresholds met of those defined’ was 98.4%. There are a total of 72 metrics e.g. 10 under ‘Submission’, 12 under ‘Technical Checks’ etc. Only one threshold was missed - Technical Check (Supplemental) - Routine (Technical). So 71 out of 72 were met, for a total of 98.6% (71/72), not the 98.4% reported. So, what am I missing? <o:p></o:p></p><p class=MsoNormal>Naela – maybe you are better placed to help me.<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>Thanks<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>Allan<o:p></o:p></p><p class=MsoNormal><span style='color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal><span style='color:#1F497D'> </span><o:p></o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b>From:</b> ICANN-CSC <icann-csc-bounces@icann.org> <b>On Behalf Of </b>Bart Boswinkel<br><b>Sent:</b> Wednesday, January 30, 2019 9:37 AM<br><b>To:</b> ICANN-CSC@icann.org<br><b>Subject:</b> [ICANN-CSC] Explanation of qualification of monthly PTI performance as included in CSC Reports<o:p></o:p></p></div></div><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal><span style='font-size:14.0pt'>Dear all,</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:14.0pt'>One of the open actions (<span style='color:black'>Action 01 26 2019: Secretariat to complete overview of CSC qualifications of PTI performance)</span> was to look-up how the CSC has defined the qualifications of its findings.</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:14.0pt'>In November 2016, before publishing their first Findings of PTI performance, the CSC agreed on the need for a limited set of qualifications and these were defined as follows:</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:14.0pt'> </span><o:p></o:p></p><p class=MsoNormal><b><span style='font-size:14.0pt'>Excellent </span></b><span style='font-size:14.0pt'>- PTI met all service level agreements (100 %) for the month of [Month Year].</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:14.0pt'> </span><o:p></o:p></p><p class=MsoNormal><b><span style='font-size:14.0pt'>Satisfactory</span></b><span style='font-size:14.0pt'> - PTI met the service level agreement for [%] of defined metrics. Missed service level agreements were satisfactorily explained and the CSC has determined that these exceptions were no cause for concern. No persistent problems were identified and no further action is needed.</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:14.0pt'> </span><o:p></o:p></p><p class=MsoNormal><b><span style='font-size:14.0pt'>Need Improvement</span></b><span style='font-size:14.0pt'> - Performance needs improvement due to a) severe degradation in meeting SLAs from previous months, b) a trend in complaints that indicate a persistent issue to be resolved, and c) a negative trend in compliance to SLAs over several months.</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:14.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:14.0pt'>Further, until June 2017, the CSC Findings also included a description of how the % compliance with defined metrics SLA was arrived at. However in the July 2017 report ( On the PTI performance over June 2017) the CSC noted: </span><o:p></o:p></p><p class=MsoNormal><i><span style='font-size:14.0pt'>The overall SLA compliance score is correctly identified in the PTI report as "Percentage of SLA thresholds met of those defined." However, prior to this report (BB: CSC Findings June 2017) the calculation has not followed this definition and has instead been based on the compliance of each individual request made, as documented in CSC reports. This calculation has changed for this report and subsequent reports to correctly meet the requirements of the SLA specification. The CSC has considered the reissuing of previous reports with the correct calculation but decided against it because the change would not be significant.</span></i><o:p></o:p></p><p class=MsoNormal><span style='font-size:14.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:14.0pt'>Ever since the description of how the % was arrived at is not included anymore, but follows the PTI report.</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:14.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:14.0pt'>Kind regards,</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:14.0pt'>Bart</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:14.0pt'> </span><o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p></div></body></html>