<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:-webkit-standard;
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:967591485;
mso-list-type:hybrid;
mso-list-template-ids:-1004888910 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Rick,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thinking back to the original discussions in the design team that first created the SLAs, one of the original drivers was to ensure 100% coverage of IANA time in the metrics even if it meant capturing too
much. One sentiment expressed in the original discussion was if there was any time unaccounted for, it would tempt IANA to sneak in delays in that unmeasured window. This is why we have some very small measurement windows for things that take milliseconds,
on the notion that if it wasn’t measured then that would create a loophole. (To be clear, I don’t think IANA is incentivized to make use of such loopholes if they exist!)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">For the technical checks, if you were to apply the principle more faithfully, it would likely break individual test runs into hundreds of constituent pieces where fractions of a second are measured of “IANA
time” creating and dispatching network queries, followed by customer time sending back a response to the query. There was a practical realization that tracking the ebb and flow of all these various tests was too complex so it ended up being considered all
as “IANA time”, on the notion that the overall performance metric should be generous enough to cater for customer induced delays, but also on the basis that IANA would be motivated to work out how to optimize further to compensate for customer delays.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I think the failure against these metrics is most apparent in the “retest” metric, because by definition these are only in cases where the TLDs are already having tech check problems, and retries of the test
run are happening. It is much harder to cancel out the aberrations with timely tests like with the “first” tech check run.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I don’t know the right answer to the question on how to evolve it. I will say that since 2016 we have realized significant performance gains in conducting these tests, so having the tests measured this way
has been a driver for meaningful improvement. All the quick wins have been implemented though. While there are still some optimization possibilities we see, it is diminishing returns and further optimizations potentially make our code a lot more complex and
potentially brittle.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Another factor is that late last year we started consulting with the technical community on how to evolve the technical checks to meet with modern requirements (the current tests were specified in 2007). The
general direction we are hearing is that IANA should be testing for more things. Assuming the number of tests grows, you can only see the duration for the pathological cases only going up, not down.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">kim<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Rick Wilhelm <Rwilhelm@PIR.org><br>
<b>Date: </b>Thursday, August 10, 2023 at 7:57 AM<br>
<b>To: </b>Kim Davies <kim.davies@iana.org>, Amy Creamer <amy.creamer@iana.org>, Bart Boswinkel via ICANN-CSC <icann-csc@icann.org><br>
<b>Subject: </b>[Ext] Re: [ICANN-CSC] July 2023 IANA Naming Function Performance Report<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks Kim, for the detail about the test.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I think that the point that I’m trying to make is that the IANA SLAs are there to measure the effectiveness of the IANA function and things that are within its responsibility.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">In this case, the timer on the SLA is inclusive of time that is outside of the IANA demarcation of responsibility. And thus, as we’ve seen, IANA can be doing a perfectly good job (even doing a fair bit of
parallelism), and because of non-responsiveness the server on the other end.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’m wondering if there is some sort of a factor that can/should be added to account for latency that is not due to IANA?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thx<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Rick<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Kim Davies <kim.davies@iana.org><br>
<b>Date: </b>Wednesday, August 9, 2023 at 3:41 PM<br>
<b>To: </b>Rick Wilhelm <Rwilhelm@PIR.org>, Amy Creamer <amy.creamer@iana.org>, Bart Boswinkel via ICANN-CSC <icann-csc@icann.org><br>
<b>Subject: </b>[EXTERNAL] Re: [ICANN-CSC] July 2023 IANA Naming Function Performance Report<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><strong><span style="font-size:9.0pt;font-family:Helvetica;color:black;background:yellow">CAUTION: This email came from outside your organization. Don’t trust emails, links, or attachments from senders that seem suspicious or you are not
expecting.</span></strong><span style="font-size:9.0pt;font-family:Helvetica;color:black"><o:p></o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:11.0pt">
<hr size="0" width="100%" align="center">
</span></div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Rick, Hi all,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Generally speaking, a lack of network response from one or more nameservers has a compounding effect across a test run in our systems. Assuming that most nameservers have both an IPv4 and IPv6 address, this
means that we would send four queries (we test each IP address via both TCP and UDP), and then we retry them 3 times before giving up for each query. The current timeout is set to 5 seconds, so this means a minimum of 60 seconds of test time eaten up for each
unresponsive nameserver per sub-test. There are efficiencies through parallel execution of the tests, but that is offset by the fact we query for different kinds of records throughout a test run (SOA, DNSKEY, NS, A/AAAA, RD-bit set). In more pathological cases,
if there are multiple nameservers all in the same network that are unreachable, it can multiply the test time further.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I looked at one test run from the individual case that caused us to exceed our SLAs last month and there were a total of 316 DNS queries sent that were not responded to throughout the course of that one test
run, which took 14 minutes and 40 seconds to complete.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks for the reference to Spec 10, it seems the pertinent standard is <1500ms response for TCP and <500ms response for UDP, for 95% of queries. Since our lookups are a one-shot real-time blocking operation,
as opposed to passive ongoing tests done around the clock for gTLD SLA monitoring, it is a bit of a different proposition. Also I understand ICANN does SLA monitoring for multiple sites and aggregates the performance, which is not something we do in IANA today.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Happy to discuss this further.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">kim</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">ICANN-CSC <icann-csc-bounces@icann.org> on behalf of Rick Wilhelm via ICANN-CSC <icann-csc@icann.org><br>
<b>Reply-To: </b>Rick Wilhelm <Rwilhelm@PIR.org><br>
<b>Date: </b>Wednesday, August 9, 2023 at 4:38 AM<br>
<b>To: </b>Amy Creamer <amy.creamer@iana.org>, Bart Boswinkel via ICANN-CSC <icann-csc@icann.org>, Bart Boswinkel via ICANN-CSC <icann-csc@icann.org><br>
<b>Subject: </b>Re: [ICANN-CSC] July 2023 IANA Naming Function Performance Report</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Amy, et al,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks for sending over the report. This might be a better topic for discussion at the next meeting meting than for the list, but I’ll try to frame it coherently:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Regarding the missed SLA:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">From what I can gather in reading the footnote, it seems that during the execution of the “One request (that) exceeded the technical check threshold of 10 minutes”, operations happened normally (i.e. kicked
off normally, experienced no IANA-induced interruptions, etc), and the extra time was spent waiting.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">It seems to me that the “technical check (retest)” should be designed with a DNS query timeout value that is sufficiently short such that waiting for the timeout(s) to expire does not cause the SLA to be violated.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’m not familiar with the exact details of the test design, but I’d point folks to the Base Registry Agreement, Specification 10 (</span><a href="https://urldefense.com/v3/__https:/protect-us.mimecast.com/s/u0LtCzpyrXSw8lwSXxy4D?domain=urldefense.com__;!!PtGJab4!7h8_obiyzfoODB9JnZyAMQXnEulacUCIJK4JWApuYJPd7QqfUSyjPtjvEnK9CjxuZrXr1QH50e54H0BaSEKFfXY$"><span style="font-size:11.0pt">https://itp.cdn.icann.org/en/files/registry-agreements/base-registry-agreement-30-04-2023-en.html
[itp.cdn.icann.org]</span> [protect-us.mimecast.com]</a><span style="font-size:11.0pt">) search for “will be considered unanswered” for examples of language that contemplate non-responsive services</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Happy to discuss.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Rick</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">ICANN-CSC <icann-csc-bounces@icann.org> on behalf of Amy Creamer via ICANN-CSC <icann-csc@icann.org><br>
<b>Date: </b>Tuesday, August 8, 2023 at 1:39 PM<br>
<b>To: </b>Bart Boswinkel via ICANN-CSC <icann-csc@icann.org><br>
<b>Subject: </b>[EXTERNAL] [ICANN-CSC] July 2023 IANA Naming Function Performance Report</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><strong><span style="font-size:9.0pt;font-family:Helvetica;color:black;background:yellow">CAUTION: This email came from outside your organization. Don’t trust emails, links, or attachments from senders that seem suspicious or you are not
expecting.</span></strong><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:11.0pt">
<hr size="0" width="100%" align="center">
</span></div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:-webkit-standard;color:black">Dear CSC,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:-webkit-standard;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:-webkit-standard;color:black">Please find attached the IANA Naming Function Performance report for July 2023. During the month of July, we met 98.3% of the SLA thresholds. This was due to missing
the SLA of:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:-webkit-standard;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:-webkit-standard;color:black">Technical Check (Retest) - Routine (Technical): One change request had nameservers that were unreachable within the technical check threshold of 10 minutes. This exception
relates to time spent waiting for nameserver responses, i.e. time waiting to timeout, multiplied by retries.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:-webkit-standard;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:-webkit-standard;color:black">We look forward to answering any questions you may have about the report.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Regards,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Amy Creamer</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Director of Operations, IANA Services</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Email: </span><a href="mailto:amy.creamer@iana.org"><span style="font-size:11.0pt">amy.creamer@iana.org</span></a><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Phone: +1-424-537-8917
</span><o:p></o:p></p>
</div>
</body>
</html>