<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Matt,<div class=""><br class=""></div><div class="">We might not come to agreement on this, particularly not in this public forum.</div><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Aug 10, 2022, at 5:42 AM, Thomas, Matthew <<a href="mailto:mthomas@verisign.com" class="">mthomas@verisign.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class=""><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite"><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Section 7.3.1<o:p class=""></o:p></div></div><div class=""><p class="MsoNormal" style="margin: 0in 0.5in 5pt; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p class=""> </o:p></p></div></div></blockquote><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">>> The criteria is explained more above (see 1 and 2), but in short, it is 100% of a *sample* of queries, and it is compared to previous work on the subject.<o:p class=""></o:p></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">This only brings in more questioning to the reliability and accuracy and biased nature of using a sampled query data. My concerns remain even with your additional explanatory text. There is no data driven foundation as to why 5 queries and 100% adherence are appropriate. This is a fundamental premise that is currently unmotivated and unsupported properly. Stating this as an approximate lower bound measurement is fine, but it needs to be treated as such and it doesn’t facilitate making definitive statements like your conclusion.</div></div></div></div></div></blockquote><div><br class=""></div><div>You keep mentioning bias, but until you explain to me how *this* sample is biased, I fail to see it. Using a sample is a completely acceptable approach. I've already explained the reasoning for 100% minimized queries within the sample. With regard to the minimum of 5 non-root queries, it is a judgment call, the intent of which is to keep provide enough query behavior that it could be tested for qname minimization, but without unnecessarily ruling out clients that might otherwise. A negative match in this case rules out resolvers with 80% or less minimization, which is below the 95% mentioned in the de Vries paper.</div><br class=""><blockquote type="cite" class=""><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">>>I'm not exactly sure what you're saying. The de Vries paper covers many facets of qname minimization, and they are very careful to distinguish which parts can be applied and compared elsewhere--and in how they apply them. I've summarized some of those points in my introductory text of this email. And again: 1) we used the same methodology for determining minimized queries in passive analysis as they did, which was based on the findings from their active analysis; 2) we applied the analysis of minimized queries to resolvers using metrics also from their paper; and 3) our technique is a heuristic.<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">>>The selection criteria ignores a more selective QNM criteria defined in the RFC such as the Qtype (e.g., A and NS) and excludes multiple implementations of QNM techniques (e.g., nonce second level labels, underscore labels, asterisk labels, etc.).</span><o:p class=""></o:p></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Yes – I’m saying profiling known QNM implementations for ground truth needs to be done (again – de Vries paper was years ago). Furthermore, the de Vries knowledge still doesn’t make it into your selection criteria. Why not test for it being NS or A qtypes? Furthermore, QNM in 2018/2019 was not a standards track RFC. Things have changed. Industry implementations have changed. The RFC is now standards track. Applying a 2018/2019 heuristic without profiling current standards is ignoring the state-of-the-art. </div></div></div></div></div></blockquote><div><br class=""></div><div>There is a lot that *could* be done. But please let's see the forest for the trees. You are making some wonderful assertions about what needs to be done. That might well be true, but 1) we are working with prior work, peer reviewed and published, and 2) for the purposes of *this* report, it is sufficient. We are looking at *trends* with a *sample* of data. Testing your hypotheses about the way things *currently* operate would be great, but it sounds like another project.</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Another point is that this analysis did not apply its measurements to “resolvers”, it applied it to any IP that queried the RSS during DITL. There is no reason to believe that some IP sending 5 queries to the RSS is a recursive resolver. This is a big distinction and again goes to why the thresholds in #1 are not motivated.</div></div></div></div></div></blockquote><div><br class=""></div><div>Yes, identifying the role of IP addresses querying the root is currently an unsolved problem, as far as I know. So we do the best we can with the data we have. This is precedent. And in this case, we are looking at trends of querying IP addresses, so it doesn't really matter if it's a resolver or not -- because we are matching it to its name collision queries behavior from other parts of the report.</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> And to the last point of it being a heuristic, sure it is a heuristic but without proper motivation and reasoning, the results are just a measurement of an unmotivated heuristic and need to be treated as such</div></div></div></div></div></blockquote><div><br class=""></div><div>Sorry, I don't understand why you keep saying "unmotivated". The point of a heuristic is that we are approximating based on data we have because we don't have exact numbers.</div><br class=""><blockquote type="cite" class=""><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">>>See introductory text, parts 3A and 3B in particular.<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><br class="">This has nothing to do with 3A or 3B. The state-of-the-art has moved significantly since 2018/2019 and de Vries with regards to QNM. I’d encourage you to reexamine the current data from 2022 (actually from 2019 on) at DITL to better understand QNM deployment. I’m looking at A root data now and I can clearly see a _sizeable percentage of QNM traffic that you are not capturing to the items I mentioned previously. I _can’t _stress _this _point _enough.</div></div></div></div></div></blockquote><div><br class=""></div>With all due respect, "I'm looking at A-root data now" does not help me. This is in part because my analysis targets 2016 through 2021, but mostly the earlier years. This is also in part because I have no idea what methodology you are using. While you suggest that the de Vries study is out of date (and it might be), the rigor with which they studied resolver behavior was very impressive, using used multiple vantage points and data sets, passive and active analysis, etc. This is academic rigor.</div><div><br class=""></div><div>But again, the whole point of the analysis is to get a sample of resolvers that do not exhibit qname minimization behaviors, with which we can re-run the name collisions analysis to look for trends. Some amount of false positives and false negatives is okay.</div><div><br class=""></div><div><blockquote type="cite" class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class=""> </span></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">>>Sorry, I'm not sure what you are getting at here. If you are referring to the longitudinal measurement plot, it was intended to show the % of ASNs over time with at least one qname minimizing resolver, as a deployment trend. Nothing more, nothing less.<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">The point is that measuring QNM at a ASN or even IP level longitudinally within the context of NCAP makes no sense. Having one IP in a large ASN that might be QNM tells us nothing material around name collisions.</div></div></div></div></div></blockquote><div><br class=""></div><div>This had nothing to do with name collisions. It was part of the longitudinal analysis of qname minimization, which was simply part of the context.</div><div><br class=""></div><blockquote type="cite" class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">>>Perhaps - but that is not within the scope of this report. Some of that discussion is had in the "Fourteen Years..." paper.<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Can you please share this paper b/c I can’t find it?<span class="Apple-converted-space"> </span></div></div></div></div></div></blockquote><div><br class=""></div><div>It will be available publicly soon. I will send you a link privately.</div><br class=""><blockquote type="cite" class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">>> This is, of course, unrelated to the qname minimization analysis. But it is an interesting hypothesis that could be tested.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">It is absolutely within scope if you are claiming a finding in the Root Cause Analysis Report that name collisions decrease via your data measurements using this passive DNS data.</div></div></div></div></blockquote><div><br class=""></div><div>As I said before, it is unrelated to the qname minimization analysis, which is the topic of this thread. It's a very interesting idea.</div><br class=""><blockquote type="cite" class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> There are existing measurements of the difference between positive and negative referral query rates– not sure why this would need to be tested given what we already know about resolver treatment of NXD responses and the differences in positive/negative TTLs.</div></div></div></div></blockquote><div><br class=""></div>I suppose because measurement, or at least applying previous work to current data, is what we do?</div><div><br class=""><blockquote type="cite" class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><br class=""></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">>><span style="" class="">With regard to the large public resolver, I'm sorry, but this this is very vague. Without any documentation and/or empirical analysis to go on, I have nothing to re-assess or improve my analysis.</span><o:p class=""></o:p></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">See:<span class="Apple-converted-space"> </span><a href="https://blog.apnic.net/2022/06/02/more-mysterious-dns-root-query-traffic-from-a-large-cloud-dns-operator/" style="color: blue; text-decoration: underline;" class="">https://blog.apnic.net/2022/06/02/more-mysterious-dns-root-query-traffic-from-a-large-cloud-dns-operator/</a> This was also disclosed years ago I believe at DNS-OARC events.</div></div></div></div></blockquote><div><br class=""></div><div><br class=""></div>Thanks.</div><div><br class=""><blockquote type="cite" class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">>>Please remember that 1) we are interested in *trends* over time and 2) we only need samples--not complete data--to get those trends. The samples are taken from the resolvers identified as non-qname-minimizing in 2021. The process and the ultimate sample sizes are well documented in the text and the table.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Trends require a consistent measurement of the underlying study over time. You have not motivated or shown why an IP should be treated equal or consistent for measurement over time. Nor have you shown that a 13 query sample is representative. Saying this sampling of data is representative is not true without additional evidence. The same needs to be said about the measurements taken from the passive recursive data used for the root cause analysis report – there is no data or proof that shows that subset of queries to the root is representative and unbiased, given they were collected by a biased subset of operators, who were willing to deploy those data collection probes. </div></div></div></div></div></blockquote><div><br class=""></div><div>I'm not really even sure how to respond to this. This is just the way measurement, sampling, and statistics work.</div></div><div><br class=""></div><div><br class=""></div><div><font color="#000000" class="">I completely stand by the qname minimization methodology used and its application in the name collisions analysis. I also agree that there might be other ways of doing it, but that doesn't make what I have done and written *wrong*. Finally, please remember the *point* of this exercise, which is not perfection but approximation.</font></div><div><br class=""></div><div>Casey</div></div></body></html>