[RSSAC Caucus] Comments on the "RSS Metrics WP presentation to RSSAC Workshop"

[Paul Hoffman]

A big thank-you to Duane and Russ for getting this document out in time for Caucus comments. Here are a few from me; I hope they help the discussion at the workshop

The repeated theme for the metrics in the document is "Thresholds for 'what good looks like'" are TBD. That is certainly what I remember from the calls and the face-to-face meeting in Prague: we didn't get anywhere close to agreement on any of them. In fact, we also didn't get anywhere on a process to fill in those TBDs.

In the meetings, there were also (unanswered) questions whether the desired threshold is for "good" or "required". Given that the metrics work is supposed to feed back into RSSAC 037/038, that difference is vitally important, given that "good" will most likely be greater than "required". Looking at the published RSSAC002 data, it is obvious that, today, some RSOs carry more of the query load than others. Looking at the number of instances, and the placement of each instance in the Internet topology, some RSOs are more likely to fail sooner under a broad DDoS attack on the RSS than others. This makes the question of which metrics are "required" more difficult to analyze.

Under "RSO Metric: Correctness", there is the statement "Each response is validated for correct DNSSEC signatures". On the last call and/or in the face-to-face meeting, a few people brought up correctness in the glue records as well because that would affect non-validating resolver operators, who make up the large majority of the customers of the RSS.

In the calls, there appeared to be mostly confusion about why we were being asked about the RSS metrics. If a concrete use case in the 037/038 realm was given, I don't remember it. What I do remember is the confusion, which often boiled down to "if a resolver can consistently get acceptable answers from even one RSO, what the RSS as a whole looks like becomes immaterial to that resolver, so measuring the RSS seems unnecessary". If RSSAC wants RSS metrics, it will need to describe their need better than what we have at this point.

--Paul Hoffman