[RSSAC Caucus] RSS Metrics update from ICANN 65

[Wessels, Duane]

Caucus Members,

There were a number of discussions about the RSS Metrics work party here at ICANN 65 in Marrakech.  Some of you may not have been able to participate due to the time zone of the local meeting.  When the transcripts and recordings are available we will let you know.  Meanwhile I want to let you know about a few discussion topics in particular.

There was some spare time on RSSAC's agenda on Tuesday so we took advantage of that to discuss some of the metrics work.   Since this was not known in advance, it was not possible for remote work party participants to join the call.  One thing we went over in this session was the use of Standard Deviation versus Percentiles to represent precision.  I presented some real data from RIPE Atlas probe measurements that demonstrated why percentiles are preferred over standard deviation.  This material is attached here as a file named stddev-vs-percentiles.pdf.  In the discussion that followed, we reached consensus that percentiles should be used going forward.

Next we discussed the "BPQ" section of the document.  There is new text here that proposes take real packet capture files and use them to determine the relationship between raw packets and query counts over UDP and TCP.  Then these can be applied in reverse by taking RSSAC002 data to calculate bandwidth and packets.  

On Wednesday we held the scheduled Metrics WP discussion.  Here we spent a lot of time talking about "near" versus "far" probes.  Near probes are probably better for some metrics (such as correctness) because it reduces or eliminates parts of the network out of the operator's control.  That is, the nearer that a probe is to a server, the more likely the response is unchanged.  However, near probes may be less desirable for certain metrics, such as latency, especially in the sense that they don't really capture latencies experienced by recursive name servers and end users.  Although there was plenty of discussion on this topic, there was no consensus  The work party needs to consider this further and make some specific proposals.

Our research fellow, John Kristoff, completed some of the work in analyzing RIPE Atlas data and applying it to the proposed metrics. I presented his materials, also attached here.  One thing in particular to note here is that these measurements are from RIPE Atlas "Anchors" and show a very wide range of measured latencies for different root servers.  We have given John some preliminary feedback and look forward to his future progress.

Another topic of discussion was the use of locally operated recursive name server software to implement some of the RSS metrics.  After thinking about this, Russ and I feel that this approach has some drawbacks.  One is that we may end up measuring the behavior of the recursive software itself, rather than the RSS.  For example, the recursive software implementation of retry behavior (timeouts etc) can affect latency results.  Also we are concerned that this approach could lead to incomparable results over time as implementations evolve and software is updated.  Therefore at this time we recommend removing these alternative approaches from the work party document.

I believe those are the highlights.  I invite anyone else to chime in with their own observations from the Marrakech meetings.

DW







-------------- next part --------------
A non-text attachment was scrubbed...
Name: stddev-vs-percentiles.pdf
Type: application/pdf
Size: 2357706 bytes
Desc: not available
URL: <http://mm.icann.org/pipermail/rssac-caucus/attachments/20190628/016b3220/stddev-vs-percentiles.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Metrics for the Root Servers and Root Server System Discussion.pdf
Type: application/pdf
Size: 13805026 bytes
Desc: not available
URL: <http://mm.icann.org/pipermail/rssac-caucus/attachments/20190628/016b3220/MetricsfortheRootServersandRootServerSystemDiscussion.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4675 bytes
Desc: not available
URL: <http://mm.icann.org/pipermail/rssac-caucus/attachments/20190628/016b3220/smime.p7s>