[RSSAC Caucus] [Ext] 48 HOUR LAST CALL : RSSAC002v4

Tue Feb 25 17:57:19 UTC 2020

I'm still having a hard time figuring out what the
'num-sources-ipv6-aggregate' number would actually *mean*, and am
somewhat concerned that it will become an attractive nuisance --
people will make assumptions about what the metric means, and will
draw dangerously flawed conclusions from this.

Recording individual (/128) level metrics is largely meaningless - a
perfectly reasonable implementation could use RFC 8273 (Unique IPv6
Prefix per Host) style deployments, and use the source address as
additional entropy, or per-port worker bindings - this would result in
one recursive server instance looking like O(num_queries). Recording
all of the addresses is infeasible, and a metric of 670 used addresses
doesn't tell you anything - if could be one server, it could be 670.

Recording /64s also seems largely meaningless - some deployments will
have a number of different "servers" in the same /64 - this means that
if you see a query from 2001:DB8:17:42:86::/64, it could be 1 instance
or 670 instances.

Unfortunately, while you and I (might!) agree on that, you *know* that
if a metric like:
num-sources-ipv6-aggregate: 565433
is published, people will immediately look at this and say "there are
five hundred sixty-five thousand four hundred thirty-three IPv6
nameservers" and will start graphing this, making predictions about
the growth of the internet, etc. Yes, they will be wrong, yes, they
shouldn't do this -- but they will. I don't know of a way to stop
this, other than expecting everyone to include a comment that says: #
The below metric doesn't mean what you think it does, go see: <link to
section in RSSACxxx> (I increasingly think that the OpenMetrics /
prometheus exposition format should have been what this used -
https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md
- there are many parsers, and comments are an expected conventions -
but that is a separate rathole...)

W

On Tue, Feb 25, 2020 at 10:06 AM Andrew McConachie
<andrew.mcconachie at icann.org> wrote:
>
>
>
> On Feb 25, 2020, at 15:32, John Heidemann <johnh at isi.edu> wrote:
>
> On Tue, 25 Feb 2020 09:32:43 +0000, Andrew McConachie wrote:
>
>
>
> On Feb 24, 2020, at 21:10, John Heidemann <johnh at isi.edu> wrote:
>
> On Mon, 24 Feb 2020 10:45:37 +0000, Andrew McConachie wrote:
>
>
>
> On Feb 21, 2020, at 23:52, Ray Bellis <ray at isc.org> wrote:
>
> On 21/02/2020 22:33, Paul Hoffman wrote:
>
> Just to be clear: are you saying that you can only produce figures
> for Section 3.5 (number of sources seen) for a subset of your nodes,
> or that you can only produce figures for a subset of your nodes for
> *all* the measurements? If the latter, that's going to mess up the
> evaluation of the measurements for the whole RSS. Even if it is just
> the former, that will make the measurements in Section 3.5 be not
> comparable to the other sections.
>
>
> Only for the unique sources measurement.
>
>
>
> Hi Ray,
>
> I changed section 6.6 to mark the ‘unique-sources’ metric as optional. This is reflected in the changelog as well.
>
> This change means that if the 'unique-sources' metric is produced by an RSO they must provide values for both 'num-sources-ipv4' and 'num-sources-ipv6-aggregate’. So produce values for both or produce values for neither.
>
> I would like to hear from other RSO’s if this approach is OK or if a different approach is needed.
>
>
> Ray asked the important question: is this metric useful.
>
> Wearing my "researcher" hat: yes, we have found number of unique-sources
> in RSSAC data useful to indicate when a RSO is subject to an attack with
> spoofed sources.
>
>
> But I can understand the challenge in computing this metric.  It is by
> far the most stressful (least compressible).
>
>
> Rather than just declaring the metric optional, what about declaring the
> precision of the answer variable?  Perhaps adding a
> "fraction-of-traffic-for-sources" field, which for some RSOs would be 1
> and others might be less than 1?  There is value in a num-sources even
> if it's based on a 10% sample of traffic, or 10% of anycast sites, and
> a sampled value might be easier for some RSOs to compute.
>
> (And yes, I recognize that sampling could be non-uniform.)
>
>  -John
>
> Hi John,
>
> I think the idea of a percent-instances-sampled makes a lot of sense, I’m just concerned about adding it this late in the process of RSSAC002v4. Is it OK to postpone this suggestion until RSSAC002v5?
>
> I can imagine a fair amount of discussion surrounding how exactly to measure precision given the different operating environments.
>
>
> That is understandable.  I'm certainly not insisting it go in.
>
> (Although making unique-sources optional is also a major change.)
>
>
> It is considering the diff between the document circulated at the beginning of this thread and the current draft RSSAC002v4. But it’s not a major change between the draft RSSAC002v4 and RSSAC002v3. num-sources-* have been optional since RSSAC002v1.
>
> I’m still interested in hearing from other Caucus members what they think about the draft RSSAC002v4 and if any other changes are needed.
> <https://docs.google.com/document/d/1na7VpPYfo4VEBfQMHwix6I2Puohvc1hvwRh0aW1cmFE/edit?usp=sharing>
>
> Please let me know by the end of this week if there are any more items to discuss.
>
> Thanks,
> Andrew
>
> _______________________________________________
> rssac-caucus mailing list
> rssac-caucus at icann.org
> https://mm.icann.org/mailman/listinfo/rssac-caucus
>
> _______________________________________________
> By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.

-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf