[tz] Pre-1970 data

[Stephen Colebourne]

How can this project successfully handle pre-1970 data? Why have I
focussed so much on IDs and their value to tzdb?

Intro
-------
In previous threads I've focussed on IDs, and laid out different ways
to describe the groups of IDs we have. For now, lets focus solely on
the two main groups
- region IDs, which represent abstract regions where clocks have been
the same since 1970
- non-region IDs, which represent locations where tzdb has, at some
point, added an ID

The key observation is that the segregation between these two groups
of IDs *did not exist* until around 2014. It was only in 2018 that the
ISO country rule was removed. It has only been since 2014 or so that
IDs have been merged. What tzdb previously offered was a set of IDs,
based on a simple rule - "ID as needed for post-1970 data, with at
least one per ISO country". Full history was available for each of
these (whether accurate or not). What has happened since is a split,
where no split previously existed. A split which favours some
locations over others.

The recent mailing list debates merely represent the cumulation of
this favouritism. The irony is that the merges are being done in the
name of equity and fairness, when the outcome has actually been the
exact opposite - picking favourites, and denigrating everywhere else.
That the approach to picking favourites is according to a standardised
largest city rule isn't really that relevant here - it is the outcome
that is unfair, not the process.

If region IDs were of different appearance (eg. numeric or textually
different) then the issues would not have arisen. The mistake was
taking a fully functional and fully integral set of IDs, and
bifurcating it into two groups. The split was actually a huge change
in the policy of tzdb, which has been added drip by drip, rather than
something that was ever fully appreciated up front.

FWIW, it is clear to me that there is an aspect of imposing a
US-centric timezone system on other parts of the world. The recent
tzdb approach of focussing entirely on timezone regions makes perfect
sense for the US, where region boundaries do not follow state lines,
and ordinary members of the public need to be aware of whether they
are in US/Mountain or US/Central. This simply isn't the timezone model
in many other parts of the world. In places like Europe and Asia, the
timezone is driven primarily by the country you live in - an ordinary
member of the public in Iceland is never going to associate with some
abstract timezone region stretching down the Atlantic that is not
named, not legally defined and is little more than a random outcome
based on tzdb's choice of 1970. Even in somewhere like Norway, an
ordinary member of the public will understand that although they
follow CET, their timezone is actually driven by their Government in
Oslo. The brilliance of the original rule - "ID as needed for
post-1970 data, with at least one per ISO country" - was that it
seamlessly handled *both* models of timezone in one unified set of
IDs. Removal of the ISO country part has completely destabilised that
balance.

As a constraint to this thread, tzdb really needs to offer one
standard view of data, not command line flags that allow different
views. If downstream projects end up with different views of the data,
it makes tzdb a much less reliable source. (tzdb can be packaged in
different ways on the same machine, for example it is undesirable for
Postgres' internal tzdb and the OS tzdb to diverge for the same
version). Given this, what needs to be nailed down is what is the
default data set that tzdb publishes - there isn't really much point
in talking about compile time flags, or that the contents of backzone
could be used by someone.


Lets look at seven options for pre-1970 data:

Pre-1970 data for regions only
-----------------------------------------
1) Pre-1970 data for regions only
- Despite looking identical to other IDs, region IDs are treated as
special/favoured
- Pre-1970 data for non-region IDs is of no importance whatsoever,
thus most get pre-1970 data from another country/continent
- The split between region and non-region locations is fully completed
- the US-centric timezone model is dominant
- An end user in Iceland is supposed to use Africa/Abidjan,
Europe/Reykjavik is treated as a historical mistake of tzdb kept
around only for backwards compatibility

Pre-1970 data for most IDs
------------------------------------
2) Pre-1970 data for each ID meeting the rule "ID as needed for
post-1970 data, with at least one per ISO country"
- The split between region and non-region locations is healed
- High quality data from places like Iceland and Norway is retained,
but low quality data from elsewhere is restored
- The pre-1970 data is simply viewed as the best available data for
the each location which can be improved over time
- Most end-users in Iceland would expect tzdb to provide pre-1970 data
from Iceland, not the Ivory Coast

3) Pre-1970 data for for all IDs that currently exist except true aliases
- This is very similar to #2, but would include something like
Montreal which was effectively mistakenly added to tzdb
- This doesn't seem as desirable as #2

4) Pre-1970 data for any ID where the pre-1970 data is high quality
- Subjective on quality, which doesn't seem like a great idea
- It does avoid bringing bad quality data back into the main tzdb distribution
- On balance, #2 has fewer places for debate to arise

5) Pre-1970 data for all IDs that currently exist except true aliases,
plus a *new* set of IDs representing regions
- As per #2, but adding IDs like "Region/12345" or "Region/Berlin"
- Regions should contain post-1970 data only, as the region is by
definition only meaningful post-1970
- It is not entirely clear what this solves over just going with #2
- This option is connected to Russ' proposals [1], although he
suggests a bigger split between timekeeping data and ID naming
- Perhaps it makes sense if the new region IDs were internal and not
normally seen by end users?

Remove pre-1970 data from general use
-------------------------------------------------------
6) No pre-1970 data whatsoever, all IDs are post-1970 only
- Project policy is that tzdb is focussed on post-1970 data only
- Paul repeatedly tells us that pre-1970 data is unreliable, and
people shouldn't use it
- Pre-1970 data would not be deleted, but would not be available in
most downstreams
- The split between region and non-region locations is effectively healed
- End users lose access to pre-1970 data, which is particularly
notable in some locations where that data is reliable, eg London
- It is unknown at this point what the user impact is of removing
pre-1970 data from major financial/business centres (no major location
has yet been merged)

7) No existing IDs get pre-1970 data, but a *new* set of IDs are
created containing it
- As per #5, the existing IDs get no pre-1970 data and the split
between region and non-region locations is healed
- New IDs, such as "Historic/Europe/London" or
"Historic/Africa/Abidjan" get created for each of the main IDs
- The historic IDs include pre-1970 data, the standard IDs do not
- There is a mechanical transformation between the historic and non-historic IDs
- Whether downstreams do or do not include the historic IDs is their
choice, potentially based on available space
- The data provided by the standard non-historic IDs remains the same
whether the downstream includes the historic IDs or not (a Good Thing)
- Users have to deliberately opt-in to get pre-1970 data, which might
make them think about the accuracy issue

Notes
--------
- I'm not discussing adding new IDs simply to represent locations
whose clocks differ only before 1970. I don't personally think that is
a job for tzdb, and even if it is, it is a job for a different thread.
- I'm not discussing what value is returned prior to 1970 when
pre-1970 data is removed. That would be a job for a different thread.
- Link vs Zone is not important for this discussion.

Summary
-------------
After a few weeks of thinking, these are the options I've come up
with. Feel free to suggest another option or variant if you think I've
missed anything obvious.

I believe that was a disaster that the brilliant "ID as needed for
post-1970 data, with at least one per ISO country" rule was removed.
It has created needless division, bifurcating a unified set of ID into
region and non-region IDs and creating backwards compatibility issues
in the data of many locations. IMO, there are two basic models of
timezones in the world, and moving tzdb from one that supported both
to one that only supports the US-centric model is simply a mistake
that needs correcting.

As such, my preference would be to adopt option 2. Option 6 or 7 could
work, and might be the best choices if we had a clean slate or if
there was some hidden pressure that the list is unaware of to remove
pre-1970 data, but they are risky options given we do not know the
impact on end-users in major financial/business centres.

Stephen

[1] https://mm.icann.org/pipermail/tz/2021-September/030518.html