[tz] Extra transition for Europe/London with 2023d

Mon Jan 8 00:51:55 UTC 2024

On Jan 7, 2024, at 12:01 PM, Brooks Harris <brooks at edlmax.com> wrote:

> Right, but I think there should be. Posix cannot distinguish an stdoff shift independent from a gmtoff shift.

So presumably:

	"stdoff shift" is short for "a shift in the offset between UTC and standard time", "standard time" being what is specified as such by law;

	"gmtoff shift" is short for "a shift in the offset between civil time and standard time".

Given that, not all shifts in the offset between UTC and civil time" are "gmtoff shifts", which is a bit confusing, given that "gmtoff" sounds as if it's an offset from "GMT" or UTC.

Not all "gmtoff shifts" constitute transitions to or from "daylight saving time", as Morocco, for example, has on some occasions introduced daylight saving time, but also shifts the clock during Ramadan.

> For example, there is a STDOFF shift at 1971 Oct 31  2:00u from 1:00 to 0:00 (a West shift) in London:
> 

> # Zone	NAME		STDOFF	RULES	FORMAT	[UNTIL]
> Zone	Europe/London	-0:01:15 -	LMT	1847 Dec  1
> 			 0:00	GB-Eire	%s	1968 Oct 27
> 			 1:00	-	BST	1971 Oct 31  2:00u
> 			 0:00	GB-Eire	%s	1996
> 			 0:00	EU	GMT/BST
> 
> Zic and TzIf reflect this change as a shift in gmtoff, not stdoff:Zic and TzIf reflect this change as a shift in gmtoff, not stdoff:
> 
>    57722399 1971-10-31 02:59:59 isdst 0 gmtoff   3600 stdoff      0 BST
>    57722400 1971-10-31 02:00:00 isdst 0 gmtoff      0 stdoff 0 GMT

RFC 8536:

	https://datatracker.ietf.org/doc/html/rfc8536

describes TZif format (to use the capitalization in the RFC) up to version 3 (version 4 is not documented in the RFC, but it's a small change to leap second handling, documented in the tzfile man page).

According to it, the local time type records contain a field named utoff (presumably changed from gmtoff), which contains "a four-octet signed integer specifying the number of seconds to be added to UT in order to determine local time" for that local time type, a Boolean field that indicates whether that local time type is to be considered DST or not, and a field that is an index into the table of time zone designation strings for the designation string to be used for that local time type.

Transitions are represented in the file by two parallel arrays (rather than an array of structures), one containing the UNIX time of the transition and once containing the local time type to which local time is transitioning.

The local time type includes an offset from UTC and a "is this DST?" flag.  Neither of those directly provide a "offset of standard time from UTC" value; at best, the "is this DST?" flag can be used to *infer* the standard time offset.

The transition in question is a transition from British Standard Time, which had a +1 hour offset from GMT, to GMT.  The "before" local time type has a utoff of +1 hour, an "is this DST?" flag of 0 (because DST wasn't observed at that point), and a designation string index that points to "BSD".  The "after" local time has a utoff of 0, an "is this DST?" flag of 0 (because it was in October, when DST was not in effect - DST wouldn't resume until next spring), and a destination string index that points to "GMT".

I don't know what program produced the output you're showing; how did it incorrectly infer that the offset between standard time and UTC before the transition was 0 rather than +1 hour (3600 seconds)?

> That's what I mean by "adjusted" for Posix sake. It gives the proper UTC offset, yes, but not for the right reason. The underlying reason was an STDOFF shift, presumably stated in the law behind it.

I don't know what "it' is here; it's not the TZif file format, as it does not contain anything that specifies a reason for a transition; at most, it provides a value to which to set tm_isdst and a value to indicate what time zone designation strings to use.  Reasons for the transitions can only be guessed at from those values, and code may well incorrectly guess the reason.  (The *correct* guess for the reasons for the transition at midnight on 1968-10-27 is "the offset between current local time and UTC changed from 0:00:00 to +1:00:00 and the designation string changed from GMT to BST", and the *correct* guess for the reasons for the transition at 2:00 UTC on 1971-10-31 are "the offset between current local time and UTC changed from +1:00:00 to 0:00:00 and the designation string changed from BST to GMT".  Neither one is "the" reason; a change in the current local time, in whether tm_isdst should be set to 0 or 1, or in the designation string are sufficient to cause a transition.)

> TzDb source has been careful to try to honor the laws and customs behind the local time rules, and I strongly support that policy.

What do you mean by "honor"?  Do you mean "give the correct results when, for example, localtime() is called", with the local time being the result of law and the designation string being a result of custom in some cases (in other cases, there doesn't appear to be a custom, and it's just given as an offset from UTC).  Or do you mean "specifies the offset of local standard time from UTC in Zone lines and the offset of current local time from local standard time in Rule lines", in which case, as far as I know, that's more a case of notational convenience for people writing zic source files and a matter of honoring laws and customs; laws and customs affected that part of the syntax to the extent that the chosen syntax means you have to do less work to translate laws to zic file text.

> I would presume this custom originates with the USA laws that specifically define the time zones as offsets from UTC and the optional one-hour DST shifts. These are the familiar behaviors of time zones where DST is "observed".

I'm not sure to which "custom" you're referring, but, if you mean "TZif files don't separately provide the local standard time and current local time offsets from UTC", that probably originated with "damn, the US DST rules changed *again* - thanks, Clorox! - and the way UNIXes handle converting to local time will have to be fixed, which involves changing a table in a library and recompiling the library and relinking all programs that use the library; can we come up with something better?", which was the origin of the tzdb project.

What 15 U.S. Code section 261 - Zones for standard time; interstate or foreign commerce:

	https://www.govtrack.us/congress/bills/89/s1404/text

states is that

	For the purpose of establishing the standard time of the United States, the territory of the United States shall be divided into nine zones in the manner provided in this section. Except as provided in section 260a(a) of this title, the standard time of the first zone shall be Coordinated Universal Time retarded by 4 hours; that of the second zone retarded by 5 hours; that of the third zone retarded by 6 hours; that of the fourth zone retarded by 7 hours; that of the fifth zone retarded [1] 8 hours; that of the sixth zone retarded by 9 hours; that of the seventh zone retarded by 10 hours; that of the eighth zone retarded by 11 hours; and that of the ninth zone shall be Coordinated Universal Time advanced by 10 hours.

which specifies the offsets from UTC for standard time in the time zones of the US, and what 15 U.S. Code § 260a - Advancement of time or changeover dates:

	https://www.law.cornell.edu/uscode/text/15/260a#a

states in subparagraph (a) is that

	During the period commencing at 2 o’clock antemeridian on the second Sunday of March of each year and ending at 2 o’clock antemeridian on the first Sunday of November of each year, the standard time of each zone established by sections 261 to 264 of this title, as modified by section 265 of this title, shall be advanced one hour and such time as so advanced shall for the purposes of such sections 261 to 264, as so modified, be the standard time of such zone during such period; however, (1) any State that lies entirely within one time zone may by law exempt itself from the provisions of this subsection providing for the advancement of time, but only if that law provides that the entire State (including all political subdivisions thereof) shall observe the standard time otherwise applicable during that period, and (2) any State with parts thereof in more than one time zone may by law exempt either the entireState as provided in (1) or may exempt the entire area of the State lying within any time zone.

which specifies that, unless some state says "no, thanks" to DST, the clocks get turned forward by 1 hour on the second Sunday of March at 2AM and turned back on the first Sunday of November at 2AM.

So section 261 "specifically [defines] the time zones as offsets from UTC" and section 260a(a) describes the "optional one-hour DST shifts".

That's not what TZif files do; they specify, for local time zone types, the *current* offset from UTC for that local time zone type, including shifts for DST or Ramadan or any other such change.

> I'm involved with several standards projects at the Society of Motion Picture and Television Engineers (SMPTE). The television and broadcast industries have used "SMPTE Timecode" (defined primarily by a standard named ST12-1) for decades, since the mid 1970s. SMPTE Timecode is used in very many ways, from labeling of video frames to synchronization and essentially glues the timekeeping of the whole industry together.
> 
> SMPTE Timecode was derived from IRIG and is quite similar to the formats used by WWV radio and others with addition of video-related metadata.
> 
> Timecode is used all over the world, where various video rates are used. For example most of Europe uses 1/25 (25 frames-per-second). These systems have an exact relation to running time in seconds and are sometimes called "integral rate". But the USA, Canada, Japan and others use the 'strange' rate off 30000/1001, called a "non-integral rate". This has all sorts of algorithmic ramifications and implementations must be careful with their math. But this is well-know in the industry. Meantime there is the matter of audio synchronization of various frequencies with video. A typical audio frequency is 48000/1.
> 
> If you're interested in more detail about video and audio rates and SMPTE Timecode you might read:
> 
> Conversion of Audio Samples to Video Frames
> Brooks Harris, May 8, 1998
> http://edlmax.com/AudioNTSC.htm
> 
> Conversion between SMPTE hh:mm:ss:ff Time Code and Frames
> Brooks Harris, EdlMax, LLC. Version V4 2015-04-04
> http://edlmax.com/SMPTETimeCodeConversion.htm
> 
> In broadcast television it's important to have the hh:mm:ss portion of the timecode run as close as possible to local time. But SMPTE timecode and equipment that depends on it cannot tolerate a discontinuity in the hh:mm:ss:video-frame sequence. To work around this limitation the industry has typically adopted a procedure called "Daily Jam". Here, DST shifts, STDOFF shifts and leap-second shifts are deferred to some local time and the  hh:mm:ss:video-frame values are reset ("jammed") to match the local wall-clock time. This is an imperfect solution which can produce a "glitch" in the hh:mm:ss:video-frame sequence, especially for systems running at "non-integer" rates. Therefore the "Daily Jam" is typically instituted at some local time least likely to disrupt normal operations. In the USA this time is typically 02:00:00 (2am). Daily Jam is imperfect but an effective work-around that has been used for decades.

Hey, I'm old enough to remember when TV stations signed off for the night.  That would have worked fine back then.

> 
> In recent years SMPTE has developed several standards related to packet-based network "streaming" of video and audio. They've elected to use IEEE Precision Time Protocol as the primary synchronization reference. The standards called ST2059-1 and ST2059-2 in particular define the relation of timecode to PTP time. These also codify the use of Daily Jam procedures.
> 
> User requirements call for use of local time together with video frame and audio-sample accuracy. And it is this set of challenges that have brought me (kicking and screaming :-) ) into the wider world of timekeeping and so to Tz Database.
> 
> In 2019 I was invited to make a presentation to ION/PTTI:
> Accurate Local Timestamps, Brooks Harris
> https://www.ion.org/publications/abstract.cfm?articleID=16763
> 
> In this I pointed out that typical timestamps were insufficient to represent truly unambiguous local time. If we have two ISO 8601 timestamps with date, time and UTC offset:
> 2023-07-01 00:01:23 -7:00
> 2023-07-01 00:04:56 -7:00
> We can normalize to UTC and determine that the first precedes the second; precedence is established. Great! This, of course, is the first of most important aspect of timekeeping. This was the whole point to Posix-time to begin with; maintain event precedence within the system.

Or, rather, to provide a monotonic scale for time-stamping events - the goals are 1) allow conversion to local time and 2) maintain monotonicity.

(Leap seconds, and the POSIX choice to mandate 86400-second days, made monotonicity a bit tricky, but I digress....)

> But this entirely misses the fact that the two timestamps may have come from two different time zones. If you add the time zone it becomes much clearer what the meaning of the two events are:
> 2023-07-01 00:01:23 -7:00 America/Los_Angeles
> 2023-07-01 00:04:56 -7:00 America/Phoenix
> 

> But this does not signal that DST was in effect in America/Los_Angeles and not observed in America/Phoenix. You could add an "isdst" flag:
> 2023-07-01 00:01:23 -7:00 America/Los_Angeles DST
> 2023-07-01 00:04:56 -7:00 America/Phoenix STD
> That works (more-or-less) in these two typical and familiar cases.

So presumably there's a requirement for those representations of local time other than "allow local time to be displayed" and "be convertible to UTC", as that's true of that time stamp format without either the timezone ID or a DST/STD indication.'

(Presumably "DST" means "not STD" rather than "daylight saving time", as 1) Morocco shifts its clocks for Ramadan but that's not DST and 2) Ireland's *summer* time is standard time.)

> But when you get to other time zones that simple logic may not hold up. My favorite "kryptonite" example is Europe/Moscow:

	...

> Take the transition at 1991-03-31 02:00:00
>              3:00	Russia	MSK/MSD	1991 Mar 31  2:00s
>              2:00	Russia	EE%sT	1992 Jan 19  2:00s
> 
> This is an STDOFF shift with a simultaneous offsetting DST shift. As Time And Date puts it:

> 
> Mar 31 No change, Mar 31, 1991 - Daylight Saving Time Started
> DST started on Sunday, March 31, 1991, 2:00:00 am. However, clocks were not changed because Moscow switched time zones at the same time.
> https://www.timeanddate.com/time/change/russia/moscow?year=1991
> 
> This is a "no-op" transition (the YMDhms sequence is not interrupted) but several important facts changed; both STDOFF and DST shifted, and the Abbreviation changed.

The fact that the designation string (which, in this case, does happen to be an abbreviation, although that is not guaranteed) changed renders it *not* a "no-op" transition from zic's point of view.

> Zic also sets the isdst flag:
> 
>   670373999 1991-03-31 01:59:59 isdst 0 gmtoff  10800 stdoff  10800 MSK
>   670374000 1991-03-31 02:00:00 isdst 1 gmtoff  10800 stdoff 10800 EEST

...which means that the isdst flag also changed, further making it not a "no-op" transition from pic's point of view.

However, it reports "EEST", not "EEDT", so *that's* probably a bug. tzname[] is set to { "MSK", "EEST" } after localtime() is called on 670374000, so the timezone code in macOS 13.6, at least, has this bug in it.  It should probably have been set to { "EEST", "EEDT" }.

> There is another objective in my work. In video we have always been able to represent any time-point within the 24-hour range, important for editorial adjustments a duration calculation. To do this comprehensively with local time you need to know that this day includes a transition and what and when that transition occurs.

Why?  Why does the local time, plus the offset from UTC of local time at that instant, not suffice to represent any time?