[tz] Extra transition for Europe/London with 2023d
Brooks Harris
brooks at edlmax.com
Tue Jan 9 21:10:37 UTC 2024
On 1/7/2024 7:51 PM, Guy Harris wrote:
> On Jan 7, 2024, at 12:01 PM, Brooks Harris<brooks at edlmax.com> wrote:
>
>> Right, but I think there should be. Posix cannot distinguish an stdoff shift independent from a gmtoff shift.
> So presumably:
>
> "stdoff shift" is short for "a shift in the offset between UTC and standard time", "standard time" being what is specified as such by law;
>
> "gmtoff shift" is short for "a shift in the offset between civil time and standard time".
>
> Given that, not all shifts in the offset between UTC and civil time" are "gmtoff shifts", which is a bit confusing, given that "gmtoff" sounds as if it's an offset from "GMT" or UTC.
stdoff is the "standard" offset from UTC of UT *without DST*. gmtoff is
the offset from UT or UTC *with DST*. There is no "dstoff" to signal the
DST value in effect, which is usually 1-hour but can be negative
(Dublin), "double summertime" or possibly some other value. This is
where the isdst flag is insufficient to cover those not 1-hour cases.
The term "standard" is some what ambiguous in general but understood
within the TzDb context as the "normal" or "base" offset from UT or UTC
>
> Not all "gmtoff shifts" constitute transitions to or from "daylight saving time", as Morocco, for example, has on some occasions introduced daylight saving time, but also shifts the clock during Ramadan.
Yes. Which brings up the terms "spring forward" and "fall back" which
imply 1-hour shifts in the spring and fall, but this doesn't work for
the four transitions in Morocco and elsewhere or for negative DST.
>> For example, there is a STDOFF shift at 1971 Oct 31 2:00u from 1:00 to 0:00 (a West shift) in London:
>>
>> # Zone NAME STDOFF RULES FORMAT [UNTIL]
>> Zone Europe/London -0:01:15 - LMT 1847 Dec 1
>> 0:00 GB-Eire %s 1968 Oct 27
>> 1:00 - BST 1971 Oct 31 2:00u
>> 0:00 GB-Eire %s 1996
>> 0:00 EU GMT/BST
>>
>> Zic and TzIf reflect this change as a shift in gmtoff, not stdoff:Zic and TzIf reflect this change as a shift in gmtoff, not stdoff:
>>
>> 57722399 1971-10-31 02:59:59 isdst 0 gmtoff 3600 stdoff 0 BST
>> 57722400 1971-10-31 02:00:00 isdst 0 gmtoff 0 stdoff 0 GMT
> RFC 8536:
>
> https://datatracker.ietf.org/doc/html/rfc8536
>
> describes TZif format (to use the capitalization in the RFC) up to version 3 (version 4 is not documented in the RFC, but it's a small change to leap second handling, documented in the tzfile man page).
>
> According to it, the local time type records contain a field named utoff (presumably changed from gmtoff), which contains "a four-octet signed integer specifying the number of seconds to be added to UT in order to determine local time" for that local time type, a Boolean field that indicates whether that local time type is to be considered DST or not, and a field that is an index into the table of time zone designation strings for the designation string to be used for that local time type.
>
> Transitions are represented in the file by two parallel arrays (rather than an array of structures), one containing the UNIX time of the transition and once containing the local time type to which local time is transitioning.
>
> The local time type includes an offset from UTC and a "is this DST?" flag. Neither of those directly provide a "offset of standard time from UTC" value; at best, the "is this DST?" flag can be used to *infer* the standard time offset.
>
> The transition in question is a transition from British Standard Time, which had a +1 hour offset from GMT, to GMT. The "before" local time type has a utoff of +1 hour, an "is this DST?" flag of 0 (because DST wasn't observed at that point), and a designation string index that points to "BSD". The "after" local time has a utoff of 0, an "is this DST?" flag of 0 (because it was in October, when DST was not in effect - DST wouldn't resume until next spring), and a destination string index that points to "GMT".
>
> I don't know what program produced the output you're showing; how did it incorrectly infer that the offset between standard time and UTC before the transition was 0 rather than +1 hour (3600 seconds)?
This is output from my modified version of zdump reading the TZIf output
of my modified version of zic. These include a modified version of
struct tm which I've renamed "struct tztm", to which I've added long int
tm_stdoff which is populated by the values of STDOFF from the TzDb
source files. This tm_stdoff value is also added to the TZIf file data.
struct tztm
{
int tm_sec; /* Seconds. [0-60] (1 leap second) */
int tm_min; /* Minutes. [0-59] */
int tm_hour; /* Hours. [0-23] */
int tm_mday; /* Day. [1-31] */
int tm_mon; /* Month. [0-11] */
int tm_year; /* Year - 1900. */
int tm_wday; /* Day of week. [0-6] */
int tm_yday; /* Days in year.[0-365] */
int tm_isdst; /* DST. [-1/0/1]*/
long int tm_gmtoff; /* Seconds east of UTC. */
const char *tm_zone; /* Timezone abbreviation. */
long int tm_stdoff; // CCT addition <<<<<<<<<<<<<<<<<<<
};
So, in this example stdoff is 0:00 on both sides of the transition.
0:00 GB-Eire %s 1996
0:00 EU GMT/BST
>> That's what I mean by "adjusted" for Posix sake. It gives the proper UTC offset, yes, but not for the right reason. The underlying reason was an STDOFF shift, presumably stated in the law behind it.
> I don't know what "it' is here; it's not the TZif file format, as it does not contain anything that specifies a reason for a transition; at most, it provides a value to which to set tm_isdst and a value to indicate what time zone designation strings to use. Reasons for the transitions can only be guessed at from those values, and code may well incorrectly guess the reason. (The *correct* guess for the reasons for the transition at midnight on 1968-10-27 is "the offset between current local time and UTC changed from 0:00:00 to +1:00:00 and the designation string changed from GMT to BST", and the *correct* guess for the reasons for the transition at 2:00 UTC on 1971-10-31 are "the offset between current local time and UTC changed from +1:00:00 to 0:00:00 and the designation string changed from BST to GMT". Neither one is "the" reason; a change in the current local time, in whether tm_isdst should be set to 0 or 1, or in the designation string are sufficient to cause a transition.)
>
>> TzDb source has been careful to try to honor the laws and customs behind the local time rules, and I strongly support that policy.
> What do you mean by "honor"? Do you mean "give the correct results when, for example, localtime() is called", with the local time being the result of law and the designation string being a result of custom in some cases (in other cases, there doesn't appear to be a custom, and it's just given as an offset from UTC). Or do you mean "specifies the offset of local standard time from UTC in Zone lines and the offset of current local time from local standard time in Rule lines", in which case, as far as I know, that's more a case of notational convenience for people writing zic source files and a matter of honoring laws and customs; laws and customs affected that part of the syntax to the extent that the chosen syntax means you have to do less work to translate laws to zic file text.
Right. As Paul said in another part of this thread, he "invents" things
where necessary or convenient. Fine. But in general these seem to try to
follow official laws if and where they exist and are clear.
>
>> I would presume this custom originates with the USA laws that specifically define the time zones as offsets from UTC and the optional one-hour DST shifts. These are the familiar behaviors of time zones where DST is "observed".
> I'm not sure to which "custom" you're referring, but, if you mean "TZif files don't separately provide the local standard time and current local time offsets from UTC", that probably originated with "damn, the US DST rules changed *again* - thanks, Clorox! - and the way UNIXes handle converting to local time will have to be fixed, which involves changing a table in a library and recompiling the library and relinking all programs that use the library; can we come up with something better?", which was the origin of the tzdb project.
Right. Perhaps "custom" is a misleading word. I meant that Posix seems
to support USA rules ok but is not complete for many other time zones
that act differently than the rules we US-based people are most familiar
with.
>
> What 15 U.S. Code section 261 - Zones for standard time; interstate or foreign commerce:
>
> https://www.govtrack.us/congress/bills/89/s1404/text
>
> states is that
>
> For the purpose of establishing the standard time of the United States, the territory of the United States shall be divided into nine zones in the manner provided in this section. Except as provided in section 260a(a) of this title, the standard time of the first zone shall be Coordinated Universal Time retarded by 4 hours; that of the second zone retarded by 5 hours; that of the third zone retarded by 6 hours; that of the fourth zone retarded by 7 hours; that of the fifth zone retarded [1] 8 hours; that of the sixth zone retarded by 9 hours; that of the seventh zone retarded by 10 hours; that of the eighth zone retarded by 11 hours; and that of the ninth zone shall be Coordinated Universal Time advanced by 10 hours.
>
> which specifies the offsets from UTC for standard time in the time zones of the US, and what 15 U.S. Code § 260a - Advancement of time or changeover dates:
>
> https://www.law.cornell.edu/uscode/text/15/260a#a
>
> states in subparagraph (a) is that
>
> During the period commencing at 2 o’clock antemeridian on the second Sunday of March of each year and ending at 2 o’clock antemeridian on the first Sunday of November of each year, the standard time of each zone established by sections 261 to 264 of this title, as modified by section 265 of this title, shall be advanced one hour and such time as so advanced shall for the purposes of such sections 261 to 264, as so modified, be the standard time of such zone during such period; however, (1) any State that lies entirely within one time zone may by law exempt itself from the provisions of this subsection providing for the advancement of time, but only if that law provides that the entire State (including all political subdivisions thereof) shall observe the standard time otherwise applicable during that period, and (2) any State with parts thereof in more than one time zone may by law exempt either the entireState as provided in (1) or may exempt the entire area of the State lying within any time zone.
>
> which specifies that, unless some state says "no, thanks" to DST, the clocks get turned forward by 1 hour on the second Sunday of March at 2AM and turned back on the first Sunday of November at 2AM.
>
> So section 261 "specifically [defines] the time zones as offsets from UTC" and section 260a(a) describes the "optional one-hour DST shifts".
>
> That's not what TZif files do; they specify, for local time zone types, the *current* offset from UTC for that local time zone type, including shifts for DST or Ramadan or any other such change.
Right. gmtoff is the sum of STDOFF and the current DST value. Thus
STDOFF and DST are confounded.
I would point out the ambiguity of the term "standard". Those laws say
it is "standard time" with or without DST.
" ...the standard time of each zone ... as modified by ... shall be
advanced one hour and such time as so advanced shall ... be the standard
time of such zone during such period"
This is consistent with how ISO 8601 defines "standard".
But the common way of referring to local time, at least in the USA, is
to say, for example, "Eastern Standard Time" or "Eastern Daylight Time".
This is more descriptive and familiar to many of us but its actually in
contradiction to the USA law as written.
>
>> I'm involved with several standards projects at the Society of Motion Picture and Television Engineers (SMPTE). The television and broadcast industries have used "SMPTE Timecode" (defined primarily by a standard named ST12-1) for decades, since the mid 1970s. SMPTE Timecode is used in very many ways, from labeling of video frames to synchronization and essentially glues the timekeeping of the whole industry together.
>>
>> SMPTE Timecode was derived from IRIG and is quite similar to the formats used by WWV radio and others with addition of video-related metadata.
>>
>> Timecode is used all over the world, where various video rates are used. For example most of Europe uses 1/25 (25 frames-per-second). These systems have an exact relation to running time in seconds and are sometimes called "integral rate". But the USA, Canada, Japan and others use the 'strange' rate off 30000/1001, called a "non-integral rate". This has all sorts of algorithmic ramifications and implementations must be careful with their math. But this is well-know in the industry. Meantime there is the matter of audio synchronization of various frequencies with video. A typical audio frequency is 48000/1.
>>
>> If you're interested in more detail about video and audio rates and SMPTE Timecode you might read:
>>
>> Conversion of Audio Samples to Video Frames
>> Brooks Harris, May 8, 1998
>> http://edlmax.com/AudioNTSC.htm
>>
>> Conversion between SMPTE hh:mm:ss:ff Time Code and Frames
>> Brooks Harris, EdlMax, LLC. Version V4 2015-04-04
>> http://edlmax.com/SMPTETimeCodeConversion.htm
>>
>> In broadcast television it's important to have the hh:mm:ss portion of the timecode run as close as possible to local time. But SMPTE timecode and equipment that depends on it cannot tolerate a discontinuity in the hh:mm:ss:video-frame sequence. To work around this limitation the industry has typically adopted a procedure called "Daily Jam". Here, DST shifts, STDOFF shifts and leap-second shifts are deferred to some local time and the hh:mm:ss:video-frame values are reset ("jammed") to match the local wall-clock time. This is an imperfect solution which can produce a "glitch" in the hh:mm:ss:video-frame sequence, especially for systems running at "non-integer" rates. Therefore the "Daily Jam" is typically instituted at some local time least likely to disrupt normal operations. In the USA this time is typically 02:00:00 (2am). Daily Jam is imperfect but an effective work-around that has been used for decades.
> Hey, I'm old enough to remember when TV stations signed off for the night. That would have worked fine back then.
It was used back then, and still used today. In those old days they
actually "jammed", interrupted, the vertical sync frequency itself
together with the timecode values, and this put a vertical roll into the
broadcast video. Today the frequency is maintained constant and only the
timecode values are "jammed" (reset).
>
>> In recent years SMPTE has developed several standards related to packet-based network "streaming" of video and audio. They've elected to use IEEE Precision Time Protocol as the primary synchronization reference. The standards called ST2059-1 and ST2059-2 in particular define the relation of timecode to PTP time. These also codify the use of Daily Jam procedures.
>>
>> User requirements call for use of local time together with video frame and audio-sample accuracy. And it is this set of challenges that have brought me (kicking and screaming :-) ) into the wider world of timekeeping and so to Tz Database.
>>
>> In 2019 I was invited to make a presentation to ION/PTTI:
>> Accurate Local Timestamps, Brooks Harris
>> https://www.ion.org/publications/abstract.cfm?articleID=16763
>>
>> In this I pointed out that typical timestamps were insufficient to represent truly unambiguous local time. If we have two ISO 8601 timestamps with date, time and UTC offset:
>> 2023-07-01 00:01:23 -7:00
>> 2023-07-01 00:04:56 -7:00
>> We can normalize to UTC and determine that the first precedes the second; precedence is established. Great! This, of course, is the first of most important aspect of timekeeping. This was the whole point to Posix-time to begin with; maintain event precedence within the system.
> Or, rather, to provide a monotonic scale for time-stamping events - the goals are 1) allow conversion to local time and 2) maintain monotonicity.
>
> (Leap seconds, and the POSIX choice to mandate 86400-second days, made monotonicity a bit tricky, but I digress....)
Quite. The leap-second is evil.
>
>> But this entirely misses the fact that the two timestamps may have come from two different time zones. If you add the time zone it becomes much clearer what the meaning of the two events are:
>> 2023-07-01 00:01:23 -7:00 America/Los_Angeles
>> 2023-07-01 00:04:56 -7:00 America/Phoenix
>>
>> But this does not signal that DST was in effect in America/Los_Angeles and not observed in America/Phoenix. You could add an "isdst" flag:
>> 2023-07-01 00:01:23 -7:00 America/Los_Angeles DST
>> 2023-07-01 00:04:56 -7:00 America/Phoenix STD
>> That works (more-or-less) in these two typical and familiar cases.
> So presumably there's a requirement for those representations of local time other than "allow local time to be displayed" and "be convertible to UTC", as that's true of that time stamp format without either the timezone ID or a DST/STD indication.'
Yes. Imagine for example a news organization collecting news feeds from
cameras in Los Angeles, Washington DC, Johannesburg, Taipei, or
anywhere else. You really need to know the local time when and where an
event happened, not just its UTC time.
>
> (Presumably "DST" means "not STD" rather than "daylight saving time", as 1) Morocco shifts its clocks for Ramadan but that's not DST and 2) Ireland's *summer* time is standard time.)
Right. To my point that "standard" is somewhat ambiguous and not all
time zones behave in the ways familiar to many of us, like in the USA.
I've learned to be very careful not to impose these familiar biases on
my implementations of local time. It just doesn't work the same way in
many places.
>> But when you get to other time zones that simple logic may not hold up. My favorite "kryptonite" example is Europe/Moscow:
> ...
>
>> Take the transition at 1991-03-31 02:00:00
>> 3:00 Russia MSK/MSD 1991 Mar 31 2:00s
>> 2:00 Russia EE%sT 1992 Jan 19 2:00s
>>
>> This is an STDOFF shift with a simultaneous offsetting DST shift. As Time And Date puts it:
>> Mar 31 No change, Mar 31, 1991 - Daylight Saving Time Started
>> DST started on Sunday, March 31, 1991, 2:00:00 am. However, clocks were not changed because Moscow switched time zones at the same time.
>> https://www.timeanddate.com/time/change/russia/moscow?year=1991
>>
>> This is a "no-op" transition (the YMDhms sequence is not interrupted) but several important facts changed; both STDOFF and DST shifted, and the Abbreviation changed.
> The fact that the designation string (which, in this case, does happen to be an abbreviation, although that is not guaranteed) changed renders it *not* a "no-op" transition from zic's point of view.
>
>> Zic also sets the isdst flag:
>>
>> 670373999 1991-03-31 01:59:59 isdst 0 gmtoff 10800 stdoff 10800 MSK
>> 670374000 1991-03-31 02:00:00 isdst 1 gmtoff 10800 stdoff 10800 EEST
> ...which means that the isdst flag also changed, further making it not a "no-op" transition from pic's point of view.
>
> However, it reports "EEST", not "EEDT", so *that's* probably a bug. tzname[] is set to { "MSK", "EEST" } after localtime() is called on 670374000, so the timezone code in macOS 13.6, at least, has this bug in it. It should probably have been set to { "EEST", "EEDT" }.
Maybe. But that's what comes out today, and that seems sufficient for
the Posix purposes. It might not be "correct", I guess?
>
>> There is another objective in my work. In video we have always been able to represent any time-point within the 24-hour range, important for editorial adjustments a duration calculation. To do this comprehensively with local time you need to know that this day includes a transition and what and when that transition occurs.
> Why? Why does the local time, plus the offset from UTC of local time at that instant, not suffice to represent any time?
With SMPTE timecode one can represent any time-point within the 24-hour
range. SMPTE timecode includes flags for the "count mode"
(non-drop-frame and drop-frame). There is no discontinuity in the
hh:mm:ss:frames counting sequence. So with a single SMPTE timecode an
application can "trim" forward or back and calculate durations from
point to point along the 24-hour timeline. There is often no relation to
actual local time-of-day, just a count from zero to 24 hours. This is
very typical in many scenarios, especially post-production (editing).
There is no need for access to any other metadata.
I extend this idea to local time time-of-day representation. Some days
have transitions, so my timestamp design carries sufficient information
to signal "this day is a transition day", "it is a transition of x value
(often DST shifts, sometimes STDOFF shifts)", and "the transition occurs
at this time-of-day". Thus, an application that does not have access to
any metadata (TZif or TzDb source file) can accurately represent any
point during that day.
Example - A "fall back" DST transition in America/New_York:
D2022-11-06T00:00:00U-04Zamerica/new_yorkAedtV2021aL27S01t-01a02cMuX UTC
01667707227
D2022-11-06T01:59:59U-04Zamerica/new_yorkAedtV2021aL27S01t-01a02cMuX UTC
01667714426
D2022-11-06T01:00:00U-05Zamerica/new_yorkAestV2021aL27S00t-01a02cMuX UTC
01667714427
D2022-11-06T23:59:59U-05Zamerica/new_yorkAestV2021aL27S00t-01a02cMuX UTC
01667797226
^^^^^^^^^^
DST transition metadata
So with a single timestamp on a given day all points of that entire day
can be represented without access to any additional metadata.
Timekeeping is fun!
Thanks,
-Brooks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mm.icann.org/pipermail/tz/attachments/20240109/8cfe47f1/attachment.htm>
More information about the tz
mailing list