[tz] tzfiles contain Unix epoch for the first transition time

Eric Erhardt Eric.Erhardt at microsoft.com
Fri Aug 14 19:16:29 UTC 2015


Here's the steps that I did.  I walked through the steps again just to make sure I got them all.

Using an Ubuntu 14.04 machine:

1. mkdir tzdata
2. cd tzdata
3. curl -O http://www.iana.org/time-zones/repository/releases/tzdata2015f.tar.gz
4. curl -O http://www.iana.org/time-zones/repository/releases/tzcode2015f.tar.gz
5. tar -xzf tzcode2015f.tar.gz
6. tar -xzf tzdata2015f.tar.gz
7. make TOPDIR=/home/eerhardt/tzdata/install install


I am reading the "V2" information of the tzfile - i.e. the 64-bit timestamp section.

Here is the byte dump of the America/Chicago file created with the above steps:

[1304]	0x54  --
[1305]	0x5a    | "TZif"
[1306]	0x69    |
[1307]	0x66  --
[1308]	0x32  --  "2"
[1309]	0x00  --
[1310]	0x00    |
[1311]	0x00    |
[1312]	0x00    |
[1313]	0x00    |
[1314]	0x00    |
[1315]	0x00    | 15 unused bytes
[1316]	0x00    |
[1317]	0x00    |
[1318]	0x00    |
[1319]	0x00    |
[1320]	0x00    |
[1321]	0x00    |
[1322]	0x00    |
[1323]	0x00  --
[1324]	0x00  --
[1325]	0x00    | tzh_ttisgmtcnt
[1326]	0x00    |
[1327]	0x07  --
[1328]	0x00  --
[1329]	0x00    | tzh_ttisstdcnt
[1330]	0x00    |
[1331]	0x07  --
[1332]	0x00  --
[1333]	0x00    | tzh_leapcnt
[1334]	0x00    |
[1335]	0x00  --
[1336]	0x00  --
[1337]	0x00    | tzh_timecnt
[1338]	0x00    |
[1339]	0xed  --
[1340]	0x00  --
[1341]	0x00    | tzh_typecnt
[1342]	0x00    |
[1343]	0x07  --
[1344]	0x00  --
[1345]	0x00    | tzh_charcnt
[1346]	0x00    |
[1347]	0x18  --
[1348]	0xf8  --
[1349]	0x00    |
[1350]	0x00    |
[1351]	0x00    | first transition time
[1352]	0x00    |
[1353]	0x00    |
[1354]	0x00    |
[1355]	0x00  --
[1356]	0xff  --
[1357]	0xff    |
[1358]	0xff    |
[1359]	0xff    | second transition time
[1360]	0x5e    |
[1361]	0x03    |
[1362]	0xfe    |
[1363]	0xa0  --


And when reading the file from /usr/share/zoneinfo/America/Chicago, I get the following bytes:

[1287]	0x54  --
[1288]	0x5a    | "TZif"
[1289]	0x69    |
[1290]	0x66  --
[1291]	0x32  --  "2"
[1292]	0x00  --
[1293]	0x00    |
[1294]	0x00    |
[1295]	0x00    |
[1296]	0x00    |
[1297]	0x00    |
[1298]	0x00    | 15 unused bytes
[1299]	0x00    |
[1300]	0x00    |
[1301]	0x00    |
[1302]	0x00    |
[1303]	0x00    |
[1304]	0x00    |
[1305]	0x00    |
[1306]	0x00  --
[1307]	0x00  --
[1308]	0x00    | tzh_ttisgmtcnt
[1309]	0x00    |
[1310]	0x07  --
[1311]	0x00  --
[1312]	0x00    | tzh_ttisstdcnt
[1313]	0x00    |
[1314]	0x07  --
[1315]	0x00  --
[1316]	0x00    | tzh_leapcnt
[1317]	0x00    |
[1318]	0x00  --
[1319]	0x00  --
[1320]	0x00    | tzh_timecnt
[1321]	0x00    |
[1322]	0xec  --
[1323]	0x00  --
[1324]	0x00    | tzh_typecnt
[1325]	0x00    |
[1326]	0x07  --
[1327]	0x00  --
[1328]	0x00    | tzh_charcnt
[1329]	0x00    |
[1330]	0x18  --
[1331]	0xff  --
[1332]	0xff    |
[1333]	0xff    |
[1334]	0xff    | first transition time
[1335]	0x5e    |
[1336]	0x03    |
[1337]	0xfe    |
[1338]	0xa0  --


So it looks like my issue is the "big bang" transition that didn't appear in the older tz files I was testing with and I didn't know this transition time existed.  This time value isn't possible to represent in .NET (since DateTime.MinValue is 00:00:00.0000000 UTC, January 1, 0001, in the Gregorian calendar).  The behavior I was seeing (getting a Unix epoch) was my own logic overflowing with this value, which is my mistake.

Here was the incorrect logic:

        // Windows NT time is specified as the number of 100 nanosecond intervals since January 1, 1601.
        // UNIX time is specified as the number of seconds since January 1, 1970. There are 134,774 days
        // (or 11,644,473,600 seconds) between these dates.
        //
        private static DateTime TZif_UnixTimeToWindowsTime(long unixTime)
        {
            // Add 11,644,473,600 and multiply by 10,000,000.
            Int64 ntTime = (((Int64)unixTime) + 11644473600) * 10000000;  //overflows for large positive or negative unixTime values
            return FromFileTimeUtc(ntTime);
        }

        private static DateTime FromFileTimeUtc(long fileTime)
        {
            // This is the ticks in Universal time for this fileTime.
            long universalTicks = fileTime + FileTimeOffset;
            return new DateTime(universalTicks, DateTimeKind.Utc);
        }

        private const int DaysTo1601 = DaysPer400Years * 4;          // 584388

        private const long TicksPerMillisecond = 10000;
        private const long TicksPerSecond = TicksPerMillisecond * 1000;
        private const long TicksPerMinute = TicksPerSecond * 60;
        private const long TicksPerHour = TicksPerMinute * 60;
        private const long TicksPerDay = TicksPerHour * 24;

        private const long FileTimeOffset = DaysTo1601 * TicksPerDay;



A couple questions I have about this "big bang" transition:

1. I shouldn't be checking explicitly for this value (0xf800000000000000), right?  I saw some code comments in zic.c that says it could potentially change in the future.

>    BIG_BANG is approximate, and may change in future versions.
       Please do not rely on its exact value.  */

2. Will there ever be more than one transition time that is before January 1, 0001?  Or will the "big bang" transition be the only one?


I'm thinking the way I'll handle the "big bang" transition is to represent it as DateTime.MinValue in .NET.  I'll check to see if the UnixTime is less than the minimum Unix Time possible, and use DateTime.MinValue instead.


Thanks Robert, Paul and Ian, for helping me out on this.

Eric


-----Original Message-----
From: Robert Elz [mailto:kre at munnari.OZ.AU] 
Sent: Friday, August 14, 2015 9:44 AM
To: Ian Abbott
Cc: Eric Erhardt; tz at iana.org
Subject: Re: [tz] tzfiles contain Unix epoch for the first transition time

    Date:        Fri, 14 Aug 2015 12:55:20 +0100
    From:        Ian Abbott <abbotti at mev.co.uk>
    Message-ID:  <55CDD728.7040401 at mev.co.uk>

  | If the initial transitions are also missing in Eric's tzfiles, perhaps
  | the bug is related to that.

After I sent the last message, I wondered if perhaps the system Eric used represented times in (fixed point) Java type notation, in milliseconds since the epoch, rather than seconds - but even with that form, the big bang timestamp doesn't overflow 64 bits (though it gets very close, just
2 non-zero bits remain, the sign, and the most significant bit).   Of
course so many meaningful bits are lost the value would be nonsense, but not 0.

Even moving the epoch back to 1900 (which some systems do) doesn't affect anything (if my back of the envelope calculation is right, the epoch would need to move back over 9 billion years - 2/3 of the way to the big bang, for a millisecond counter to produce 0 for the unix style big bang timestamp).
[Do not rely upon, nor quote off this list, that value - I did not verify.]

Of course, if the internal representation is microseconds (or any more precise unit) since the epoch (1970 or anything else plausible) then the big bang would (if overflow protection isn't perfect) turn into 0.

In any of those representations, very recent times, like 1883, fit in
64 bits just fine.

kre



More information about the tz mailing list