[tz] tzcode test suite

Steve Summit scs at eskimo.com
Fri Jan 15 13:30:51 UTC 2016


Paul wrote:
> Adding 1900 to tm_year can overflow, if tm_year is close to INT_MAX. 
> quick_timegm is full of gotchas like that. As it's not needed and is 
> likely to have bugs in testing, how about if we remove it?

Oh, it's needed, and its shortcomings were actually a deliberate
design decision!  I'll explain.

First of all, for core functions like gmtime and mktime that have
billions of possible input combinations and lots of internal
complexity, I favor a mixture of explicitly-crafted and automatic
tests.  Thus the distinction between the "scripted" and
"exhaustive" modes of this test suite.  The exhaustive/automatic
tests have a marvelous way of ferreting out obscure bugs that
explicitly handcrafted tests would never have discovered.
(You know this, but I'm just laying out the background.
And I don't honestly expect tzcode to have any of those obscure
bugs at this point, but it's the principle that counts.)

For the "automatic" tests, the challenge is obviously to
automatically generate the expected results, without doing a
full, parallel, and equally potentially buggy rewrite of the
complicated algorithm under test.  In this case, the trick is to
use a simpler and more obviously correct parallel implementation,
that takes liberties which would not be acceptable in the
production code, but which are contrived to be just good enough
to adequately perform the intended testing.

I'll add a comment to explain this, but quick_timegm is precisely
the simpler, more obviously correct, parallel reimplementation
on which the legitimacy of the "exhaustive" tests depends.
It would be very wrong to replace the call to quick_timegm with
a call to plain timegm, since timegm is (one of) the functions
under test!  Calling timegm there would be to allow a buggy
timegm to generate compatibly buggy test cases for itself,
which it would then fraudulently pass all of.  Quis custodiet
ipsos custodes.

The real timegm takes rather extreme pains to avoid overflow near
the edges, and I respect that.  quick_timegm takes no such pains,
because (a) that makes the code simpler and more obviously
correct, and (b) it doesn't need to.  The intent is that the
"exhaustive" tests are not to be run right out to the edges of
time_t's range.  We can use explicit, handcrafted, scripted
(i.e. non-automatic) tests out near the edges, instead.

Make sense?

(But I do need to find and dust off an old, 32-bit machine to
test the test suite on.  Funny to be calling a 32-bit machine "old".)


More information about the tz mailing list