[tz] Regression Tests and Revision Controls

Brian Inglis Brian.Inglis at systematicsw.ab.ca
Sat Aug 9 16:54:47 UTC 2014


On 2014-08-08 17:21, Lester Caine wrote:
> That I think is a nice summary Tim
>
> All I would add is perhaps a reference to the amount of work required to
> fix the regressions in testing third party interfaces like PHP and Java
> and other platforms. It is the fact that we can not gauge if any of the
> changes do affect users that is the problem. That someone now gets a
> different answer in a grey area of the data may well not be important,
> but not knowing that a change HAS happened means that the effect of a
> change can not be reviewed? And perhaps the very evidence we are looking
> for discovered ...

Perhaps some comments on the scale and effort required for commercial/professional
regression test suites and SOPs would be appropriate.

I would expect a regression test suite to run zic to list all transitions in all zones,
diff that output against the baseline output, look for release notes explaining why each
change has occurred, document that; follow up any unexplained diffs with the maintainer,
and document that, then pass the diff and documentation to the product release manager,
for review and reappraisal or approval to promote the change to a production release.

If someone could contribute such a regression test script and baseline output, it would
allow the maintainer to better evaluate the potential impact of any proposed changes
on products using the package.

I also agree that normal revision control practice is to create a (tagged) branch for each
potential change, review the diffs against the trunk, make any other changes indicated and
add to the branch, then add the branch to the proposed changes expected to be merged into
the trunk in the next release. On release, add the release tag to everything in the trunk.

This also has the benefit that downstream consumers of the packages can more easily create
their own forks which could exclude some changes or include their own patches.

Each branch change would also normally include the output of the regression test suite
run against those changes and checked in with the code.

-- 
Take care. Thanks, Brian Inglis


More information about the tz mailing list