[ICANN-CSC] Notification of a recent service performance incident

Amy Creamer amy.creamer at iana.org
Wed Jun 8 20:08:42 UTC 2022

Dear CSC,

We would like to inform you of a system issue that degraded our service performance recently. This issue will impact SLA performance for May, and is expected to have an impact on our performance in June.

On 20 May 2022, during a planned upgrade to the Root Zone Management Systems (RZMS), an engineer managing the release mistakenly altered a system system setting that caused automatically generated emails from the system to not be delivered to external parties. These emails included prompts for TLD contacts to authorize change requests, and notifications of system events such as technical check remedy and completion. Additionally, users were unable to receive links via email to reset their passwords.

The issue was not caught through our SLA monitoring as emails were being sent successfully from the system, but not being routed correctly once they had left RZMS. Customers first reporting issues on 27 May 2022, and the issue was identified and resolved on 31 May 2022.

We’ve identified that six TLD change requests and five users attempting to reset their password were impacted. While the issue did not trigger any deviating from the SLAs as they are measured, we believe it would be consistent with the intent of the SLAs for these to show these instances as breaching the SLAs. For this reason, we have proactively adjusted the timings for the SLAs in these cases where possible.

IANA is working with ICANN’s Engineering and IT Department on adding additional safeguards on future upgrades to prevent this issue from recurring.

We are happy to answer any questions you may have, either on the list or at our meeting next week.


Amy Creamer
Director of Operations, IANA Services
Email: amy.creamer at iana.org<mailto:amy.creamer at iana.org>
Phone: +1-424-537-8917
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mm.icann.org/pipermail/icann-csc/attachments/20220608/28718328/attachment.html>

More information about the ICANN-CSC mailing list