[ICANN-CSC] Notification of a recent service performance incident

[Gaurav Vedi]

Thanks Amy for keeping us informed.

Glad that the issue has been fixed. As Brett suggested, we should briefly
discuss this in the next CSC meeting.
Just a quick question. Did the engineer manually changed the system setting
or changed some automation/deployment scripts incorrectly which caused the
issue ? May be worth hardening the process here for robust deployment.

Regards,
Gaurav Vedi

On Thu, Jun 9, 2022 at 10:44 AM lpyadav via ICANN-CSC <icann-csc at icann.org>
wrote:

> Dear Amy
> Thank FYI
> Laxmi Prasad Yadav
> Nepal
> ----- Original Message -----
> From: Brett Carr via ICANN-CSC <icann-csc at icann.org>
> To: Amy Creamer <amy.creamer at iana.org>, Gaurav Vedi via ICANN-CSC <
> icann-csc at icann.org>
> Sent: Thu, 09 Jun 2022 18:29:55 +0545 (NPT)
> Subject: Re: [ICANN-CSC] Notification of a recent service performance
>  incident
>
> Amy,
>     Thanks for keeping us informed, it feels like a one off issue that you
> have identified a fix for and are learning lessons from (we’ve all been
> there), so I don’t believe there is anything for the CSC to worry about
> here. Perhaps worth spending 10 mins in next weeks meeting to give us a
> quick overview of what happened, the impact and changes you are making to
> ensure it doesn’t re-occur.
>
> Thanks
>
> Brett
>
> --
> Brett Carr
> Manager DNS Engineering
> Nominet UK
>
>
> From: ICANN-CSC <icann-csc-bounces at icann.org> on behalf of Amy Creamer
> via ICANN-CSC <icann-csc at icann.org>
> Date: Wednesday, 8 June 2022 at 21:08
> To: Gaurav Vedi via ICANN-CSC <icann-csc at icann.org>
> Subject: [ICANN-CSC] Notification of a recent service performance incident
> Dear CSC,
>
> We would like to inform you of a system issue that degraded our service
> performance recently. This issue will impact SLA performance for May, and
> is expected to have an impact on our performance in June.
>
> On 20 May 2022, during a planned upgrade to the Root Zone Management
> Systems (RZMS), an engineer managing the release mistakenly altered a
> system system setting that caused automatically generated emails from the
> system to not be delivered to external parties. These emails included
> prompts for TLD contacts to authorize change requests, and notifications of
> system events such as technical check remedy and completion. Additionally,
> users were unable to receive links via email to reset their passwords.
>
> The issue was not caught through our SLA monitoring as emails were being
> sent successfully from the system, but not being routed correctly once they
> had left RZMS. Customers first reporting issues on 27 May 2022, and the
> issue was identified and resolved on 31 May 2022.
>
> We’ve identified that six TLD change requests and five users attempting to
> reset their password were impacted. While the issue did not trigger any
> deviating from the SLAs as they are measured, we believe it would be
> consistent with the intent of the SLAs for these to show these instances as
> breaching the SLAs. For this reason, we have proactively adjusted the
> timings for the SLAs in these cases where possible.
>
> IANA is working with ICANN’s Engineering and IT Department on adding
> additional safeguards on future upgrades to prevent this issue from
> recurring.
>
> We are happy to answer any questions you may have, either on the list or
> at our meeting next week.
>
>
> Regards,
>
> Amy Creamer
> Director of Operations, IANA Services
> Email: amy.creamer at iana.org<mailto:amy.creamer at iana.org>
> Phone: +1-424-537-8917
>
> _______________________________________________
> ICANN-CSC mailing list
> ICANN-CSC at icann.org
> https://mm.icann.org/mailman/listinfo/icann-csc
>
> _______________________________________________
> By submitting your personal data, you consent to the processing of your
> personal data for purposes of subscribing to this mailing list accordance
> with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and
> the website Terms of Service (https://www.icann.org/privacy/tos). You can
> visit the Mailman link above to change your membership status or
> configuration, including unsubscribing, setting digest-style delivery or
> disabling delivery altogether (e.g., for a vacation), and so on.



-- 
Regards,
Gaurav Vedi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mm.icann.org/pipermail/icann-csc/attachments/20220609/a46010ff/attachment.html>