|
Hi all,
Further to my previous email, we have now completed our root cause analysis. Please see the following service announcement for the details.
Thanks again for your patience and apologies for the inconvenience this has caused.
Anton
From: Anton Strydom <anton at apnic dot net>
Date: Wednesday, 16 March 2016 11:41 am To: "apnic-talk at lists dot apnic dot net" <apnic-talk at lists dot apnic dot net> Subject: Update on APNIC IPv4 reverse DNS zones validation Hi all,
I wanted to provide an update on the problem which affected all APNIC IPv4 reverse DNS zones.
APNIC was notified via the public DNS operations mailing lists on Wednesday, 16/03 at 00:30 UTC+10 of a DNSSEC problem which affected all APNIC IPv4 reverse DNS zones.
During the reverse zone updates, which happened on Tuesday, 15/03 at 09:15 UTC+10, incorrect DS records were used. Once we became aware of the problem, the APNIC network operations team began working on resolving the issue immediately and all zones were corrected
by Wednesday, 16/03 at 02:40 UTC+10. Due to DNS cache effects, the old cached data may take time to expire in the public DNS and be replaced with the new information.
APNIC do have testing scripts to check for this kind of operational problem; however, in this instance the checks did not detect the issue. The team is currently investigating the root cause of this monitoring failure.
We would like to thank the operations community for reporting the issue to us. We recognise this is a significant service outage.
A more detailed service announcement is being compiled and I'll email the list again with that information once it is completed. Thanks again for your patience and apologies for the inconvenience this outage caused.
Regards
Anton
Technical Services Director
APNIC
|