ICONZ Network Status
For the latest network status and for information on upcoming promotions and events, follow us on Twitter at http://twitter.com/ICONZGroupNZ
Current Network Status
There are currently no known issues on the ICONZ network
Recent Issues / Planned Maintenance
Incident Report – Sky Tower Power Failure – 24.08.2010 12:59 – 15:20
31st August 2010
Incident Date/Time: 24.08.2010 12.59pm – 2.30pm
Incident Duration: Approximately 2.5 hours
Services Impacted: International connectivity (approx. 10 mins), ICONZ Extend Wireless (approx. 2 hours), Araneo (approx. 2.5 hours).
Incident Summary:
At approximately 12.59pm on the 24th of August 2010, ICONZ Monitoring Systems presented alerts for ICONZ equipment located at the Auckland Sky Tower, as well as alerts for ICONZ Extend Wireless and Araneo services, and our primary international connectivity gateway via Vocus.
ICONZ engineers were immediately alerted to the problem, and made contact with staff at the Sky Tower, who confirmed that had been an electrical incident at Skycity which had resulted in a Fire Brigade Callout, and required full electrical isolation of the Skytower Datacentre. We were advised that power would be restored as quickly as possible. At this time, ICONZ equipment at Sky Tower appeared to be operating on UPS battery power, although devices belonging to some of our suppliers appeared to be offline.
At approximately 1.05pm, international connectivity was restored when Vocus equipment at the Sky Tower came online running on UPS. ICONZ Extend Wireless and Araneo services still appeared to be offline at this time.
At approximately 1.45pm ICONZ confirmed that the fire at Sky Tower had been suppressed, and that mains power would be restored shortly.
At approximately 2.25pm, mains power was restored to the Sky Tower, and all ICONZ equipment returned to normal. ICONZ Extend Wireless services were also restored at this time, but Araneo service remained offline.
Araneo reported to ICONZ that it appeared that power to their UPS had run down, and thus they required access to the Sky Tower to manually reset their equipment and restore power. Due to safety restrictions, they were unable to gain access until approximately 3.15pm, at which time full service was restored. Araneo have assured us that they are taking steps to mitigate this risk moving forward.
It should be noted that customers may have been unable to access various services hosted on other Service Provider’s networks – several other providers were also affected by this outage.
If you have any further questions or issues regarding this outage, please don’t hesitate to contact us. For a full history of events, and for up-to-date information on future network events, customers are encouraged to visit our Twitter page at http://twitter.com/ICONZGroupNZ.
Araneo Connectivity Issues - 24.08.2010 - Resolved
We are currently experiencing issues with our Araneo network link due to a major incident at sky tower. A network technician has been dispatched and we hope to have this resolved soon
*** This issue is now resolved, incident report to follow shortly ***
International Connectivity issues - 24.08.2010 - Resolved
We are currently experiencing international connectivity issues. Our engineers are working on the issue at the moment and hope to have this resolved soon.
*** This issue is now resolved, incident report to follow shortly ***
Incident Report – Auckland-Wellington Network Issues – 17.08.2010 01:40 – 02:40
Incident Date/Time: 17.08.2010 1.40am – 02.40am
Incident Duration: Approximately 1 hour
Impact: Loss of connectivity for ICONZ customers in Wellington
Incident Summary:
On the morning of 17th August 2010, ICONZ Auckland-Wellington inter-metro connectivity supplier (TelstraClear) was performing planned (non-service impacting) maintenance on their IP Core Network.
During the course of this maintenance, at approximately 01:40am TelstraClear experienced an unexpected service outage which caused a loss of connectivity on both our primary and secondary (redundant) links between Auckland and Wellington.
ICONZ engineers were immediately notified, and the problem was escalated to TelstraClear, who confirmed a fault within their network. This fault was resolved at approximately 02:40am, restoring full service for ICONZ customers.
In terms of impact, ICONZ customers in Wellington would have noticed a substantial loss of connectivity. Wellington-bound traffic for all other customers failed over to diverse paths, but users may have noticed marginally increased latency and alternate routing during the incident window.
We have been assured by TelstraClear that they have investigated cause of the outage and will be taking action to prevent a recurrence.
If you have any further questions or issues, please don’t hesitate to contact us.
Kind Regards,
ICONZ Corporate Support
TELECOM PLANNED EVENT NOTIFICATION - 20/8/2010 03:00 – 06:00
Maintenance Date/Time: Friday 20th August 03:00 – 06:00
Expected Impact: 10 minutes between 03:00 - 06:00
Services Impacted: All ICONZ ADSL connections will be affected
Maintenance Summary:
Telecom will be performing maintenance to the Mayoral Drive Juniper Node which will cause all ICONZ ADSL connections to be dropped for 10 minutes during the hours of 03:00 - 06:00 on 20/08.
The connection should automatically connect up after the outage, if you notice any problems a reboot of the CPE should be the first step in bringing the connection up. If you are still not connected please give our customer care team a call on 0800 843 638
If you have any further questions or issues, please don’t hesitate to let us know.
17/08/10 - Wellington Primary and Secondary circuits down.
Telstraclear have informed us about a major network issue at their end in wellington which is causing some of our customers connectivity issues. Unfortunately we have not been given a ETA for the issue to be resolved. Updates will be posted as soon as our engineers receive any new information.
As always, if you have any further questions or issues, please don’t hesitate to let us know.
[RESOLVED] at 2:30am.
Incident Report – Telecom Peering Issues – 05/08/2010 21:37 – 01:00
Incident Date/Time: 05/08/2010 9.37pm – 06/08/2010 1.00am
Incident Duration: 3 hours 23 minutes
Impact: Loss of ICONZ peering link to Telecom (traffic failed over to diverse paths)
Incident Summary:
At approximately 9.37pm on the 5th of August 2010, ICONZ monitoring systems detected an outage to the ICONZ-Telecom peering link. The problem was immediately escalated to ICONZ engineers, and a trouble ticket was raised with Telecom.
ICONZ engineers diagnosed a hardware problem at the Sky Tower, and resolved the issue at approximately 1.00am, restoring full service.
Traffic immediately failed over to diverse paths, but customers may have noticed alternate routing and slightly degraded performance to Telecom hosts during the outage period.
If you have any further questions or issues regarding this incident, please don’t hesitate to let us know.
Network issues between ICONZ and Telecom - 05/08/2010 9.30pm approx - RESOLVED
At approximately 9.30pm on 05/08/2010 the peering between ICONZ and Telecom dropped. Our network engineers were made aware of the issue and started working to resolve the problem. Traffic was rerouted to our secondary link until the issue was resolved.
If you have any further questions or issues regarding this incident, please don’t hesitate to let us know. An Incident report will be available shortly
Incident Report – Server Secure (Shared Firewall) Degraded Service – 29/07/2010 14:51 – 15:01
Friday 30th July 2010
Incident Date/Time: Thursday 29th July 2.51pm – 3.01pm
Incident Duration: Approximately 10 minutes
Impact: 70%+ packet loss to all customer servers behind the shared firewall
Incident Summary:
At approximately 2.51pm on the 29th of July 2010, ICONZ monitoring systems identified service issues with our shared firewall service (Server Secure).
The problem was immediately escalated to our engineers, who diagnosed a Distributed Denial of Service attack targeted at a specific host behind the firewall resulting in heavy packet loss to all customers hosted behind the cluster.
Since the attack was mostly generated from offshore hosts, ICONZ engineers blocked international traffic to the target IP address. This significantly reduced the load on the Firewall Cluster and returned full service to other hosted customers at approximately 3.01pm, restoring service for affected customers.
As of this morning, our engineers have confirmed that the DDoS attack has subsided and have removed the null-route, restoring full service to the affected customer. We do not anticipate any further service issues at this time, but will be closely monitoring the situation throughout the day.
If you have any further questions or issues regarding this incident, please don’t hesitate to let us know.
14.07.2010 - International Connectivity - Degraded Service – 16:30 - 16:40
Incident Date/Time: Wednesday 14th July 4:30pm-4.40pm (approx)
Incident Duration: Approximately 10 minutes
Impact: Approximately 50% packet loss to international hosts from the ICONZ network
At approximately 4.30pm on the 14th of July 2010, ICONZ experienced a DOS attack from a single IP address directed at one of our colocation customers. This caused congestion across our primary international uplink,which would have resulted in heavy packet loss for traffic across that link.
The source IP address of the attack was quickly identified by our engineers and blocked by our upstream provider at approximately 4.40pm, at which time full service was restored.
If you have any further questions or issues regarding this incident, please don’t hesitate to let us know.
ICONZ Planned Maintenance – Email Planned Maintenance– 27/7/2010 05:00 – 07:00
Maintenance Date/Time: Tuesday 27th July 05:00 – 07:00
Expected Impact: 2-3 Minutes (approx)
Maintenance Summary:
All e-mail will be unavailable for 2-3 minutes as the mail store reboots after planned upgrades are installed on the mail store. No POP or IMAP services will be available during this time. Outbound (SMTP) mail will be unavailable for 2-3 minutes as the load balancer reboots. Inbound mail will fail, however sending servers should treat this as a temporary failure and retry.
As always, if you have any further questions or issues, please don’t hesitate to let us know.
ICONZ Planned Maintenance – Vector Planned Maintenance– 12/7/2010 00:01 – 04:00
Maintenance Date/Time: Monday 12th July 00:01 - 04:00
Expected Impact: 5 Minutes (approx)
Maintenance Summary:
Vector have advised us of planned maintenance occurring during the above window. This will cause a 5 minute expected outage to our link with IPSTAR at some point within the maintenance window.
As always, if you have any further questions or issues, please don’t hesitate to let us know.
Incident Report – Loss of International Connectivity – 24/05/2010 14:33 - 14:50 (NZST)
Friday 28th May 2010
Incident Date/Time: 24/05/2010 2.33pm -2.48pm
Incident Duration: 15 minutes (approx)
Impact: International Connectivity
Incident Summary:
At 2.33pm on Monday the 24th of May 2010, a power failure on the Auckland North Shore caused a transmission ring failure for our primary international supplier (Vocus). The Vocus network team was immediately notified of the problem and service was restored at approximately 2.39pm.
ICONZ customers would have experienced a 6-minute outage to international connectivity as the link was restored, followed by approximately 9 minutes of degraded service as international routing tables repopulated.
Following on from this, Vocus have conducted and audit of vulnerabilities and reassured us that they have taken the necessary steps to prevent a recurrence of this problem.
24/05/2010 - International Connectivity – 14:35 - RESOLVED
We experienced international connectivity issues between 14:35 and 14:45 on Monday the 24th of May, due to a problem with our primary international provider. They have now resolved the issue and international connectivity should be restored. An incident report will follow upon the completion of an analysis of the fault. ICONZ would like to apologise for any inconvenience this may have caused.
If you are still experiencing issues, please contact our support team on 0800 THE NET (0800 843 638).
18/05/2010 3:50am – 5.30am - Power Failure at Sky Tower
Incident Date/Time: Tuesday 18th May 2010 3:50am – 5.30am
Incident Duration: 100 mins (approx)
Impact:
• International connectivity: 2 minute outage as service failed over to redundant link
• National Connectivity: 60 minute service disruption – traffic re-routed through alternate paths
• Wired Country and Kordia customers – approximately 100 minute service outage.
Incident Summary:
At approximately 3.50am on the 18th of May 2010, ICONZ lost power to core network equipment located atthe Sky Tower. Engineers were notified and dispatched immediately to investigate.
Upon arrival, engineers identified the cause of the outage and commenced remedial work. Full service was restored at 5.30am.
We are pleased to report that all services (with the exception of Wired Country and Kordia) failed over to redundant infrastructure, and service disruption for our customers was minimal. However, in light of this event, ICONZ will be looking to further improve redundancy at the Sky Tower and further minimize the risk profile of our network.
20/04/2010 2.00pm - Packet loss to Wellington - resolved
ICONZ customers will currently be experiencing packet loss over our inter-city link to Wellington.
Our engineers are currently working on the issue and hope to have a resolution shortly
20/04/2010 9.30am-10.30am Packet Loss to TelstraClear
ICONZ customers will have experienced approximately 15-20% packet loss to hosts on the TelstraClear network from around 9.30am, due to a routing issue at ICONZ. Engineers were immediately notified, and restored full service at approximately 10.30am.
If you are still experiencing issues, please contact our support team on 0800 THE NET (0800 843 638).
Incident Report – Loss of connectivity to USA – 30/03/2010 09:30 – 09:47
30th March 2010
Incident Date/Time: Tuesday 30th March 2010 9.30am – 9.47am
Incident Duration: Approximately 17 minutes
Impact: Loss of connectivity to USA
Incident Summary:
At approximately 9.30am on Tuesday the 30th of March 2010, our primary international carrier Vocus experienced a physical cut to one of their fibre circuits resulting in a loss of connectivity to the USA for all ICONZ customers.
The loss of connectivity we experienced was caused by a fault with the configuration of the failover process at the Vocus end; the Vocus circuit did not fail over cleanly to the secondary Vocus link. This is a teething issue with the changeover from Pacnet to Vocus as our upstream provider. Vocus has assured ICONZ that a fully automated failover will be in place by 1st April and our engineers are monitoring progress.
We’d like to apologize for any inconvenience caused.
As always, if you have any further questions or concerns regards to this incident, please don’t hesitate to contact us.
No Connectivity to USA - 30/03/2010 - 9.30am - 9.47am (approx)
ICONZ customers would have been unable to access hosts in the USA between approximately 9.30am and 9.47am today, due to a network fault. Full service has now been restored, and our engineers are monitoring the situation closely.
We will be conducting full root cause analysis of this problem, and posting further information as it comes to light.
Incident Report – International Connectivity – 26/03/2010 14:20 – 14:35
Incident Date/Time: Friday 26th March 2.20pm – 2.35pm (approx)
Incident Duration: Approximately 15 minutes
Impact: Loss of international connectivity
Incident Summary:
At approximately 2.20pm today the 26th of March 2010, our primary upstream provider for international connectivity (Vocus) experienced a physical fault with one of their fibre links to the Sky Tower, causing a loss of international connectivity for all ISPs using Vocus, including ICONZ.
Vocus have redundant connectivity on this link, but it appears traffic did not immediately fail over to their secondary circuit. ICONZ were prepared to fail over to our redundant international provider, but Vocus restored service at 2.35pm, negating the need to do so.
Our engineers are will be on high alert over the weekend monitoring for any further problems, and will be working with Vocus to identify the reason for the delay in failover. Moving forward, we will be rigorously testing redundancy to ensure that we do not see future recurrences of this issue.
As always, if you have any further questions or concerns regards to this incident, please don’t hesitate to contact us.
Incident Report – Slow Speeds to the USA – 24/03/2010 14:00 – 14:15
Published 25th March 2010
Incident Date/Time: Wednesday 24th March 2pm-2.15pm (approx)
Incident Duration: Approximately 15 minutes
Impact: Slow speeds and/or timeouts to hosts in the USA
Incident Summary:
At 2pm yesterday the 24th of March 2010, Vocus (our primary upstream provider for international connectivity) experienced a problem in their network which resulted in slow speeds and timeouts for customers attempting to reach hosts in the USA.
Our engineers quickly identified the problem and escalated to Vocus, who resolved the issue at approximately 2.15pm, restoring normal service.
We understand the problem was caused by a mis-configuration at Vocus, and we are following up with them to ensure that our customers do not experience a repeat of this issue.
If you have any further questions, please contact our support team on 0800 843 638 or help@iconz.net
ICONZ Planned Maintenance – International Connectivity – 23/3/2010 02:00 – 03:00
Maintenance Date/Time: Tuesday 23rd March 2.00am-3.00am
Expected Impact: 2 x micro-outages as we fail over to redundant connectivity
Maintenance Summary:
Vocus (our primary international connectivity provider) have advised us of planned maintenance occurring during the above window. This will cause a 10 minute drop to our link to Vocus at some point within the maintenance window. During this drop, international connectivity will fail over to our secondary provider, Global Gateway.
ICONZ customers will notice a micro-outage as the service fails over, and another 10 minutes later as we cut back over to Vocus. You may also notice different routing paths for international hosts, as traffic is routed through Global Gateway.
As always, if you have any further questions or issues, please don’t hesitate to let us know.