ICONZ Network Status
For the latest network status and for information on upcoming promotions and events, follow us on Twitter at http://twitter.com/ICONZGroupNZ
Current Network Status
There are no current issues on the ICONZ network
Recent Issues / Planned Maintenance
Incident Report – Server Secure (Shared Firewall) Degraded Service – 29/07/2010 14:51 – 15:01
Friday 30th July 2010
Incident Date/Time: Thursday 29th July 2.51pm – 3.01pm
Incident Duration: Approximately 10 minutes
Impact: 70%+ packet loss to all customer servers behind the shared firewall
Incident Summary:
At approximately 2.51pm on the 29th of July 2010, ICONZ monitoring systems identified service issues with our shared firewall service (Server Secure).
The problem was immediately escalated to our engineers, who diagnosed a Distributed Denial of Service attack targeted at a specific host behind the firewall resulting in heavy packet loss to all customers hosted behind the cluster.
Since the attack was mostly generated from offshore hosts, ICONZ engineers blocked international traffic to the target IP address. This significantly reduced the load on the Firewall Cluster and returned full service to other hosted customers at approximately 3.01pm, restoring service for affected customers.
As of this morning, our engineers have confirmed that the DDoS attack has subsided and have removed the null-route, restoring full service to the affected customer. We do not anticipate any further service issues at this time, but will be closely monitoring the situation throughout the day.
If you have any further questions or issues regarding this incident, please don’t hesitate to let us know.
14.07.2010 - International Connectivity - Degraded Service – 16:30 - 16:40
Incident Date/Time: Wednesday 14th July 4:30pm-4.40pm (approx)
Incident Duration: Approximately 10 minutes
Impact: Approximately 50% packet loss to international hosts from the ICONZ network
At approximately 4.30pm on the 14th of July 2010, ICONZ experienced a DOS attack from a single IP address directed at one of our colocation customers. This caused congestion across our primary international uplink,which would have resulted in heavy packet loss for traffic across that link.
The source IP address of the attack was quickly identified by our engineers and blocked by our upstream provider at approximately 4.40pm, at which time full service was restored.
If you have any further questions or issues regarding this incident, please don’t hesitate to let us know.
ICONZ Planned Maintenance – Email Planned Maintenance– 27/7/2010 05:00 – 07:00
Maintenance Date/Time: Tuesday 27th July 05:00 – 07:00
Expected Impact: 2-3 Minutes (approx)
Maintenance Summary:
All e-mail will be unavailable for 2-3 minutes as the mail store reboots after planned upgrades are installed on the mail store. No POP or IMAP services will be available during this time. Outbound (SMTP) mail will be unavailable for 2-3 minutes as the load balancer reboots. Inbound mail will fail, however sending servers should treat this as a temporary failure and retry.
As always, if you have any further questions or issues, please don’t hesitate to let us know.
ICONZ Planned Maintenance – Vector Planned Maintenance– 12/7/2010 00:01 – 04:00
Maintenance Date/Time: Monday 12th July 00:01 - 04:00
Expected Impact: 5 Minutes (approx)
Maintenance Summary:
Vector have advised us of planned maintenance occurring during the above window. This will cause a 5 minute expected outage to our link with IPSTAR at some point within the maintenance window.
As always, if you have any further questions or issues, please don’t hesitate to let us know.
Incident Report – Loss of International Connectivity – 24/05/2010 14:33 - 14:50 (NZST)
Friday 28th May 2010
Incident Date/Time: 24/05/2010 2.33pm -2.48pm
Incident Duration: 15 minutes (approx)
Impact: International Connectivity
Incident Summary:
At 2.33pm on Monday the 24th of May 2010, a power failure on the Auckland North Shore caused a transmission ring failure for our primary international supplier (Vocus). The Vocus network team was immediately notified of the problem and service was restored at approximately 2.39pm.
ICONZ customers would have experienced a 6-minute outage to international connectivity as the link was restored, followed by approximately 9 minutes of degraded service as international routing tables repopulated.
Following on from this, Vocus have conducted and audit of vulnerabilities and reassured us that they have taken the necessary steps to prevent a recurrence of this problem.
24/05/2010 - International Connectivity – 14:35 - RESOLVED
We experienced international connectivity issues between 14:35 and 14:45 on Monday the 24th of May, due to a problem with our primary international provider. They have now resolved the issue and international connectivity should be restored. An incident report will follow upon the completion of an analysis of the fault. ICONZ would like to apologise for any inconvenience this may have caused.
If you are still experiencing issues, please contact our support team on 0800 THE NET (0800 843 638).
18/05/2010 3:50am – 5.30am - Power Failure at Sky Tower
Incident Date/Time: Tuesday 18th May 2010 3:50am – 5.30am
Incident Duration: 100 mins (approx)
Impact:
• International connectivity: 2 minute outage as service failed over to redundant link
• National Connectivity: 60 minute service disruption – traffic re-routed through alternate paths
• Wired Country and Kordia customers – approximately 100 minute service outage.
Incident Summary:
At approximately 3.50am on the 18th of May 2010, ICONZ lost power to core network equipment located atthe Sky Tower. Engineers were notified and dispatched immediately to investigate.
Upon arrival, engineers identified the cause of the outage and commenced remedial work. Full service was restored at 5.30am.
We are pleased to report that all services (with the exception of Wired Country and Kordia) failed over to redundant infrastructure, and service disruption for our customers was minimal. However, in light of this event, ICONZ will be looking to further improve redundancy at the Sky Tower and further minimize the risk profile of our network.
20/04/2010 2.00pm - Packet loss to Wellington - resolved
ICONZ customers will currently be experiencing packet loss over our inter-city link to Wellington.
Our engineers are currently working on the issue and hope to have a resolution shortly
20/04/2010 9.30am-10.30am Packet Loss to TelstraClear
ICONZ customers will have experienced approximately 15-20% packet loss to hosts on the TelstraClear network from around 9.30am, due to a routing issue at ICONZ. Engineers were immediately notified, and restored full service at approximately 10.30am.
If you are still experiencing issues, please contact our support team on 0800 THE NET (0800 843 638).
Incident Report – Loss of connectivity to USA – 30/03/2010 09:30 – 09:47
30th March 2010
Incident Date/Time: Tuesday 30th March 2010 9.30am – 9.47am
Incident Duration: Approximately 17 minutes
Impact: Loss of connectivity to USA
Incident Summary:
At approximately 9.30am on Tuesday the 30th of March 2010, our primary international carrier Vocus experienced a physical cut to one of their fibre circuits resulting in a loss of connectivity to the USA for all ICONZ customers.
The loss of connectivity we experienced was caused by a fault with the configuration of the failover process at the Vocus end; the Vocus circuit did not fail over cleanly to the secondary Vocus link. This is a teething issue with the changeover from Pacnet to Vocus as our upstream provider. Vocus has assured ICONZ that a fully automated failover will be in place by 1st April and our engineers are monitoring progress.
We’d like to apologize for any inconvenience caused.
As always, if you have any further questions or concerns regards to this incident, please don’t hesitate to contact us.
No Connectivity to USA - 30/03/2010 - 9.30am - 9.47am (approx)
ICONZ customers would have been unable to access hosts in the USA between approximately 9.30am and 9.47am today, due to a network fault. Full service has now been restored, and our engineers are monitoring the situation closely.
We will be conducting full root cause analysis of this problem, and posting further information as it comes to light.
Incident Report – International Connectivity – 26/03/2010 14:20 – 14:35
Incident Date/Time: Friday 26th March 2.20pm – 2.35pm (approx)
Incident Duration: Approximately 15 minutes
Impact: Loss of international connectivity
Incident Summary:
At approximately 2.20pm today the 26th of March 2010, our primary upstream provider for international connectivity (Vocus) experienced a physical fault with one of their fibre links to the Sky Tower, causing a loss of international connectivity for all ISPs using Vocus, including ICONZ.
Vocus have redundant connectivity on this link, but it appears traffic did not immediately fail over to their secondary circuit. ICONZ were prepared to fail over to our redundant international provider, but Vocus restored service at 2.35pm, negating the need to do so.
Our engineers are will be on high alert over the weekend monitoring for any further problems, and will be working with Vocus to identify the reason for the delay in failover. Moving forward, we will be rigorously testing redundancy to ensure that we do not see future recurrences of this issue.
As always, if you have any further questions or concerns regards to this incident, please don’t hesitate to contact us.
Incident Report – Slow Speeds to the USA – 24/03/2010 14:00 – 14:15
Published 25th March 2010
Incident Date/Time: Wednesday 24th March 2pm-2.15pm (approx)
Incident Duration: Approximately 15 minutes
Impact: Slow speeds and/or timeouts to hosts in the USA
Incident Summary:
At 2pm yesterday the 24th of March 2010, Vocus (our primary upstream provider for international connectivity) experienced a problem in their network which resulted in slow speeds and timeouts for customers attempting to reach hosts in the USA.
Our engineers quickly identified the problem and escalated to Vocus, who resolved the issue at approximately 2.15pm, restoring normal service.
We understand the problem was caused by a mis-configuration at Vocus, and we are following up with them to ensure that our customers do not experience a repeat of this issue.
If you have any further questions, please contact our support team on 0800 843 638 or help@iconz.net
ICONZ Planned Maintenance – International Connectivity – 23/3/2010 02:00 – 03:00
Maintenance Date/Time: Tuesday 23rd March 2.00am-3.00am
Expected Impact: 2 x micro-outages as we fail over to redundant connectivity
Maintenance Summary:
Vocus (our primary international connectivity provider) have advised us of planned maintenance occurring during the above window. This will cause a 10 minute drop to our link to Vocus at some point within the maintenance window. During this drop, international connectivity will fail over to our secondary provider, Global Gateway.
ICONZ customers will notice a micro-outage as the service fails over, and another 10 minutes later as we cut back over to Vocus. You may also notice different routing paths for international hosts, as traffic is routed through Global Gateway.
As always, if you have any further questions or issues, please don’t hesitate to let us know.