ICONZ Network Status
Current Network Status
ICONZ Planned Maintenance – Usage Reports and E-mail – 09/02/2010 and 16/02/2010 05:00 – 07:00
Maintenance Date/Time: 09/02/2010 5.00am-7.00am and 16/02/2010 5.00am-7.00am
Expected Impact:
09/02/2010: Usage stats unavailable from the ICONZ Portal (http://portal.iconz.co.nz)
16/02/2010: ICONZ e-mail systems unavailable
Maintenance Summary:
As part of our ongoing efforts to maintain service stability and improve customer experience, ICONZ will be performing planned maintenance on the dates listed above. Unfortunately, this will cause minor service outages during the windows listed:
On the 9th of February 2010, usage statistics will be unavailable through the ICONZ Portal (http://portal.iconz.co.nz) for approximately 30 minutes between the 5.00am and 7.00am maintenance window.
On the 16th of February 2010, access to ICONZ e-mail will be unavailable for approximately 30 minutes between the 5.00am and 7.00am maintenance window.
No e-mail or usage data will be lost during this maintenance, but these systems will be inaccessible at the times listed.
We apologise for any inconvenience, and invite you to contact us if you have any further questions or concerns.
Planned Maintenance – Change to International Connectivity – 03/02/2010 05:00 – 07:00
Maintenance Date/Time: 03/02/2010 5.00am – 7.00am
Expected Impact: No service outage is expected. There will be changes in routing as International Connectivity fails over to our Global Gateway link at 5am, and then another change as we migrate to our new primary international provider, Vocus.
Maintenance Summary:
As part of our ongoing efforts to deliver the best possible service to our customers, ICONZ is changing our primary supplier for international connectivity from Pacnet (formerly AsiaNetCom) to Vocus.
The migration of traffic to the new link will take place between 5.00am and 7.00am on Wednesday the 3rd of February 2010. During this time, traffic will be routed to our secondary international provider - Global Gateway. Following the completion of this work, all international traffic will be routed through Vocus.
We do not expect any service outage during the migration. We do expect improved performance to most hosts in Australia (amongst other locations) following this change.
If you have any further questions or concerns regarding the above, or if you experience any problems following the change, please don’t hesitate to let us know.
Previous Issues
Wired Country Issue - 12/01/10 2.30pm-12.00am - Resolved
Wired country customers in the Auckland area will be experiencing a loss of connection due to Fibre cable being cut at one of the major base stations.
Wired country have field force technicians on site investigating the issue.
Wired Country Issues - 4/1/2010 9.30am-11.00pm - Resolved
Wired Country customers in the North Island may be experiencing difficulties connecting to Wireless or Wired Country Fibre connections. The fault arose due to a fibre cable being severed in a central Auckland construction zone. This issue is expected to be resolved before 11pm this evening.
Wired Country Issues - 23/12/2009 9.28am-1.38pm - Resolved
Wired Country customers in and around South Auckland may have been unable to connect to the internet this morning, due to a fault at the Kaipara Road base station.
Wired Country have rectified this fault, and connectivity was restored at around 11.30 AM.
ICONZ Email Issues - 21/12/2009 12.06pm-4.00pm - Resolved
Some ICONZ email users may have experienced inbound mail bouncing back to the sender between between 12.06pm and 4.00pm today, due to a configuration error at our side. Our engineers have permanently resolved the issue, and we don't expect any further problems.
International Connectivity Issues - 17/12/2009 - 4:05am - 4:35am - Resolved .
ICONZ customers may have experienced international connectivity issues from approximately 4:05am on 17/12/2009, until 4:35am. Our engineers have resolved this issue.
Webmail Performing Slowly - 15/12/2009 - 1.00pm-1.30pm - Resolved
We experienced slow performance on our webmail service, which will have caused issues for customers attempting to check e-mail via their web browsers between 1pm and 1.30pm. Customers checking e-mail via POP3 mail programs (such as Outlook or Thunderbird) were unaffected.
The issue was resolved by our engineers at 1.30pm.
Minor Packet Loss - Araneo Connections - 11/12/2009 - 5:00pm - 12:00am Resolved
ICONZ Araneo customers may have suffered from a low level of packet loss during the times indicated above due to a problem within the Kordia network. The problem was resolved by Kordia technicians at 12.00am.
Planned Maintenance - Wired Country Services - Friday/Saturday (11-12th December) - 23:30 - 01:30
ICONZ Wired Country customers may experience several drops in service totalling up to 30 minutes within the above timeframe, due to planned maintenance being performed in the area by Compass Communications.
Araneo Connection Loss - 09/12/2009 - 11:30am - 12:00pm - Resolved
We suffering from extremely high packet loss and possible disconnections with Araneo Wireless connections. The issue was within the Araneo network. We are unsure of the cause however a we do not believe it was related to yesterday's problems.
Araneo Connection Issues - 08/12/2009 - 12:00pm - 1:45pm - Resolved
Resolved 8/12/2009 1:45pm
We had an issue between our network and the Araneo network resulting in high latencies and packet loss. The cause was a router which was dropping packets due to high cpu load. Araneo is setting up monitoring on this router so that this does not occur again.
International Connectivity Issues - 07/12/2009 - 3.55am - 4:20am - Resolved
Resolved 7/12/2009 4:20 a.m.
ICONZ customers may have experienced some minor slowness and latency to international hosts at 3:55am on 7/12/2009 due to a brief DDOS attack on one of our customers. Our engineers identified and blackholed the traffic at 4:20am, which resolved the issue
Vector Planned Maintenance – Auckland core link upgrade – 14/12/2009 00:00-00:30
Thursday 3rd December 2009
Maintenance Window: Monday 14th December 2009 12.00am – 12.30am
Expected Outage: Approximate 30 seconds
Services Impacted: ICONZ Vector Fibre Connections in Auckland
Maintenance Summary
We have received notification of planned maintenance being performed by Vector Communications, which will impact ICONZ Vector Fibre Customers in Auckland.
Vector have stated the following regarding the outage:
As part of our continuous improvement program at Vector Communications, we are upgrading our existing network core links to a new 10Gig ring. We have allocated a 30min maintenance window for this change; however the expected maximum actual downtime for your services will be no more than 30 seconds.
If you have any further questions or issues regarding this change, please don’t hesitate to let us know.
Citylink Planned Maintenance – Citylink and WIX – 07/12/2009 00:00-06:00
3rd December 2009
Maintenance Window: 07/12/2009 12.00am – 6.00am
Expected Outage: A few seconds during the above window
Services Impacted: Citylink fibre connections, ICONZ WIX connectivity
Maintenance Summary
We have received notice from Citylink of some planned maintenance occurring at their Lambton Quay facility in Wellington on Monday the 7th of December 2009.
The work will take place between 12.00am and 6.00am, with an expected outage of a few seconds during this time, affecting ICONZ Citylink fibre customers, and connectivity to ICONZ WIX.
If you have any further questions or concerns, please don’t hesitate to let us know.
Recent Network Issues
Wired Country Outage - Papakura Area - 30/11/2009 12:18 p.m. - 1:30 p.m
Wired Country customers would have experienced some issues with connectivity this afternoon at the times listed above.
Wired country or Compass communications engineers attended the site of the basestation and repaired the fault at around 1.20 p.m, the engineers had the basestation back up and running for customer use at 1.30 p.m.
Telecom and Telstra peering - 30/11/2009 12.20pm - 2.50pm
We experienced a spike in traffic to our Telecom and Telstra peering links which caused slow speeds between ICONZ and Telecom/Telstra hosts at the time above.
Our engineers identified the source of the traffic and blocked it at approximately 2.50pm.
Incident Report (Part 2) – Packet Loss at Airedale Street – 23/11/2009 - 24/11/2009
Incident Date/Time:
23/11/09 16:35 - 24/11/09 09:48
Incident Duration: Intermittent periods of packet loss at the times listed above.
Impact: Intermittent spikes in packet loss to all traffic passing through ICONZ Airedale Street routers. This would primarily have impacted ICONZ Colocation customers, but will also have caused issues for some connectivity customers.
Incident Summary:
ICONZ experienced heavy packet loss at our primary Airedale Street Catalyst routers (which caused the service issues above) at the times listed above.
We have now identified that the root cause of these issues was a compromised virtual server on our VERSA platform, which pushed close to 900Mbps of FTP data out to international locations in short bursts at the times listed. These bursts caused CPU load on the aforementioned routers to spike to 100%, resulting in packet loss as outlined above.
In response to these issues, ICONZ has restricted bandwidth availability for the affected server cluster to 100Mbps in an effort to mitigate the impact of a server compromise in future.
As always, if you have any further questions or concerns regarding the above, please don’t hesitate to let us know.
Kind Regards,
ICONZ Technical Support
0800 THE NET (0800 843 638)
help@iconz.net
Incident Report (Part 1): Packet Loss at Airedale Street 23/11/2009 - 24/11/2009
Incident Date/Time: Monday 23rd November 2009, 4.35pm-9.30pm & Tuesday 24th November 6.45am - 10.00am
Incident Duration: Intermittent periods during the above windows
Impact: Intermittent spikes in packet loss to all traffic passing through ICONZ Airedale Street routers. This would primarily have impacted ICONZ Colocation customers, but will also have caused issues for some connectivity customers.
Incident Summary:
ICONZ experienced intermittent network issues today and yesterday, which were caused by significant spikes in load on our Airedale Street routers. Our engineers have resolved the immediate customer impacts of this, and have been working hard to isolate the root cause of the problem.
We believe we have discovered the source of the issue, and will be sending a full Incident Report tomorrow once we have completed our investigations and have a robust plan for action.
As always, if you have any further questions or concerns regarding the above, please don't hesitate to let us know.
--
International Connectivity Issues - 24/11/2009 5pm-5.50pm
We experienced a small window of international packet loss due to a spike in international load caused by a customers server which was compromised. Our engineers identified this issue promptly and service was restored shortly thereafter. This occured from 5:10PM until 5:15PM. This issue is now resolved.
UPDATE: We had a recurrence of the packet loss for approximately 30 seconds at 5:50PM. The machine responsible has now been removed from our network.
--
UPDATE - Intermittent Aucklahnd Network Issues - 24/11/2009 8.45am-9.45am
We have been experiencing intermittent issues with connectivity at ICONZ Airedale Street, causing 20-50% packet loss for all colocation and VERSA services, and for some connectivity customers for around 1-2 minutes at a time.
The first outage occurred at 8.35am, and there have been several following it, occuring at approximately 10 minute intervals. Our engineers have made a change, which appears to have stabilized the issue for the moment. We are keeping the problem under monitoring though, and will post further updates when we have confirmed the problem is resolved.
--
International Connectivity Issues - 22/11/2009 - 2.30pm - Resolved
Resolved on - 23/11/2009 9:07:48 a.m.
ICONZ ADSL customers may have experienced international connectivity issues from approximately 2.30pm on 22/11/2009, until 9.07am when our engineers resolved the issue.