ICONZ Network Status
For the latest network status and for information on upcoming promotions and events, follow us on Twitter at http://twitter.com/ICONZGroupNZ
Current Network Status:
There are no known issues affecting our network.
Recent issues / Maintenance:
ICONZ MAINTENANCE - VERSA MAINTENANCE - 16/12/2011 7:00 AM - 9:00AM
Iconz would like to inform our customers that the scheduled maintenance for our VERSA platform was extended. This maintenance unfortunately meant that the Virtual Infrastructure Client was not be able to be accessed. Although the infrastructure client was unavailable all platforms remained running without issue and remote connections to virtual machines operated correctly.
If you have any further questions or issues regarding this incident, please do not hesitate to let us know via contacting our Corporate Support Team.
Kind Regards,
ICONZ Corporate Support
ICONZ Planned Maintenance – VERSA S2 Hosted Virtual Machine Outage – 03/12/2011 12:00 – 05:00
Saturday 3rd December 2011
Maintenance Date/Time: 3/12/2011 12:00am - 5:00am
Expected Impact: One power off and power on operation per virtual machine resulting in an outage of approximately five minutes.
Maintenance Summary:
As part of our ongoing commitment to improving service experience for our customers across all of our platforms, ICONZ will be performing planned maintenance on our VERSA S2 platform during the above maintenance window.
During this outage window, customer virtual machines will be powered off and powered on once. This will result in an outage of approximately five minutes per virtual machine.
If you do have any non-BAU activity scheduled during this window, please let us know and we will consider the best course of action. If you have any other questions or concerns regarding the above, please don’t hesitate to let us know.
Kind Regards,
ICONZ Corporate Support
CityLink Wellington outage 24/11/2011 12:00 AM - 2:00 AM
We are currently experiencing an outage on our Wellington CityLink network.
12:45am Update: Tentative ETA of 2:00am.
Citylink resolved the issue and restored connectivity by 2:00AM.
Issue Resolved
Planned CRM and Usage portal outage 23/11/2011 - 10:00pm - 12:00am
Our CRM and Usage portal will be unavailable on the 23/11/2011 for a two hour period between 10:00pm and 12:00am. Customers and staff will be unable to access our usage portal. Our staff will not be able to access customers information including customer product information during this time. We apologise for any inconvenience caused.
Mail Server Connectivity Loss 22/11/2011 - 8:30pm - 9:30pm
Between the hours of 8:30pm and 9:30pm we experienced connectivity issues to our POP mail servers. This issue has now been resolved and we apologise for any inconvenience this caused.
Packet Loss 21/11/2011 - 1:50pm - 2:00pm
At approximately 1:50pm we experienced a brief period of packet loss. Engineers have resolved the cause of the issue, we apologise for any inconvenience caused.
Vector Fibre Outage 18/11/2011 - 9:06 am - 10:05am
At 9:06am a vector fibre link between Auckland and Wellington was cut, resulting in loss of service over this link. Customers may have noticed some congestion on our connectivity services while traffic takes a redundant path during this time period. We apologise for any inconvenience this may have caused, and ask you to call us if you have any questions or problems.
Vector resolved the issue and our service returned at 10:05am
Vector Fibre Outage 10/11/2011
As of 2:25pm A vector fibre link between Auckland and Wellington has resulted in loss of service. There is no customer impact at present. We are currently waiting for further information from Vector
VERSA Backup Suite outage 18/10/2011
At 6:30 p.m. on the 18/10/2011 a server hardware failure has resulted in an interruption of service for VERSA Backup Suite customers. No restores are available to customers at this time.
ICONZ Engineers are currently working to resolve the issue anticipate restoring service within the next few hours. Backup requests are currently on hold until this issue is resolved.
Update as of 21/10/2011: 11am - Our engineers have resolved the hardware issue, and backups are successfully running. This outage is now resolved. Incident report to follow shortly
Wired Country Outage - Hamilton Area - 12/10/2011
Compass communications have advised of an outage to the Wired Country service in Hamilton, due to a power fault. Current ETA is around 12-1am tonight.
Telecom Outage 03/10/2011
10:50am - 12:13PM - Telecom had an outage affecting around 10,000 ADSL broadband users in Auckland due to an equipment failure in the Remuera exchange. The outage affected users in other suburbs; we had reports from users affected from Mission Bay through to Mt Wellington.
ICONZ Unplanned Maintenance 30/09/2011
1:50pm - We've had a small 5 minute outage to our mail system this afternoon between 1:40pm and 1:45pm. Our Engineers are currently investigating the cause and we will post an update when we have more information.
ICONZ Unplanned Maintenance 15/9/2011
15/9/2011 10:15AM - Some users on the ICONZ mail system may not have received Email messages this morning. This issue has been resolved however queued messages may take some time to arrive while the mail queues clear. We apologise for any inconvenience caused.
ICONZ Unplanned Maintenance 8/9/2011 - 10:25am - 10:39am
8/9/2011 10:39am - The ICONZ mail server was greylisting messages. This was an issue for 15minutes from 10:25am and has since been resolved. No mail was lost during this outage.
ICONZ Planned Maintenance – Email – 08/09/2011 – 7:30am
8th September 2011
Maintenance Date/Time: 08/09/2011 :00am – 1:30am (approx.)
Expected Impact: Loss of email services for ICONZ POP3 customers for 10-15 minutes
Maintenance Summary:
There will be an outage of 10-15 minutes to POP3 and Webmail access while we perform routine maintenance. If you experience any issues with email outside of this timeframe please contact our helpdesk on 0800 843 638. We apologise for any inconvenience caused.
ICONZ Planned Maintenance – Kordia – 02/09/2011 – 01:00am - 6:00am
Impact Expected: 2 hours in total
Expected Impact: All Kordia Extend services
Maintenance Summary: Kordia are replacing their core routers.
Network Outage - 15th August 2011
Incident Date/Time: 13h33 - 13h53
Impact: HSNS customers
Incident Summary: Telecom are currently experiencing a network outage affecting all HSNS customers. We will have further detail from Telecom by 13h50
Citylink Connectivity Outage - 28th July 2011
Incident Date/Time: Thursday, July 28, 2011 11:30 AM - 12:00 PM
Impact: Citylink and Araneo customers
Incident Summary:
Citylink experienced a Major Network Outage in the Auckland Peering Exchange at 11:30am. This affected all Auckland Citylink connectivity as well as Araneo connectivity.
Citylink engineers resolved the issue at 12:00pm and our engineers confirmed our connectivity had returned.
We are expecting an Incident Report from Citylink regarding the outage, and will send a copy to the affected customers once we have received it.
ICONZ Informer – Hazard Notification due to UPS Maintenance
15th July 2011
What is happening?
As part of routine preventative maintenance, and in the wake of successful maintenance carried out on one of our UPS’s on Thursday 14th July, ICONZ contractors will be performing work on a second UPS in our Level 1 Datacentre between ~1140 and ~1500hrs on Thursday 21 July. This is the second of two UPS’s that requires servicing.
During this time the ICONZ Level 1 Datacentre will not have UPS protection.
Why are ICONZ doing this?
In order to ensure their long-term reliability, the UPS’s must periodically be taken out of service for maintenance.
How does this affect me?
In real terms, this should not cause any noticeable affect to ICONZ services, or ICONZ customers.
In order to ensure uninterrupted operations, the ICONZ Datacentres will be operating on Generator for the duration of the maintenance.
Our UPS and Generator facilities are regularly tested and serviced to ensure that they’re ready to come online whenever needed, however due to the small increase in risk posed by a period without UPS protection, we have opted to provide advance notice to our customers. We are taking every reasonable precaution given the nature of the work, and are not expecting any operational impact.
Questions
If you would like to ask us any further questions regarding this change, or you have any concerns over impact to your business, please don’t hesitate to contact our Corporate Support Team to discuss this further. Regards,
ICONZ Corporate Support
ICONZ - FTP Scratch Space
12th July 2011 - 9:00am - 9:40am
We experienced an outage with our FTP Scratch Space service. It was escalated to our Managed Hosting team at 9:00 am. They resolved the situation and continued the service by 9:40am.
There are no further expected issues at this time.
ICONZ Informer – Hazard Notification due to UPS Maintenance
11th July 2011
What is happening?
As part of routine preventative maintenance, ICONZ contractors will be performing work on one of our Datacentre UPS units between ~1140 and ~1500hrs on Thursday 14 July.
During this time the ICONZ Level 1 Datacentre will not have UPS protection.
Why are ICONZ doing this?
In order to ensure their long-term reliability, the UPS’s must periodically be taken out of service for maintenance.
How does this affect me?
In real terms, this should not cause any noticeable affect to ICONZ services, or ICONZ customers.
In order to ensure uninterrupted operations, the ICONZ Datacentres will be operating on Generator for the duration of the maintenance.
Our UPS and Generator facilities are regularly tested and serviced to ensure that they’re ready to come online whenever needed, however due to the small increase in risk posed by a period without UPS protection, we have opted to provide advance notice to our customers. We are taking every reasonable precaution given the nature of the work, and are not expecting any operational impact.
Questions
If you would like to ask us any further questions regarding this change, or you have any concerns over impact to your business, please don’t hesitate to contact our technical support team on 0800843638
6 July Vector Fiber connectivity issue.
Vector Fiber and International Connectivity Issues. This may affect customers with Vector Fiber connections as well as all customers’ international transit.
Issue resolved, please contact us if you are still experiencing connectivity issues.
Unplanned work: An emergency reboot of our firewall needs to take place. There will be a 5 minute outage that will affect all ICONZ customers.
Data Scheduled: 1st July 2011 - 11h05
Planned work: Planned maintenance is taking place on ICONZ webservers
Data Scheduled: 8th June 2011 - 22h00
Impact: Customers will lose access to their website for 10 minutes.
Phishing email asking for username and password being sent.
Incident Date/Time: Saturday, June 04, 2011 12:05 AM
Impact: Email customers
Incident Summary:
Email being sent from <support@internet.co.nz> requesting users to respond with username and password details.
We have prevented this message from being sent again and removed it any remaining copies from our servers.
If you have any further questions or issues regarding this incident, please don’t hesitate to let us know.
International Connectivity – Degraded Service – Thursday 19 May 21:38:20 – 21:44:25
Incident Date/Time: Thursday 19 May 21:38 – 21:44
Incident Duration: Approximately 6 minutes
Impact: Approximately 80% packet loss to international hosts from the ICONZ network
Incident Summary:
We appear to have experienced a brief packet loss issue at approximately 21:38.
Our engineers have isolated and resolved the fault as at 21:44 May 19th. An Incident report in regards to this issue shall follow.
If you have any further questions or issues regarding this incident, please don’t hesitate to let us know.
ICONZ Planned Maintenance – ADSL Connectivity – 03/05/2011 – 1:00am
26th April 2011
Maintenance Date/Time: 03/05/2011 1:00am – 1:30am (approx.)
Expected Impact: Loss of connectivity for ICONZ ADSL customers for 5-30 minutes
Maintenance Summary:
Telecom has advised us of planned maintenance affecting our ADSL customer base. Maintenance is scheduled for 1:00am on Tuesday the 3rd of May 2011, and an outage of up to 5 minutes is expected at this time.
Once the maintenance is completed, customer modem/routers will need to re-authenticate to restore connectivity. This will occur automatically, but may it may take up to 30 minutes or more for all customers to reconnect.
For customers still offline after 30 minutes, we recommend rebooting your router/modem. If you continue to experience problems, please don’t hesitate to contact us.
Wired Country Outage 5/05/2011 - 10:00AM - 12:00PM Resolved
Impact: Loss of all Wired Country services
Outage Summary: At approximately 10am, Wired Country experienced an outage on their main link between the Sky Tower and Pukekohe which has affected the entire network.
Estimated Time to Resolution: No ETR at present, but this has been escalated as a Priority 1 event. Wired Country will advise of any further developments.
Citylink outage 01/5/2011 - 10:50pm - 12:00am Resolved
Impact: Loss of connectivity to Citylink customers in Wellington
Outage Summary: The outage began approx 10:50pm on Sunday 01/5/11. Citylink are aware of the outage, have identified a hardware fault and they are currently implementing repairs.
Estimated Time to Resolution: No ETR at present. Citylink will advise.
Vocus Maintenance 29/04/2011 04h00 - 05h00
During the above window, Vocus engineers will be performing an emergency card swap and software upgrade of a border router in Auckland. Customers may experience an outage of up to 10 minutes while the maintenance takes place.
Vocus Maintenance 19/04/2011 04h00 - 0600
During the above window, Vocus Engineers will be performing an emergency software upgrade of a border router located at Palo Alto, CA. This will result in PAIX peering being unavailable for up to 10 minutes whilst the router reloads. Peering that would usually traverse PAIX, will route via other peering locations of via transit providers.
ICONZ Informer – Hazard Notification due to Generator Testing
28th March 2011
What is happening?
As a test of our redundant capabilities, ICONZ intend to carry out a full operational load test on our Generator plant between 0600 and 0700 on Wednesday, March 30th. During this time the Generators will be started and then in a coordinated fashion, operational load for ICONZ facilities will be transferred to the Generators for an extended run test.
Why are ICONZ doing this?
At the ICONZ Datacentres we provide full UPS and Generator backup capabilities to support the operations of ICONZ customers in the event of mains power failure. As part of our regular maintenance, our Generator plant is regularly spun up and tested to ensure its availability in case of need, though we do not frequently transfer operational load to the Generators – the UPS’s alone provide sufficient cover for the majority of mains-power interruptions that we experience.
It is however good practice to periodically test the Generators at operational loading levels, to ensure that they are prepared to carry our Datacentres and key communication circuits should it be required for an extended period.
ICONZ have been advised of upcoming maintenance work which will see us without mains power for several hours (a further informer will be released ahead of time with more detail). This load test will establish our preparedness for this event and allow sufficient time to address any problems that may be identified.
How does this affect me?
In real terms, this should not cause any noticeable affect to ICONZ services, or ICONZ customers. Our UPS and Generator facilities are regularly tested and serviced to ensure that they’re ready to come online whenever needed. The cutover between Mains and Generator is not entirely without risk, so as planned works we have opted to provide a hazard notification to our customers.
Should any problems develop we would immediately transfer service back to Mains supply. The UPS’s provide continuous power while load is moved from source to source.
Questions
If you would like to ask us any further questions regarding this change, or you have any concerns over impact to your business, please don’t hesitate to contact our Corporate Support Team to discuss this further.
Regards,
ICONZ Corporate Support
Mail Delays – 25/3/2011 – 10AM – 4:30PM - Resolved
Services Affected: Email
Impact: Email delays
Description: We are currently experiencing email delays. ICONZ engineers are currently working on the issue.
UPDATE - 11:30AM - Our engineers have identified and resolved the cause of the delays. Delays will continue until the mail servers have had time to de-queue.
ICONZ apologises for any inconvenience.
Iconz Linux Hosting issues- 20/03/2011 09:43 - 20/03/2011 12:43 -Resolved
Iconz would like to advise our customers of a current outage affecting customer's hosted on our Voltron hosting server.
Our engineers have been alerted and are currently investigating the issue.
We apologise for any inconvenience this outage may cause, and invite you to contact us if you have any further queries or concerns.
Kind Regards,
ICONZ Corporate Support
ICONZ Mail Delay Issues - 24/02/2011 12:05 - 24/02/2011 20:45 - RESOLVED
Iconz would like to advise our customers that we are currently experiencing mail delays with emails being sent from the Iconz mail cluster
This issue will affect all customers using the Iconz mail system to send or receive emails.
Our engineers have been alerted to the issue and are investigating the fault.
We apologize for any inconvenience this causes.
Wired Country North Island Network Outage - 22/02/2011 09:43 - 01/03/2011 12:00 - RESOLVED
Iconz would like to advise our customers of a current network outage affecting Wired Country users.
Wired Country have isolated the fault and customer's should now be coming back online. Wired Country are looking to restore services with one customer at a time, please do not reboot the NIA.
As always please feel free to contact our helpdesk if you have any questions regarding this outage or require further assistance.
Kind Regards,
ICONZ Corporate Support
0800 11 11 48
Telecom Auckland Broadband outage- 20/02/2011 10:45 - 20/02/2011 17:33 - RESOLVED
Iconz would like to advise our customers of a current network outage affecting Auckland Business and Residential customers loss of service.
This outage will be affecting customers between Mayoral Drive and Papatoetoe network nodes. Telecom is currently investigating the issue and we shall update our network status as more information comes to light.
As always please feel free to contact our helpdesk if you have any questions regarding this outage or require further assistance.
Kind Regards,
ICONZ Corporate Support
0800 11 11 48
ICONZ Planned Maintenance – Power Upgrades at Skytower – 17 February 2011 0600-0700hrs
10 February 2011
Maintenance Date/Time: Thursday 17 February 2011 6am-7am
Expected Impact: As described below:
From 0600 to 0610:
Loss of Service for ICONZ Customers connected via
· Wired Country
· ICONZ Extend Wireless (ICONZ Kordia services)
Possible degradation of service for all ICONZ customers with respects to connectivity to:
· Telecom New Zealand Domestic Network [traffic will reroute via alternate path]
From 0600 to 0700:
HAZARD ONLY - ICONZ Primary International Transit, Auckland Peering Exchange connectivity, Wired Country and ICONZ Extend Wireless. No outage expected.
Maintenance Summary:
As part of our ongoing commitment to service stability, ICONZ engineers will be carrying out planned maintenance work on the power supply at our Skytower Point of Presence on the morning of Thursday 17 February 2011 ICONZ.
ICONZ customers connected via Wired Country or Extend Wireless services will experience a service outage of 10 minutes between 6am and 6.10am.
All ICONZ customers may also experience degradation of service to hosts on the Telecom NZ domestic network.
This maintenance will also affect ICONZ Primary International Transit, Auckland Peering Exchange connectivity, Wired Country and ICONZ Extend Wireless, although no outage is expected for these services – this is a hazard notice only.
We apologise for any inconvenience that these scheduled changes, which should result in improved service availability and reliability. If you have any concerns, we invite you to contact our Corporate Support team using the details below.
Kind Regards,
ICONZ Corporate Support
0800 11 11 48
Vocus Maintenance - 22/02/2011 03:00 - 22/02/2011 06:00
During the above window, the SCCN will be performing maintenance in Takapuna NZ to replace some multi fibre cables.
1) Replace fibres on NZ-US circuit - this will result in a complete outage of this circuit for up to 5 minutes. All NZ-US traffic for this period will re-route via Australia.
2) Switch hit and back to main with loss of protection for up to 20 minutes on the NZ-US, there should be no impact to customers here.
3) Replace fibres on AU-NZ circuit - this will result in a complete outage of this circuit for up to 5 minutes. All AU-NZ traffic for this period will re-route via the USA.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Citylink Scheduled Outage - 20/02/2011 20:00 - 20/02/2011 22:00
CityLink would like to advise of two items of maintenance work during the standard maintenance window. This outage will only affect customers in Majestic Tower.
Expected Outage Duration: Less than 1 minute per customer.
Reason: Upgrade network switch to current software level.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ICONZ Outbound mail issues - 28/01/2011 12:55 - 28/01/2011 13:27 - RESOLVED
Iconz would like to advise our customers that we are currently experiencing issues with our international connectivity.
This issue has occurred since 12:55PM and is an ongoing issue.
This issue will most probably affect all international usage and our Kordai and Araneo customers.
Our engineers have been alerted to the issue and are investigating the fault.
We apologize for any inconvenience this causes.
This issue has been resolved and an incident report will follow
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ICONZ Outbound mail issues - 27/01/2011 13:53 - 27/01/2011 -RESOLVED
We are currently experiencing delays in our outbound mail. Our engineers have been alerted and are currently investigating the issue.
We apologise for any inconvenience this outage may cause, and invite you to contact us if you have any further queries or concerns.
This issue has been resolved
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ICONZ Usage portal delays - 31/12/2010 - 06/01/2011 - RESOLVED
We are currently experiencing delays to our usage portal. The delays will not affect the billing process, but will mean you can't see your latest usage.
We do not have an ETA for resolution at this stage, but are working hard to get it fixed.
Citylink Scheduled Outage – 6/12/2010 12:00am – 6:00am - COMPLETED
6th December 2010
Maintenance Window: 6/12/2010 12.00am – 6.00am
Expected Outage: There will be a variable outage duration on the network from one minute up to an hour depending on service and connection type.
Services Impacted: Citylink Customers located in Wellington
Maintenance Summary
Citylink are currently performing planned maintenance on the Wellington switch, scheduled from 12:00am to 6:00am on December 6, 2010. This will result in a variable outage duration from one minute to an hour.
ICONZ apologises for any inconvenience.
ICONZ Citylink Fibre Issues 3/12/2010 16:00 - 16:42
Services Impacted: Citylink Customers located in Wellington
Incident Summary:
Citylink have advised us that of a fault in Wellington affecting Iconz Citylink Fibre Customers. We have not received an ETA on resolution as yet, but will post further updates as they come to light
This issue has been resolved
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Iconz Dial up connectivity and intermittent mail issue – Friday 03 December 6:30AM - 7:49AM
Iconz Dial-up customer's will be experiencing authentication issues. Customers will also be experiencing mail issues currently, as a result of this the Iconz webmail system appears to be experiencing issues as well. Our engineers have been notified and are investating the issue.
We apologise for any inconvenience this outage may cause, and invite you to contact us if you have any further queries or concerns.
This Issue has been resolved
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ICONZ Scheduled Outage / Emergency configuration change – Telecom NZ Peering – Thursday 9 December 00:30
Services Impacted: ICONZ Customers can expect a momentary interruption of communication with Telecom NZ networks.
Maintenance Summary
We wish to advise customers of planned maintenance on our Telecom NZ Peering link. This will take no more than a couple of minutes, and traffic should be rerouted automatically, though you may notice a brief period of increased latency preceeded by a very brief loss of connectivity between ICONZ and Telecom NZ’s networks.
The work is to be carried out by Telecom New Zealand. Impact to our operations is expected to be minimal-to-nil.
We apologise for any inconvenience caused by this configuration change.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Changes to the ICONZ Usage Portal - 4/12/2010
What is happening?
Soon you will notice some small changes we are introducing into our backend systems, in particular the portal through which you log on to check monthly usage. The changes will be relatively minor and mainly affect the way things look and how some information is presented.
Why are we doing this?
As part of our ongoing commitment to providing better service to our customers, we are migrating all of our systems onto a new platform. This will better enable us to improve our service offering and will position us to introduce new and exciting products in the future.
How will this affect you?
The changes are minor, and do not affect the way you log on. Your username and password stay the same, so you do not need to do anything differently to what you do today.
The URL has changed. To reach the new secure portal directly, click here. If you have the current URL saved as a shortcut, you will be redirected to the new portal on the 4th of December.
If you do not currently have a username and password to check your usage and would like one, please contact our Customer Care team and we will be happy to set this service for you.
When are we doing this?
We are planning to cut over to our new system on Saturday, 04 December, 2010 at 9am. Please be aware that we will not be available to complete any work other than urgent provisioning on this day.
Any questions?
If you have any questions please do not hesitate to call or email our Customer Care team.
Regards
ICONZ Customer care
0800 THE NET (0800 843 638)
help@iconz.net
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Citylink Scheduled Outage – Wellington Fibre Connectivity – 6/12/2010 12:01am – 2:00am
25th November 2010
Maintenance Window: 6/12/2010 12.01am – 2.00am
Expected Outage: There will be a variable outage duration on the network from
one minute up to an hour depending on service and connection type.
Services Impacted: Citylink Fibre connectivity
Maintenance Summary
As part of an ongoing program to improve the reliability and performance of our network, CityLink staff will be performing maintenance during our standard maintenance window on the Ethernet network as follows:
• Where: Datacom Building, 84 Abel Smith Street and Lambton House, 160
Lambton Quay, Wellington.
• Date: Monday 6th December 2010
• Time: 12:01am - 2:00am
• Impact: There will be a variable outage duration on the network from one minute up to an hour depending on service and connection type.
We apologise for any inconvenience this outage may cause, and invite you to contact us if you have any further questions or concerns.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NZ Unplanned Outage / Emergency Maintenance – Telecom NZ Peering – 24/11/2010 15.26 – 15.37hrs
Services Impacted: ICONZ Customers experienced a 2 minute loss of communication with Telecom NZ networks followed by approx. 9 minutes of degraded service to both Telecom and TelstraClear.
Outage / Maintenance Summary
ICONZ had arranged for an urgent upgrade to our peering circuit with Telecom NZ to meet a customers short-notice request. This was due to take place prior to 1700hrs today; it was indicated to ICONZ that impact to the link would be minimal-to-nil (max duration 1 minute). Due to the short notice this work could not be scheduled for outside of normal business hours as would normally be the case.
It appears that the upgrade was actioned by Telecom at 1526hrs this afternoon, at which time our peering session with Telecom dropped unexpectedly. It took approximately 2 minutes for traffic to completely fail over to TelstraClear. Whilst operating in this fashion customers communicating with both Telecom and TelstraClear may have experienced degraded performance; the Telecom link was restored at 1537 at which time services returned to normal.
The severity of the impact to ICONZ services was unexpected and our engineers are working with Telecom to establish exactly why we were impacted, and to ensure it does not reoccur. We apologise to those customers adversely affected.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
TELECOM PLANNED EVENT NOTIFICATION – 19/11/2010 00:01 – 05:00
Maintenance Date/Time: Friday 19th November 00:01 – 06500
Expected Impact: 10 minutes between 00:01 - 05:00
Advisory Notice Only
Maintenance Summary:
Telecom will be performing an upgrade to the Airedale Street and Federal Street Ethernet Nodes between 00:01-05:00 on Friday, 19 November. There is no expected outage on the ICONZ network as any affected circuits will fail over to our redundant circuits. This upgrade is expected to take 10 minutes.
If you notice any issues or problems, please give our customer care team a call on 0800 843 638
If you have any further questions or issues, please don’t hesitate to let us know.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ICONZ Emergency Maintenance - International Failover – 15/11/2010 11.00am – 11.15am
Maintenance Window: 15/11/2010 11.00am – 11.15am
Services Impacted: International connectivity will fail over to our secondary link (Global Gateway) –
there will be no outage
Maintenance Summary
Due to urgent maintenance required to our primary international gateway, ICONZ will be performing a controlled failover to our secondary international provider (Global Gateway) at approximately 11.00am this morning. We anticipate this maintenance will be complete within 15 minutes, at which time services will fail back to our primary gateway (Vocus).
Customers will notice slightly different routing for international traffic during this period. We do not
anticipate any service outage.
We apologise for the short notice of this change, and invite you to contact our support team if you
have any further questions or issues.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ICONZ Citylink Fibre Issues 13/11/2010 09:00 - 13:00 - Resolved
Services Impacted: Citylink Customers located in Wellington
Incident Summary:
Due to a major power disruption in Wellington, a number of ICONZ Citylink customers may have been without service during the above window. Core services were restored at approximately 13:00 but some customers may have experienced further issues if their premises were impacted by the outage. Citylink advised full service restoration at approximately 9.15am on Monday the 15th of November.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ICONZ - Telecom Connectivity Issues 11/10/2010 12:00 - Resolved
Services Impacted: ICONZ toTCNZ Transit
Incident Summary:
ICONZ Connections and Co-Located servers that are directly communicating with servers and clients on Telecom NZ connections are experiencing on-going packet loss. This is due to our Telecom link being saturated with higher than expected traffic. Our Network Engineers are aware of the issue and are looking in to a long-term fix.
We don't have an ETR at this stage but will post an update as soon as further updates become available.
This issue has been resolved.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ICONZ Vector Fibre and International Connectivity Issues 05/10/2010 00:20 – 02:30
Incident Time: 12.20am – 02.30am (approx)
Incident Duration: 2 hours 10 minutes (approx)
Impact: Loss of international connectivity (30 mins), Loss of service for Vector Fibre customers (2 hours)
Incident Summary:
On the 5th of October, at approximately 12.20am, ICONZ monitoring systems presented alerts for international connectivity and several Vector Fibre services. The issue was immediately escalated to ICONZ network engineers, who began fault diagnostics.
At approximately 12.50am ICONZ engineers were advised of a power-related fault at a major Vector node in Auckland CBD. This caused service issues for several Vector customers, including ICONZ’s primary supplier of international connectivity, Vocus. The impact to Vocus caused ICONZ to lose international service. Additionally, some ICONZ customers who are connected via Vector experienced a loss of connectivity.
At this time, ICONZ engineers migrated international connectivity to our secondary provider (Global Gateway), restoring international connectivity.
Vector technicians resolved their fault at approximately 2.30am, restoring service to Vocus and other Vector Fibre customers. Upon confirmation of service restoration, ICONZ engineers migrated international connectivity back to Vocus.
ICONZ are liaising with both Vocus and Vector, with a view to preventing further issues of a similar nature. In the meantime we apologise for any inconvenience caused by this unscheduled service outage.
It should be noted that customers may have been unable to access various services hosted on other Service Provider’s networks – several other providers were also affected by this outage.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Incident Report – International Connectivity – Degraded Service – Thursday 09 September 08:09 – 08:22
Incident Date/Time: Thursday 9 September 08:09-08:22 (approx)
Incident Duration: Approximately 15 minutes
Impact: Approximately 80% packet loss to international hosts from the ICONZ network
Incident Summary:
At approximately 08:09am on the 9th of September 2010, ICONZ experienced a DOS attack from a single IP address directed at a large number of ICONZ IP addresses. This caused congestion across our primary international uplink, which resulted in high packet loss for traffic across that link.
The source IP address of the attack was quickly identified by our engineers and blocked by our upstream provider at approximately 08:22am, at which time full service was restored.
If you have any further questions or issues regarding this incident, please don’t hesitate to let us know.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Incident Report – Sky Tower Power Failure – 24.08.2010 12:59 – 15:20 31st August 2010
Incident Date/Time: 24.08.2010 12.59pm – 2.30pm
Incident Duration: Approximately 2.5 hours
Services Impacted: International connectivity (approx. 10 mins), ICONZ Extend Wireless (approx. 2 hours), Araneo (approx. 2.5 hours).
Incident Summary:
At approximately 12.59pm on the 24th of August 2010, ICONZ Monitoring Systems presented alerts for ICONZ equipment located at the Auckland Sky Tower, as well as alerts for ICONZ Extend Wireless and Araneo services, and our primary international connectivity gateway via Vocus.
ICONZ engineers were immediately alerted to the problem, and made contact with staff at the Sky Tower, who confirmed that had been an electrical incident at Skycity which had resulted in a Fire Brigade Callout, and required full electrical isolation of the Skytower Datacentre. We were advised that power would be restored as quickly as possible. At this time, ICONZ equipment at Sky Tower appeared to be operating on UPS battery power, although devices belonging to some of our suppliers appeared to be offline.
At approximately 1.05pm, international connectivity was restored when Vocus equipment at the Sky Tower came online running on UPS. ICONZ Extend Wireless and Araneo services still appeared to be offline at this time.
At approximately 1.45pm ICONZ confirmed that the fire at Sky Tower had been suppressed, and that mains power would be restored shortly.
At approximately 2.25pm, mains power was restored to the Sky Tower, and all ICONZ equipment returned to normal. ICONZ Extend Wireless services were also restored at this time, but Araneo service remained offline.
Araneo reported to ICONZ that it appeared that power to their UPS had run down, and thus they required access to the Sky Tower to manually reset their equipment and restore power. Due to safety restrictions, they were unable to gain access until approximately 3.15pm, at which time full service was restored. Araneo have assured us that they are taking steps to mitigate this risk moving forward.
It should be noted that customers may have been unable to access various services hosted on other Service Provider’s networks – several other providers were also affected by this outage.
If you have any further questions or issues regarding this outage, please don’t hesitate to contact us. For a full history of events, and for up-to-date information on future network events, customers are encouraged to visit our Twitter page at http://twitter.com/ICONZGroupNZ.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Incident Report – Auckland-Wellington Network Issues – 17.08.2010 01:40 – 02:40
Incident Date/Time: 17.08.2010 1.40am – 02.40am
Incident Duration: Approximately 1 hour
Impact: Loss of connectivity for ICONZ customers in Wellington
Incident Summary:
On the morning of 17th August 2010, ICONZ Auckland-Wellington inter-metro connectivity supplier (TelstraClear) was performing planned (non-service impacting) maintenance on their IP Core Network.
During the course of this maintenance, at approximately 01:40am TelstraClear experienced an unexpected service outage which caused a loss of connectivity on both our primary and secondary (redundant) links between Auckland and Wellington.
ICONZ engineers were immediately notified, and the problem was escalated to TelstraClear, who confirmed a fault within their network. This fault was resolved at approximately 02:40am, restoring full service for ICONZ customers.
In terms of impact, ICONZ customers in Wellington would have noticed a substantial loss of connectivity. Wellington-bound traffic for all other customers failed over to diverse paths, but users may have noticed marginally increased latency and alternate routing during the incident window.
We have been assured by TelstraClear that they have investigated cause of the outage and will be taking action to prevent a recurrence.
If you have any further questions or issues, please don’t hesitate to let us know.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
TELECOM PLANNED EVENT NOTIFICATION - 20/8/2010 03:00 – 06:00
Maintenance Date/Time: Friday 20th August 03:00 – 06:00
Expected Impact: 10 minutes between 03:00 - 06:00
Services Impacted: All ICONZ ADSL connections will be affected
Maintenance Summary:
Telecom will be performing maintenance to the Mayoral Drive Juniper Node which will cause all ICONZ ADSL connections to be dropped for 10 minutes during the hours of 03:00 - 06:00 on 20/08.
The connection should automatically connect up after the outage, if you notice any problems a reboot of the CPE should be the first step in bringing the connection up. If you are still not connected please give our customer care team a call on 0800 843 638
If you have any further questions or issues, please don’t hesitate to let us know.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Incident Report – Telecom Peering Issues – 05/08/2010 21:37 – 01:00
Incident Date/Time: 05/08/2010 9.37pm – 06/08/2010 1.00am
Incident Duration: 3 hours 23 minutes
Impact: Loss of ICONZ peering link to Telecom (traffic failed over to diverse paths)
Incident Summary:
At approximately 9.37pm on the 5th of August 2010, ICONZ monitoring systems detected an outage to the ICONZ-Telecom peering link. The problem was immediately escalated to ICONZ engineers, and a trouble ticket was raised with Telecom.
ICONZ engineers diagnosed a hardware problem at the Sky Tower, and resolved the issue at approximately 1.00am, restoring full service.
Traffic immediately failed over to diverse paths, but customers may have noticed alternate routing and slightly degraded performance to Telecom hosts during the outage period.
If you have any further questions or issues regarding this incident, please don’t hesitate to let us know.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Incident Report – Server Secure (Shared Firewall) Degraded Service – 29/07/2010 14:51 – 15:01
Incident Date/Time: Thursday 29th July 2.51pm – 3.01pm
Incident Duration: Approximately 10 minutes
Impact: 70%+ packet loss to all customer servers behind the shared firewall
Incident Summary:
At approximately 2.51pm on the 29th of July 2010, ICONZ monitoring systems identified service issues with our shared firewall service (Server Secure).
The problem was immediately escalated to our engineers, who diagnosed a Distributed Denial of Service attack targeted at a specific host behind the firewall resulting in heavy packet loss to all customers hosted behind the cluster.
Since the attack was mostly generated from offshore hosts, ICONZ engineers blocked international traffic to the target IP address. This significantly reduced the load on the Firewall Cluster and returned full service to other hosted customers at approximately 3.01pm, restoring service for affected customers.
As of this morning, our engineers have confirmed that the DDoS attack has subsided and have removed the null-route, restoring full service to the affected customer. We do not anticipate any further service issues at this time, but will be closely monitoring the situation throughout the day.
If you have any further questions or issues regarding this incident, please don’t hesitate to let us know.