News & Information
Today between 2:30 PM and 3:45 PM EST some clients experienced a network connectivity outage affecting access to client services. Our infrastructure team was immediately aware of the issue and escalated troubleshooting efforts, successfully resolving the outage within the timeframe noted.
Current Status: ✅ Resolved
All affected client services are now fully accessible. If you are still experiencing any connectivity issues, please contact our support team for assistance.
We apologize for any inconvenience this may have caused and thank you for your patience.
UPDATE:
Our upstream providers have updated their routing tables. We are looking into adding additional redundancy on our end to help mitigate this possibility in the future.
One of our upstream cloud providers currently is experiencing routing issues, affecting some servers in the WebPal Cloud.
We are in active communication with the provider's technical support.
At the same time we are switching clients to our redundant alternative upstream.
We will update with expected resolution time shortly.
Dear Clients,
We would like to follow up regarding the network issue that occurred earlier today. At approximately 11:30 AM (ET) on June 9th, a network disruption was identified that may have impacted some users' access to our services.
Our technical team immediately began investigating the issue and worked diligently to implement a solution. We are pleased to report that the issue was fully resolved by 1:30 PM (ET), and all systems have since returned to normal operation.
If you are still experiencing any difficulties, please don’t hesitate to contact our support team at 1-855-200-0320 for assistance.
We appreciate your patience and understanding throughout this incident.
Sincerely,
The WebPal Cloud/Palomino Support Team
Affecting Other - Upstream Network
Problem and Remediation
At 10:03pm, we observed a network outage of Cogent Co - which affects several of our servers located in Hamilton and their connectivity to the public internet.
Root cause
All servers are running as normal and data storage is not affected. Connectivity to certain subnets of the public internet is not operational. The root cause is a loss of connection by an upstream provider (Cogent Co). We are currently in communication with the provider to resolve this issue as soon as possible and identify the root cause on their end.
https://downdetector.ca/status/cogent/
Updates
Cause & Resolution
The root was a power outage to a remote network switch, which had to be resolved by field technicians.
This was accomplished by 3:16am EST.
All our monitors are showing connectivity and systems are back to normal now.
Affecting Other - Upstream Network
At 3:20am EDT, one of our upstream network providers, COGECO, experienced a severe network outage, as also reported by third parties.
This caused some of our Hamilton IP ranges to be temporarily unreachable, until switchover to our redundant provider COGENT completed.
By around 4:30am EDT, all IPs were fully reachable worldwide, as tested with our thirdparty monitoring services.
At the moment, all our services are fully operational.
The cause of the COGECO issue is still unknown, but we will continue to monitor with our upstream provider.
For more information, check https://downdetector.ca/status/cogeco/
Affecting Other - WAN Network Routing
On Tuesday Jan 23 afternoon at 12:01 EST our on-call support team and network operations was informed of a network outage by our monitoring system.
As a result of the outage, incoming traffic on certain IPs in the 69.90 sub-range was not routed to the correct cloud servers.
Several services were affected until our team resolved the issue and restored connectivity at 14:46 EST.
Third party personnel at our Toronto data centre were involved briefly during this process.
The resolution involved a forced restart of the affected router and restoration of network settings.
During this time, our support team was updating affected clients on progress via support tickets.
Affecting System - Storage Network
Problem and Remediation
At 08:15, we observed a switch reboot within our cloud storage network, which caused a number of cloud servers to report storage failures. This was reported by our internal and third-party monitoring services. Our on-site team noticed the issue immediately and initiated remediation and server reboots over the following 90 minutes.
Root cause
Further investigation showed that this issue was caused by a power interruption within the building control, combined with a local redundant power supply failure due to overload.
Preventative Action
Replacement of faulty local UPS and provisioning of additional UPS redundancy. Coordination with building control on power switch overs.
