August 2020 Scheduled Maintenance
Scheduled Maintenance Report for Progress MOVEit Cloud
Postmortem

Dear Customer,
 
MOVEit Cloud experienced an issue during our planned maintenance window on Sunday 08.16.2020 that resulted in an outage.  This outage impacted all of our MOVEit Cloud North America customers.
 
The outage started at August 16, 2020 at 18:00 EDT and was resolved August 17, 2020 at 13:00 EDT.  We sincerely apologize for this disruption in service.

Cause:  
During the scheduled maintenance on August 16, 2020, the CAPS Cloud Operations team was performing a backup of a NetApp server, to protect against any data loss as we were preparing for a new high availability (HA) deployment.  While preparing for the data backup, we encountered an error that our response process did not address. 
 
The MOVEit Cloud team worked with Microsoft and NetApp engineers to remediate the problem, while ensuring that no data was lost.

Postmortem:  
During our internal root cause analysis (RCA) and postmortem, we have identified next steps and process improvements that we wanted to share with you:

  • We are working closely with NetApp on backup procedures and the deployment of a new HA configuration in MOVEit Cloud as quickly and safely as possible.
  • We are reviewing all aspects of our current infrastructure to test the HA and failover capabilities that we have in place.
  • Additional training has been instituted to ensure all of our engineers are strictly following the documented change control processes.
  • We are looking at additional ways to improve our internal & external communications.
     
    We want to stress that your uptime is of the utmost importance to us and take prompt action to notify you of potential issues. We sincerely apologize for any inconvenience this situation may have caused you.  
     
    Thank you,
     The Progress MOVEit Team
Posted Aug 20, 2020 - 08:00 CDT
Completed
Marking this incident as closed.
Posted Aug 17, 2020 - 19:22 CDT
Verifying
Dear MOVEit Cloud customers,

The MOVEit Cloud service was restored a little after noon and all testing looks good.

We will continue to monitor MOVEit Cloud for the next few hours before we close out this incident.
Posted Aug 17, 2020 - 12:59 CDT
Update
Dear MOVEit Cloud customers,

We've made significant progress on restoring service. The good news is that we are currently testing everything on the test server and we will be turning production servers back on shortly for some testing.

We will send an update once testing is complete.
Posted Aug 17, 2020 - 11:40 CDT
Update
Dear MOVEit Cloud customers,

We wanted to post a quick update that we're still continuing to make progress towards getting the service back online. We've been tirelessly working with our Cloud service vendor on this since yesterday and we will continue to work on this and provide you with updates as quickly as possible.
Posted Aug 17, 2020 - 09:38 CDT
Update
Dear MOVEit Cloud customers,

We are still continuing to make progress towards getting the service back online and will continue to provide updates as soon as possible.

We understand the impact that this has and we sincerely apologize for the inconveniences and issues that this is causing.

We will continue to post updates on the status page as they become available this morning. Please note that MOVEit Support will only have the same updates and ETA that we've posted here.
Posted Aug 17, 2020 - 07:11 CDT
Update
Dear MOVEit Cloud customers,

We are continuing to make progress towards getting the service back online and will provide an update as soon as possible.
Posted Aug 17, 2020 - 05:46 CDT
Update
Dear MOVEit Cloud customers,

We are continuing to make progress towards getting the service back online and will provide an update in another hour.
Posted Aug 17, 2020 - 03:53 CDT
Update
Dear MOVEit Cloud customers,

We are still very actively working with Microsoft and other teams on the MOVEit Cloud outage in North America. We unfortunately don't have an ETA currently to share on when this will be back online but we will continue to send out updates via the status page over the next hour or two.
Posted Aug 17, 2020 - 01:07 CDT
Update
We wanted to send out a quick update that we are still very actively working with Microsoft on this and unfortunately we don't have an ETA just yet on when the service will be back online but we will continue to work on this and send out updates via the status page until this is resolved.
Posted Aug 16, 2020 - 19:32 CDT
Update
"All North America MOVEit Cloud clusters are currently offline due to a technical issue encountered during the scheduled maintenance. We are actively working with Microsoft to diagnose and resolve this issue. We will provide additional updates as more information becomes available via the status page."
Posted Aug 16, 2020 - 18:00 CDT
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Aug 16, 2020 - 15:00 CDT
Scheduled
We will be patching all of the MOVEit Cloud servers and expect there to be minimal downtime for this maintenance window while servers are patched and rebooted.
Posted Dec 02, 2019 - 09:19 CST
This scheduled maintenance affected: North America - Cluster 1, North America - Cluster 2, and Europe - Cluster 1.
← Current Status Powered by Atlassian Statuspage