Postmortem report of incident at October 31st, 2019

At 17:47 (Norwegian time) on October 31st, 2019 our hosting provider initiated a disaster test in their datacenter in Halden, Norway. The test was meant to simulate a power failure in the datacenter and engage the diesel generators for backup power. Unfortunately, none of their customers was warned ahead of this test, and the test was done outside of scheduled service windows.

The test went horribly wrong, and power for the whole datacenter was not restored as expected. This again leading to all our products and services went down.

We got immediate notifications of the problem through our monitoring systems, which runs in another datacenter, and started working with the hosting provider to restore services. We also did status updates for customers regularly during the incident.

Unfortunately, bringing up a datacenter to full speed after such an incident takes time, and our services were not fully operational until 22:45 Norwegian time. There was no loss of data due to the incident.

Scheduled integrations that were set to run between our systems and customer’s systems were not executed between 17:47 and 22:45. If you have custom integrations that were affected by the outage, please contact our Customer Care, and they will provide help.

We will take actions with the hosting provider to make sure such test are never run outside of scheduled service windows.

We apologize to all our customers affected by the outage. This should never have happened, and we will do our very best, together with our subcontractors, to make sure it does not happen again.

‌

Best regards

Flemming Ottosen

Technology Director

Infotjenester AS