iPaper Incident History

1. postmortemSep 8 · 07:10 UTC

   Summary

   Due to an ongoing issue with our VXLAN where the IPsec protocol sometimes desynchronizes causing nodes to loose communication one way, we saw all servers having issues communicating properly with other internal services.

   Impact

   Caused some app-servers to be unable to serve admin, viewer, api and popups

   Cause

   We have an underlying issue with our provider where very small networking glitches can cause the VXLAN IPsec protocol to get out of sync, which causes communications issues between nodes in our cluster.

   Resolution

   While we can manually reset this and recover, which we did in this instance, we are still working with our vendors for a permanent fix.

   To mitigate this, we are adding even more infrastructure and capacity to prevent more downtime.

Reported by iPaper on their status page.

1. postmortemSep 7 · 05:37 UTC

   Summary

   Due to an ongoing issue with our VXLAN where the IPsec protocol sometimes desynchronizes causing nodes to loose communication one way, we saw all servers having issues communicating properly with other internal services.

   Impact

   Caused some app-servers to be unable to serve admin, viewer, api and popups

   Cause

   We have an underlying issue with our provider where very small networking glitches can cause the VXLAN IPsec protocol to get out of sync, which causes communications issues between nodes in our cluster.

   Resolution

   While we can manually reset this and recover, which we did in this instance, we are still working with our vendors for a permanent fix.

   To mitigate this, we are adding even more infrastructure and capacity to prevent more downtime.

Reported by iPaper on their status page.

1. ResolvedSep 5 · 14:49 UTC

   This incident has been resolved.

Reported by iPaper on their status page.

1. postmortemSep 7 · 05:36 UTC

   Summary

   Due to an ongoing issue with our VXLAN where the IPsec protocol sometimes desynchronizes causing nodes to loose communication one way, we saw all servers having issues communicating properly with other internal services.

   Impact

   Caused some app-servers to be unable to serve admin, viewer, api and popups

   Cause

   We have an underlying issue with our provider where very small networking glitches can cause the VXLAN IPsec protocol to get out of sync, which causes communications issues between nodes in our cluster.

   Resolution

   While we can manually reset this and recover, which we did in this instance, we are still working with our vendors for a permanent fix.

   To mitigate this, we are adding even more infrastructure and capacity to prevent more downtime.

Reported by iPaper on their status page.

1. ResolvedSep 4 · 22:39 UTC

   This incident has been resolved.

Reported by iPaper on their status page.

1. postmortemSep 4 · 15:32 UTC

   Summary

   We are seeing some degraded performance across flipbooks and we have identified the issue to be a networking issue, we are working with our sub-providers on a permanent fix

   Impact

   We were seeing some request taking longer than usual causing degraded performance causing some requests to fail with timeouts across all flipbooks products.

   Cause

   A software networking issue caused some nodes to respond slower which cascaded to these requests to start timing out.

   Resolution

   The immediate fix was restarting the networking layer and this caused the requests to now resolve at normal speeds.

   We are working with our providers on a permanent fix.

Reported by iPaper on their status page.

1. ResolvedSep 2 · 16:34 UTC

   This incident has been resolved.

Reported by iPaper on their status page.

1. postmortemSep 4 · 15:30 UTC

   Summary

   Due to an ongoing issue with our VXLAN where the IPsec protocol sometimes desynchronizes causing nodes to loose communication one way, we saw all servers having issues communicating properly with other internal services.

   Impact

   Caused some app-servers to be unable to serve admin, viewer, api and popups

   Cause

   We have an underlying issue with our provider where very small networking glitches can cause the VXLAN IPsec protocol to get out of sync, which causes communications issues between nodes in our cluster.

   Resolution

   While we can manually reset this and recover, which we did in this instance, we are still working with our vendors for a permanent fix.

   To mitigate this, we are adding even more infrastructure and capacity to prevent more downtime.

Reported by iPaper on their status page.

1. ResolvedAug 31 · 09:46 UTC

   Closing as there have been no issues observed since the fix was implemented

Reported by iPaper on their status page.

1. postmortemSep 4 · 15:27 UTC

   Summary

   One of our nodes had a small milliseconds networking outage, which caused the IPsec connection used for our VXLAN between the node and another node to get out of sync. The node which lost connection was running one of our clustered caching instances, so the other instances took over with no issues. Once we manually fixed the node they were able to talk together again, which caused them to now try and sync up, but because they had been running for an extended time apart they failed to properly sync, and the application servers started to fail when using the caches.

   Impact

   Caching was unable to serve request fully and partially depending on which instance was used, which caused us to not load flipbooks viewer and admin.

   Cause

   The caching nodes were out of sync due to a networking issue on the internal VXLAN, which when fixed caused the instances to try and sync data and failed.

   Resolution

   We rolled entirely new caching instances, and updated all applications to use the new instances across all of our applications.

   We are working with our providers to implement a resilience fix to the core IPsec feature so this should not happen in the future. We have added extra monitoring as well to better catch this exact scenario.

   We will also be working on trying to understand why the sync failed, and we will be changing our clustering strategy to avoid the issue entirely.

Reported by iPaper on their status page.

1. ResolvedAug 27 · 08:16 UTC

   The issue has been resolved and we have seen no more errors with the new caching.

Reported by iPaper on their status page.

1. postmortemSep 4 · 15:25 UTC

   Summary

   Branded domains starting with www or that do not have HTTPs enabled were incorrectly redirected to their apex domains, and forcebly from HTTP to HTTPs, which our applications then redirected back to HTTP, causing a redirect loop.

   Impact

   The viewer was unable to serve any www or http branded domain, as it would always redirect away.

   Cause

   This was a misconfiguration on the Load Balancers where it would always redirect www and http.

   Resolution

   We have disabled these redirects, and updated our internal configurations and checks to make sure this cannot happen again.

Reported by iPaper on their status page.

1. ResolvedAug 24 · 11:36 UTC

   The incident has been resolved

Reported by iPaper on their status page.

1. postmortemSep 4 · 15:24 UTC

   Summary

   A bug in our vendors software with IPv6 internal DNS resolution, caused us to be unable to resolve our own as well as branded domains internally.

   Impact

   Some services like flipbooks processing, relies on being able to call internal services to fetch assets for flipbooks, this failed and in turn failed to processes flipbooks that relied on assets like media library items.

   Cause

   Internal DNS resolution for the nodes were unable to resolve domains, which caused enrichment automations to fail.

   Resolution

   The immediate fix was to manually add the DNS entries to each node that needed them, to avoid more failing processing's.

   Our vendor as since released a permanent fix which resolved all issues with internal DNS resolution, and our temporary fix has been removed again.

Reported by iPaper on their status page.

1. ResolvedAug 24 · 06:00 UTC

   We saw some enrichment automation jobs failing in the hours after the migration if they requested assets from internal domains.Due to a vendor bug in internal DNS request handling for IPv6, we were unable to resolve domains internally (system and branded domains) which caused fetching assets to fail.We implemented a workaround which solved the errors while waiting for a fix from our vendor.

Reported by iPaper on their status page.

1. postmortemAug 26 · 12:21 UTC

   Incorrect date, time and number formatting in the Backend API and shop integrations

   Summary

   During a planned migration of our application servers on 24 August 2026 (04:00–08:00 CEST), dates, times and numbers returned by our Backend API and shop integrations started using a different regional format than before. Times were returned with a period instead of a colon (12.30 instead of 12:30), and decimal numbers with a comma instead of a period (1,50 instead of 1.50). We became aware of the issue the same morning and it was resolved on 25 August at 02:18 PM CEST. All formatting is now back to how it was before the migration.

   Impact

   The Backend API and our shop integrations were affected. Anyone with a custom implementation that depended on the exact previous formatting of dates, times or numbers could see errors or read values incorrectly, since the payloads we sent contained a different separator than expected. Because the two new servers were configured differently from each other, the same request could return either format depending on which server handled it, making the behaviour appear intermittent.

   One customer reported the issue, but it could have affected any integration relying on exact formatting during the affected period. Payloads delivered during the window cannot be corrected after the fact, so if you stored or processed values from the Backend API or a shop integration between 24 and 25 August, we recommend verifying them and re-fetching if needed.

   Cause

   The new application servers were provisioned with different regional settings. One used a Danish locale (comma as decimal separator and period as time separator) and the other a US English locale (period as decimal separator and colon as time separator). Values formatted according to the server's regional setting therefore changed compared to the previous setup, and were inconsistent between the two servers.

   Resolution

   We identified the difference in regional settings as the cause and deployed a code change that explicitly locks the formatting used by all affected applications to the same format as before the migration, independent of how the underlying server is configured. Formatting is now consistent across all servers and matches pre-migration behaviour. No action is required from customers, and integrations that were failing are working again.

Reported by iPaper on their status page.

1. ResolvedAug 24 · 06:00 UTC

   Dates, times and numbers were returned incorrectly by our backend API and shop integrations. Times used a period instead of a colon (12.30 instead of 12:30), and decimals used a comma instead of a period (1,50 instead of 1.50). This caused custom implementation targeting our backend API to fail if they depended on the formatting we had prior to migration.It has been resolved and all formatting is back to how it was before migration.

Reported by iPaper on their status page.

1. postmortemSep 4 · 10:43 UTC

   Summary

   During the migration we saw flipbooks failing to be served to end users, we quickly discovered this was a misconfigured cache that was not the correct size, which caused it to fill up and stop serving requests when we switched the load to the new servers. We had the issue resolved with new caching instances after 8 minutes of total downtime.

   Impact

   Caching is used for a wide array of things to be able to serve a flipbook and the admin, as this was unable to serve requests our applications failed.

   Cause

   During setup, the caching instances were deployed with an incorrect volume size, causing the instances to fill up once we moved over the traffic.

   Resolution

   We have expanded the volumes and made sure our scripts are correct for deploying any future caching instances.

Reported by iPaper on their status page.

1. ResolvedAug 24 · 04:30 UTC

   During the migration we saw flipbooks failing to be served to end users, we quickly discovered this was a misconfigured cache that was not the correct size, which caused it to fill up and stop serving requests when we switched the load to the new servers.We had the issue resolved with new caching instances after 8 minutes of total downtime.

Reported by iPaper on their status page.

1. resolvedAug 19 · 08:00 UTC

   Due to a faulty update from one of our infrastructure vendors our Load Balancers were unable to serve traffic for around 8 minutes while a deployment was being released that should have had zero downtime. We are working with the vendor to resolve the issue and in preventing it happening again.

Reported by iPaper on their status page.

1. CompletedAug 24 · 06:00 UTC

   The scheduled maintenance has been completed.

Reported by iPaper on their status page.