Zid Incident History

1. postmortemOct 25 · 12:19 UTC

   Incident Post-mortem Report

   Dashboard exports and imports delayed for all Stores

   Date	Oct 25, 2026, 11:25 AM until 2:26 PM (UTC+3, KSA time). Duration: 3 hours 1 minute
   Type of Incident	Service degradation: background processing
   Title	A very large customer import stalled dashboard exports and imports for all Stores
   Impact	132 export requests from 66 Stores were delayed, by 81 minutes on average and up to 3 hours 43 minutes. 3 of them timed out and must be requested again. The customer imports of the Store that started the large import were delayed until 3:12 PM.
   Total impact on uptime	No outage. Stores, checkout and the dashboard worked the whole time. Only exports and imports waited.

   Summary

   Dashboard exports (orders, customers, wallet, abandoned carts and others) and dashboard imports are built in the background by one shared pool of workers. At 11:23 AM a Merchant started a customer import with about 1.05 million rows. The platform splits every import into parts of 100 rows, so this one became 10,486 parts.

   After each part, the platform checks whether the whole import has finished, so it can send the Merchant a summary. That check loaded every part of the import, rows included, into memory: about 200 MB per check for this import, with up to five checks running at once. The workers ran out of memory and restarted again and again. Export requests from every Store waited behind the import parts.

   We shipped a code-level change so the check only asks whether any part is still pending and adds up the final totals in the database. It no longer loads the rows, so its memory use no longer grows with the size of the import.

   Timeline (KSA time)

   Time	Event
   11:23 AM	A Merchant starts a customer import of about 1.05 million rows. By 11:27 AM it is split into 10,486 parts.
   11:25 AM	The first export request that has to wait is created. Later requests queue behind the import parts.
   ~11:34 AM	The workers start running out of memory and restarting. From here they finish about 5 import parts per restart and almost no exports.
   12:26 PM	A Merchant reports to support that 3 exports requested since 11:30 AM are still processing.
   ~1:15 PM	Product Engineering starts the investigation and traces the restarts to the import completion check.
   1:54 PM	Product Engineering prepares the change, with a test that reproduces the memory growth.
   2:08 PM	The workers start clearing the backlog, although they still restart from time to time.
   2:26 PM	The last delayed export finishes. Exports are back to normal.
   2:34 PM	The change is live.
   ~3:00 PM	The worker restarts stop.
   3:09 PM	3 export requests that had waited more than 3 hours are marked as timed out.
   3:10 PM	The large import finishes, except one part of 100 rows that the platform retries automatically.

   What was affected

   Area	State during the incident	How we know
   Dashboard exports, all types and all Stores	Delayed: 132 requests from 66 Stores, 81 minutes on average, up to 3 hours 43 minutes	Export request records. Orders 72, detailed orders 21, wallet 11, abandoned carts 6, customers 6, reviews 5, order preparation 4, coupons 3, other 4.
   Exports that timed out	Failed: 3 requests from 2 Stores	Export request records. These Merchants need to request the export again.
   Exports still processing	1 request	Export request records. The platform retries it automatically.
   Customer imports of the Store that started the large import	Delayed until 3:12 PM: 9 imports	Import records. All parts finished except one, which is retried automatically.
   Imports from other Stores	Not affected	No other Store started an import during the incident
   Export file contents	Not affected	Each file is built from live data when it runs, so delayed files are complete
   Stores, checkout and the dashboard	Not affected	Only the background workers for exports and imports restarted

   Action taken

   1. Found the cause: the workers that build exports and imports were restarting because the import completion check ran out of memory.
   2. Shipped a code-level change: the completion check counts pending parts and adds up the totals in the database instead of loading every part. A test fails on the old behaviour and passes on the new one.
   3. Confirmed that the delayed exports finished, and identified the 3 timed-out requests so support can ask those Merchants to request them again.

   Conclusions

   One Merchant's import should not hold up every other Store's exports. Exports and imports share the same workers, so a heavy import delays everyone. The change removes this cause.

   Work that runs once per part must not grow with the size of the whole import. The completion check read the entire import after every part. That was fine for typical imports and failed only for a very large one. With the change, an import of this size no longer exhausts the workers' memory.

Reported by Zid on their status page.

1. postmortemOct 9 · 11:54 UTC

   ملخص الحادثة: خلل في ظهور الثيمات وتخصيصها للمتاجر — ٩ أكتوبر ٢٠٢٦ ماذا حدث ظهرت بعض واجهات المتاجر يوم ٩ أكتوبر ٢٠٢٦ دون تنسيقاتها، فبدت صفحاتها نصوصاً وروابط متتابعة بلا تصميم. بدأ الخلل قرابة الساعة 1:06 ظهراً بتوقيت السعودية، وانتهى قرابة الساعة 2:48 عصراً. السبب أوقف أحد مزودي البنية التحتية خدمة التخزين التي تُحمَّل منها ملفات تصاميم المتاجر، والسبب خطأ في الفوترة والمحاسبة لدى المزوّد. لم يكن الخلل تقنياً، ولم يكن من جهة زد. الأثر • تأثر شكل واجهات المتاجر لدى العملاء طوال مدة الخلل. • لم تتأثر بيانات المتاجر ولا المنتجات ولا الطلبات، ولم يُفقد منها شيء. • ظلت صور المنتجات تعمل طوال مدة الخلل. الحل تواصلنا مع المزوّد مباشرة حتى صحّح الخطأ لديه وأعاد تفعيل الخدمة، فعادت المتاجر إلى شكلها الطبيعي دون أن يلزم التجار أي إجراء. ما سنفعله لمنع تكراره سنبني مصدراً بديلاً لملفات تصاميم المتاجر، تنتقل إليه المتاجر تلقائياً إذا تعطّل المصدر الأساسي، فلا يتوقف ظهور المتاجر بسبب مزوّد واحد. نعتذر عن الإزعاج الذي سببه هذا الخلل لكم ولعملائكم، ونشكر لكم صبركم وثقتكم. --- Incident summary: Store themes and customizations not loading — October 9, 2026 What happened On October 9, 2026, some storefronts loaded without their styling, so pages showed as plain text and links. The issue began at about 1:06 PM Saudi time and ended at about 2:48 PM. Cause One of our infrastructure providers suspended the storage service that serves store theme files because of a billing and accounting error on their side. This was not a technical failure, and it was not caused by Zid. Impact • Storefronts looked unstyled to shoppers for the duration of the incident. • Store data, products and orders were not affected, and nothing was lost. • Product images kept working throughout. Resolution We worked with the provider directly until they corrected their error and re-enabled the service. Stores returned to normal without merchants having to do anything. Preventing a repeat We will add a backup source for store theme files. If the primary source fails, stores will switch to the backup automatically, so a single provider can no longer take down how stores look. We apologize for the disruption to you and your customers, and we thank you for your patience and trust.

Reported by Zid on their status page.

1. ResolvedOct 9 · 03:00 UTC

   تم حل المشكلة: عادت تصاميم المتاجر إلى العملتجارنا الأعزاء،عادت الخدمة إلى العمل، وتظهر واجهات المتاجر الآن بتصاميمها المعتادة.إن ظهر متجرك لأحد العملاء دون تنسيق، فيكفي تحديث الصفحة أو فتحها من جديد. لا يلزمك أي إجراء آخر.سنواصل متابعة الخدمة للتأكد من استقرارها. نعتذر عن الإزعاج، ونشكر لكم صبركم وتفهمكم.---Resolved: Store designs are loading againDear Merchants,The service is back, and storefronts are loading with their usual designs.If a shopper still sees a store without styling, refreshing the page will fix it. You don't need to do anything else.We will keep monitoring to make sure the service stays stable. We apologize for the disruption and thank you for your patience.

Reported by Zid on their status page.