Historical record of incidents for SegMetrics
Report: "Redis cluster having issues"
Last updateThis incident has been resolved.
Our caching cluster is currently having issues. Data import and collection is continuing as normal. We are looking at solving this as soon as possible.
Report: "API Data Import Delay"
Last updateThis incident has been resolved.
We experienced a delay in data importing last evening from 11pm PST due to an issue with our upstream data provider. We have fixed the issue, increased importing speed for all accounts, and are currently running through the backlog of data imports. We expect the backlog to be fully imported by 11am PST.
Report: "Google Ads Import Delay"
Last updateWe experienced a delay in updating Google Ads performance data between 12:00 PM and 4:00 PM PST due to an issue with the Google Ads API. The problem has now been resolved, and data updates have returned to normal.
Report: "Delayed data (Up to 3 hours)"
Last updateData delay has been fixed, and we are now running at standard capacity.
We are continuing to monitor for any further issues.
We're seeing a slight delay in data imports. Import data may be delayed up to 3 hours. We are currently working with the data center to identify the bottleneck.
Report: "Delayed data"
Last updateThis incident has been resolved.
Over the weekend we became aware of an issue with adding imported data to the read databases. Data is being imported to our processing buffers, but are not being propagated to the read databases in a timely fashion for certain accounts. We have replaced the replication databases, and should be seeing the most recent data on all accounts. We're continuing to collect access and analytics data through our tracking shards with no data loss.
Report: "SSL Certificate Error"
Last updateWe became aware of a SSL expiration issue on the reporting application this morning. There was an issue with the certificate on our CloudFlare proxy not being correctly updated. We are currently working with CloudFlare to identify the root cause. This certificate issue only applied to the reporting application, and had no effect on the tracking or importing services.
Report: "This is an example incident"
Last updateWhen your product or service isn’t functioning as expected, let your customers know by creating an incident. Communicate early, even if you don’t know exactly what’s going on.
Empathize with those affected and let them know everything is operating as normal.
As you continue to work through the incident, update your customers frequently.
Let your users know once a fix is in place, and keep communication clear and precise.
Report: "Infusionsoft Imports Running Slow"
Last updateThis incident has been resolved.
We were alerted to a slowdown with Infusionsoft's importers that were causing data to be significantly delayed. We've currently resolved the situation and are monitoring the data import speed to make sure there are no more delays.
Report: "Data import slowdown"
Last updateWe were alerted to a slowdown with our data importers that were causing data to be significantly delayed. We've currently resolved the situation and are monitoring the data import speed to make sure there are no more delays.
Report: "Importer Slowdown"
Last updateThis incident has been resolved.
We've noticed a slowdown with adding imported data to the read databases. Data is being imported, to our processing buffers, but are not being propagated to the read databases in a timely fashion. The development team is working to identify the cause of the slowdown and fix it.
Report: "Database Resizing"
Last updateThis incident has been resolved.
Our upstream database provider had an unscheduled maintenance window for our database clusters. Downtime was approximately 5 minutes, during which time the dashboard was not available. Data import was uninterrupted during this time.
Report: "Upstream database issues"
Last updateThis incident has been resolved. All data connections are back up with no data loss. We will be sending out a full update shortly.
Our upstream data provider is having connection issues with the database service that our analytics dashboards run on. We're continuing to collect access and analytics data through our tracking shards with no data loss.
Report: "Degradation of Pixel Tracking"
Last updateWe identified an issue with the tracking pixel that was causing new visitors with certain browsers to not be tracked due to a browser-related cryptographical error. We use secure cryptographic tracking mechanisms to keep both customer and site visitor data secure. In this case a subset of browsers experienced issues related to a non-standard implementation of cryptographical functionality. This incident was identified by our data team and a solution sent out within 15 minutes. Total affected time was 25 minutes.
Report: "Upstream database issues"
Last updateThis incident has been resolved.
We've noticed a slowdown with adding imported data to the read databases. Data is being imported, to our processing buffers, but are not being propagated to the read databases in a timely fashion. We've increased our throughput to the processor and moved the data processors to a different queueing system. We're continuing to collect access and analytics data through our tracking shards with no data loss.
Report: "Upstream database issues"
Last updateWe've noticed a slowdown with adding imported data to the read databases. Data is being imported, to our processing buffers, but are not being propagated to the read databases in a timely fashion. We've increased our throughput to the processor and moved the data processors to a different queueing system. We're continuing to collect access and analytics data through our tracking shards with no data loss.