Is mux Down Right Now? Check if there is a current outage ongoing.

We are currently experiencing a service disruption affecting Mux Data Livestream Health Stats API and Real-Time engagement metrics. The issue is related to an ongoing outage with one of our underlying infrastructure providers. Our engineering team is actively monitoring the situation and working to mitigate impact where possible. We will continue to provide updates as we learn more.

We are currently investigating errors in Mux Video, specifically difficulties on delivery and ingest of some assets, that began at approximately 18:00 UTC. We apologize for the inconvenience and will share an update once we have more information.

Beginning UTC 18:43, we are seeing incomplete data for some customers and are investigating

We are currently investigating an issue where we're seeing elevated error rates delivering individual segments for HLS playback of videos. We will provide updates as we have them.

This incident has been resolved.

We have narrowed down these issues to networking issues in an underlying service provider's infrastructure, and are working with them to minimize any impact. Our team continues to actively monitor the systems for any impact.

There was a typo in the original message, which should have read (the key word, "not", was missing): At this time, all systems have recovered and operating normally. However, this is the third disruption of this type since ~10:50 UTC, and our team is actively working on ensuring that this does *not* re-occur.

Around 16:50 UTC, Mux experienced a brief disruption due to network connectivity issues affecting some webhook events and dashboard sessions. During this period: - Some customers may have experienced delayed webhook event delivery. - Attempts to access the Mux Dashboard may have resulted in errors. - API requests to endpoints such as /video/v1/assets and /video/v1/live-streams may have intermittently failed or taken longer to complete. - Some beacon collection for Data may be interrupted, causing errors making network requests on clients. The SDKs will retry these requests, resulting in no data loss. At this time, all systems have recovered and operating normally. However, this is the third disruption of this type since ~10:50 UTC, and our team is actively working on ensuring that this does not re-occur.

Between 10:55 and 11:01 UTC, Mux experienced a brief disruption due to network connectivity issues affecting some webhook events and dashboard sessions. This also occurred from 13:35 and 13:39 UTC. During this period: - Some customers may have experienced delayed webhook event delivery. - Attempts to access the Mux Dashboard may have resulted in errors. - API requests to endpoints such as /video/v1/assets and /video/v1/live-streams may have intermittently failed or taken longer to complete. At this time, all systems have recovered and operating normally.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

For this issue, in particular, some assets that are determined to be non-standard inputs due to high bitrate or GOPs that are too long are the ones that are erroring on ingest more often than they should. We are continuing to work on a fix for this issue, and will have an update on a fix soon.

The issue has been identified and a fix is being implemented.

We are currently investigating an issue where we're seeing elevated numbers of assets erroring on ingest when previously they would have succeeded. The majority of asset ingest is unaffected.

This incident has been resolved.

Ingest processing throughput has fully recovered and ingest queues have been processed. We are continuing to monitor the situation.

Job processing throughput continues to be delayed. We are continuing to work towards additional mitigations.

We are continuing to work towards a fix for the ingest delays. Tasks continue to be accepted and processed, however they continue to be delayed. We are deploying additional mitigations to improve processing speed.

The issue has been identified and a fix is being implemented.

The team is continuing to investigate the cause of delays in ingest. An update will be provided as soon as we have identified the issue.

We are currently investigating delays in video ingestion that began around 2:50 AM UTC.

This incident has been resolved.

As of 20:12 UTC, the team has implemented a fix and latency has returned to normal. At this point we expect all low latency live streams to be fully operational.

The issue has been identified and a fix is being implemented.

Starting at 17:40 UTC we began experiencing elevated latency for some low latency live stream delivery. The team is actively investigating.

This incident has been resolved.

A fix has been implemented for the webhook delivery delays. As of 17:50 UTC, webhook delivery metrics have fully recovered. We are currently monitoring the system performance and can confirm that all systems are operating normally.

We are currently investigating an issue where webhook deliveries are experiencing unexpected delays starting 16:25 UTC on March 14th.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have identified a fix for this issue and are working on deploying this fix. We will provide further updates as we have them.

We have identified an issue where the data for each video view sent to Kinesis or Pubsub via View Stream Exports is incorrect. This started around 10PM UTC on Feb 26th, and in the video views sent, some fields such as video_duration and watch_time (and others) are reported as 0. The data in the views via the dashboard or API are accurate, this is only in the data sent through the View Stream Exports. Our team is working on a fix at the moment, and we will provide updates as we have them.

We experienced a disruption in our ability to process delivery usage data, resulting in a reduced count of delivered minutes from 08:10 UTC to 09:31 UTC. We are working with our CDN vendor to understand if that usage will be backfilled. Any minutes that were unable to be processed will not be billed. We apologize for any inconvenience.

This incident has been resolved.

The team has mitigated the issue, and error rates have returned to normal at 17:01 UTC. From then on, generating image, subtitle, storyboard, and gif requests should be fully operational. We are continuing to monitor the health of the system.

We are investigating an increase in errors in generating image, subtitle, storyboard, and gif requests starting from 15:38 UTC.

This incident has been resolved.

We continue to monitor the health of the system.

We have identified and mitigated the issue. Error rates have returned to normal, and the backlog of jobs was processed by 08:57 UTC. We continue to monitor the health of the system.

We are investigating an increase in errors serving media segments, and delays in video processing starting from 08:29 UTC.

There was a disruption in our ability to process a subset of delivery usage data that will reflect a reduced amount of delivered minutes from 15:18 until 17:40 UTC. Any minutes that were unable to be processed will not be billed. We apologize for any inconvenience.

Starting at approximately 17:23 UTC on Nov 7th, we experienced delays on new on-demand asset ingest of up to 5 minutes on some assets. This issue has been resolved and all ingest caught up by 17:28 UTC

Between 19:54 and 20:20 UTC processing of all on-demand video ingest experienced delays. A fix was deployed at 20:14 UTC and processing delays were fully resolved at 20:20 UTC.

We are continuing to monitor the system.

The issue has been identified and a fix has been deployed. All delayed processing has recovered and video ingestion is operating normally.

We are currently investigating an issue that is delaying processing of newly ingested on-demand videos.

Between 7:00 and 7:15 UTC a small number live streams experienced elevated rebuffering rates, and some may have experienced unexpected disconnects during that time.

This incident has been resolved

Our team has identified the issue as an unexpected node shut-down and is monitoring the situation. We’ll post an update as soon as the issue is fully resolved.

Some real-time data is missing for some customers. We don't believe this is an ongoing issue.

At about 12:09 UTC through 15:20 UTC we experienced slightly higher rebuffering rates on a small set of live streams due to elevated networking errors.

This incident has been resolved.

We have finished the processing backlog of ingest jobs and Mux Video is operating normally. The engineering team continue to monitor the system health.

We have identified and are actively mitigating the issue. Issues with playback are resolved as of 13:30 UTC, and we are working through the ingest backlog.

We are currently investigating an issue causing live and on-demand playback issues and increased time in processing assets starting from 12:35 UTC.

We are currently investigating an issue causing increased time in processing assets starting from 12:35 UTC.

This incident has been resolved.

The team have identified and resolved an issue causing a delay in webhook delivery. Customers might have seen delayed webhook events between 03:50 UTC and 4:05 UTC.

This incident has been resolved.

A fix has been implemented and we are monitoring the results

The engineering team have identified the issue and are working towards resolution. Error rates are improving.

We are investigating an increase in errors when delivering images, subtitles, and storyboards. Video ingest and playback is not affected.

From 22:13 until 22:25 PST on June 25th, we experienced an issue causing incomplete and delayed data in the Data Monitoring Dashboard and APIs for customers configured for data ingest in the EU. No data was lost, and this data has been backfilled.

From 03:02 until 03:07 UTC on June 23rd, we experienced an issue where the data in the monitoring dashboard and API was incomplete. This issue has been resolved, though the data during that timeframe may be incomplete.

This incident has been resolved.

A fix has been implemented and we are monitoring results. Some views will be backfilled over time, while some will remain missing. This affects only EU customers.

We are rolling out a fix for this issue at 21:00-21:20 UTC. During that time we expect some EU views will be missing and unrecoverable. We will update when complete.

The issue has been identified and a fix is being implemented. There has been no further impact since 18:50 UTC.

We are continuing to investigate the root cause of this issue. Customers will see some views missing between 18:10 and 18:50 UTC

We are currently investigating an issue where some views are missing for customers in the EU between 18:10 and 18:25 UTC

This incident has been resolved

Our systems are reporting healthy with no disruption of service. We will continue to monitor and post an update as soon as the issue is fully resolved.

Our systems are reporting healthy and we do not observe any active disruption to our services. However, there are currently ongoing service provider disruptions that we continue to monitor while a fix is in progress. We will provide additional updates if we see any service impact.

The issue has been identified and a fix is being implemented.

We are currently investigating possible availability issues for Mux Video.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

/data/v1/metrics APIs were unavailable between 23:35 and 23:49 UTC. A fix has been identified and is rolling out.

We are currently investigating this issue.

This incident has been resolved. There was no customer impact.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented & we are monitoring the results. Users might have seen intermittent playback failures between 16:54 UTC and 16:59 UTC for a subset of requests.

We are currently investigating an issue that might cause playback failures starting at 16:54 UTC.

This incident has been resolved. Customers may have noticed delayed webhook delivery between 17:55 UTC and 18:12 UTC.

A fix has been implemented and we are monitoring the results. Customers might have seen delayed webhook events between 17:55 UTC and 18:12 UTC

The issue has been identified and a fix is being implemented.

We are currently investigating a delay in webhook delivery.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

The issue has been identified and we are working on a fix. Customers may see delayed data in their monitoring dashboard.

We are currently investigating this issue.

The majority of views collected between 3/22 19:30 UTC to 3/25 14:00 UTC are missing their device name, model, category & brand. Affected views will not be backfilled/corrected, but views going forward should again be reporting as normal.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and we are working on a fix. Customers affected should see about 5 minutes of delay in the monitoring dashboard.

We are investigating an increase in latency for Mux Data collection affecting some customers beginning 18:50 UTC

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investing delays in Webhooks being sent.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are investigating increased latency on Mux Data Dashboards.

This incident has been resolved.

A fix has been implemented and we are monitoring results.

We have identified the issue and our team is working on a fix. Affected customers will see degraded CCV values in their monitoring dashboard and delayed views & metrics starting at 12:20 UTC.

We are experiencing elevated data collection timeouts for a subset of Mux Data customers and are investigating

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The team is investigating increased playback failures and rebuffering on Mux Video starting between 11:25 AM and 11:32 UTC.

From 2024-02-22 20:44:33.904 UTC to 2024-02-23 21:37:21.719 UTC, customers using low-latency mode live streams may have experienced increased rebuffering and lower quality video playback.

This incident has been resolved.

We implemented a fix throttling suspicious activity and are seeing API availability return to normal. We are continuing to monitor the situation.

We are continuing to investigate this issue.

We are currently investigating issues with our APIs.

This incident has been resolved.

We are continuing to investigate this issue.

We are currently investigating issues ingesting Live streams. We will update this page once we know more.

This incident has been resolved.

A fix has been implemented and data has been backfilled for all customers. There was no data loss. We will continue to monitor the system for stability.

The issue has been identified and a fix is being implemented.

We continue to investigate this issue. EU ingest customers are not seeing any data after 22:36:00 UTC.

We are continuing to investigate this issue.

Starting at 22:36:00 UTC on Jan 31st, data present in the Monitoring Dashboards of Mux Data are incomplete for customers configured for data ingest in the EU. We are investigating

This incident has been resolved.

We have implemented a fix on Mux Video side to remove `#EXT-X-SESSION-DATA` tags from playlists. You should still upgrade to SDK version 3.5.1 or later.

We have identified an issue with the Mux Data AVPlayer SDK that results in elevated app crashes under the following conditions: - AVPlayer Data SDK version before 3.5.1 - HLS playback when the HLS playlist contains `#EXT-X-SESSION-DATA` tags If you are experiencing this issue you should upgrade to version 3.5.1 or later. In the meantime, for customers using Mux Video for playback, we are currently working on a fix to remove `#EXT-X-SESSION-DATA` tags from playlists to prevent this crash on older SDK versions. If you're not using Mux Video for playback you may want to consider doing the same until you can upgrade the SDK version.

This incident has been resolved.

Playback error rates are returning to baseline after rolling back a recent change. We are continuing to monitor the recovery.

We are currently investigating an issue with elevated playback error rates.

This incident has been resolved.

The issue has been identified and remediated. We continue to monitor the situation

Starting at 19:16:59 UTC on Jan 9th we are experiencing an issue where the data present in the Monitoring Dashboards of Mux Data may be incomplete for customers configured for data ingest in the EU.

This incident has been resolved. Live streams on the single affected pod in our EU region were impacted between 14:18 UTC - 14:55 UTC.

Error rates on the affected pod in the EU region have returned to baseline, there is currently no customer impact; we continue monitoring.

We are currently investigating an issue with a small subset of live streams in our EU region, causing playback failures on affected streams.

This incident has been resolved.

A fix has been implemented & we are monitoring the results. Low-latency live streams started before 16:59 UTC that have experienced playback issues will need to be restarted.

Our team has identified the issue and is actively rolling out a fix. We are starting to see recovery for a subset of users.

Our team has identified the cause of some viewers not being able to view low-latency live streams, and is actively working on a fix.

Our team is continuing to investigate issues with viewing low-latency live streams. We’ll continue to update you here.

We are investigating an issue with low-latency live streams, causing playback to fail. We apologize for the inconvenience and will share an update once we have more information.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating an issue that is causing the data in the Monitoring Dashboard and API to intermittently be incomplete or delayed.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have identified the issue and deployed a patch to restore functionality to all services. Our team continues to monitor the system for any issues.

Our team is continuing to work to resolve functionality, and will provide more updates as we have them.

We are currently investigating an issue where we're seeing elevated response times and 5xx errors for requests to api.mux.com. This is affecting Video and Data API routes, and is also causing intermittent issues accessing dashboard.mux.com. Live streaming and playback of video assets is unaffected.

Due to an outage in a CDN partner's service, we do not have complete delivery usage data for the time period of 11:45 UTC until approximately 12:31 UTC, and we are unable to recover this missing data. This missing data will not appear in the Delivery Usage API, and will not be charged for.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Our team is continuing to investigate this issue as a top priority, and will provide updates as we have them.

We are currently investigating an issue where as of approximately 22:30 UTC on Oct 30th we are experiencing elevated connection errors for live streams. This may cause some encoders to be unable to start broadcasting, or also cause issues when trying to reconnect. Video playback is unaffected

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are investigating an issue causing errors for requests to the live stream input health and viewer counts APIs on stats.mux.com.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are experiencing an issue causing data in the Monitoring Dashboard and APIs to be partially delayed. We have identified the issue and are working on a fix right now.

This incident has been resolved.

The team have mitigated the issue, and access to the unavailable views are now restored.

The team have identified the issue causing the data to not be available, and are working to restore access.

We're investigating an issue causing historic metrics before 10th October 2023 to be unavailable for a small number of customers.