Historical record of incidents for Pexip Service
Report: "SIP Guest Join calls to Microsoft Teams meetings with 13-digit Meeting IDs may fail to connect"
Last updateThis incident has been resolved.
The Cloud OTJ fix has been implemented, and we are currently monitoring the results. For customers using the Ad Hoc Join for Teams macro on Cisco endpoints, we recommend updating the meetingId regex in the msteams_macro_settings.js configuration file to support 13-digit Meeting IDs. If you have any questions or need support related to this update, please reach out to your Pexip authorized support representative.
A fix has been implemented for the SIP Guest Join call connectivity issue. Our team is now working on a related problem where Microsoft Teams meetings with 13-digit Meeting IDs may not be correctly recognized by Cloud OTJ. A fix for this is currently being implemented.
We are currently investigating an issue affecting SIP Guest Join calls to Microsoft Teams meetings. Users may be unable to connect to meetings when the Meeting ID is 13 digits long.
Report: "Pexip Control Center displays incorrect System status"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Issue has been mitigated. We continue to monitor.
We are currently investigating an issue with Pexip Control Center where it displays incorrect system status of devices. No other services are being affected. Devices are being provisioned and registered correctly and calls are working as expected.
Report: "Pexip Control Center displays incorrect System status"
Last updateThis incident has been resolved.
A fix has been implemented and registered endpoints are now showing their correct system status. We will continue to monitor.
We are continuing to investigate this issue.
We are currently investigating an issue with Pexip Control center where it displays incorrect system status of devices. No other services are being affected. Devices are correctly being provisioned and registered and calls are working as expected.
Report: "Call Connection Issues at Oslo PoP"
Last updateNo further issues have been observed or reported and all systems are operational at this time.
A fix has been implemented and we are monitoring the results.
While we continue investigating the root cause, we have implemented a mitigation measure by rerouting registrations and call traffic away from the Oslo PoP.
We are currently investigating potential calling issues impacting the OSL (Oslo) PoP. We will provide updates as more information becomes available.
Report: "Intermittent call connectivity issues with Microsoft Teams CVI"
Last updateMicrosoft have now declared Service Restored.
Microsoft Engineering has mitigated the issue on their end, and our monitoring indicates no join failures in the past 30 minutes. The team will continue to monitor the situation for further verification.
We are continuing to monitor for any further issues.
Microsoft Engineering has mitigated the issue on their end, and our monitoring indicates no join failures in the past 30 minutes. The team will continue to monitor the situation for further verification.
We are seeing no further call join failures on the service. We continue to monitor.
Microsoft is currently investigating the issue under: TM1045640 - Users may be unable to join meetings within the Microsoft Teams service.
We have Identified a problem connecting to the Microsoft Back-end We continue investigations
We are currently investigating intermittent Microsoft Teams CVI call failures and assessing the scope of the issue.
Report: "Intermittent Call Failures in APAC"
Last updateFrom 09:07 AM to 09:33 AM UTC, a small number of VTC endpoints in APAC may have experienced difficulty joining Microsoft Teams meetings via Pexip CVI. The issue has been identified and resolved by the Operations team.
Report: "A Subset of Users May have experienced Issues Connecting to Teams Meetings via Pexip CVI"
Last updateThis incident has been resolved.
Today 2025-03-04 between 09:30 UTC and 10:03 UTC, some endpoints connecting to the United Kingdom PoP may have experienced issues joining Microsoft Teams meetings via Pexip CVI. The issue has been identified and mitigated, we continue to monitor.
Report: "A Subset of Users May Experience Issues Connecting to Teams Meetings via Pexip CVI"
Last updateFrom 10:02 AM to 10:07 AM UTC, a small number of VTC endpoints in Europe may experienced difficulty joining Microsoft Teams meetings via Pexip CVI. The issue has been identified and resolved by the operation team.
Report: "Endpoints May Not Be Actively Registered"
Last updateThis incident has been resolved.
Endpoint registration is returning. We are continuing to monitor for any further issues.
The Pexip team is investigating the loss of registrations for Endpoints. This may have also caused some calls to disconnect.
Report: "VMR Dialout "Invite by dialling" may not work"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Youtube streaming from VMR's was failing today, a fix has been applied and we are monitoring."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Report: "Intermittent Call Failures in Frankfurt PoP"
Last updateNo further issues have been observed or reported. All systems are operational at this time.
A fix has been implemented and we are monitoring the results.
The issue has been identified as related to intermittent DNS resolution failures, and our team is actively working on a mitigation. Further updates will follow as we work towards a full resolution.
We are currently investigating an intermittent issue that might affect SIP calls from unregistered in our Frankfurt Point of Presence (PoP).
Report: "New Call Setup and access to Control Center are currently not accessible"
Last updateNo further issues have been observed or reported and all systems are operational at this time.
The Ops team has completed validation and is continuing to monitor for issues.
We have rolled out a fix and are beginning to see normalization, are continuing to validate the service.
We have identified an issue with new calls are failing to setup and access to Control Center is also not working. The Ops teams are working to revert a recent change and we anticipate calls and Control Center will be working again soon.
Report: "Some H323 calls are dropping after 21 or 31 minutes"
Last updateThe issue is now resolved.
Fix is working on all new calls. The issue is mitigated and we are monitoring.
We believe the fix should work on all new calls. Existing calls made before the fix was applied at around 11:30 UTC may still drop
The fix has not resolved all cases - we continue to investigate.
The problem has been identified and a fix has been put in place. We are monitoring.
We are investigating
Report: "Call setup failures in Europe observed this morning at around 08:30 UTC and 09:05 UTC"
Last updateCall setup failures in Europe this morning at around 08:30 UTC and 09:05 UTC. The issue has been mitigated, and we are observing the affected part of the platform.
Report: "Intermittent RingCentral SIP Guest Join Call Failures"
Last updateThis incident has been resolved.
Issue has been identified and a fix has been applied. We are currently monitoring.
Pexip is currently investigating an issue with intermittent RingCentral SIP Guest Join call failures.
Report: "SIP messages missing for new calls in Pexip Control Center (Diagnostics and Live Meetings)"
Last updateThe incident has been resolved, and SIP messages have been displaying as expected since 7:00 PM UTC on September 19, 2024.
A fix has been implemented, and we are currently monitoring the results. Please note that it may take some time for SIP messages to appear in the Control Center for certain calls.
The issue has been identified. We are working on a fix.
We are investigating this issue. Messaging for some calls can be seen in Analytics.
Report: "H.323 Call Issue in Americas Region"
Last updateAn issue with H.323 calls in the Americas region began on September 2, 2024, at 06:50 AM UTC, potentially causing H.323 devices connecting to Teams CVI or Google Meet meetings to be unable to receive media from the Pexip Service. The issue has now been addressed and resolved as of September 4, 2024, at 08:06 AM UTC.
Report: "Customers may receive duplicate email notifications for endpoint statuses"
Last updateThis issue has now been resolved.
The cause of this issue has been identified and a fix has been deployed and tested. We are currently monitoring the situation for additional issues.
We are currently investigating reports that customers are receiving duplicate emails being sent out for endpoint online/offline status changes from Pexip Control Center. (PCC)
Report: "Poly Group Endpoints Provisioning Issue in APAC"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We're investigating reports of Poly Group endpoints in the APAC region being unable to connect to the provisioning service. Our team is working on identifying the cause and will share updates as soon as possible.
Report: "Pexip Contol Center could be falsely reporting endpoints as Offline, also showing old calls in "Live Meetings""
Last updateThis incident has been resolved
The issue has been identified and a fix has been implemented. We are currently monitoring.
We also see old calls in "Live Meetings" New calls ARE appearing, meaning no lack of functionality. We continue investigations on the issues.
Pexip Control Center could be falsely reporting endpoints as Offline when they are fully operational. Please use Analytics to verify the true status of the endpoints. Our Team is currently looking into this.
Report: "Pexip Contol Center could be falsely reporting endpoints as Offline"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results. Please note that it may take some time for the registration status of all Video Systems to be fully updated. Subscribers to notifications in Pexip Control Center might receive occasional false offline or online alerts, which they can safely ignore.
Pexip Control Center could be falsely reporting endpoints as Offline when they are fully operational. Please use Analytics to verify the true status of the endpoints. Our Team is currently looking into this.
Report: "Registered Endpoints not registering to closest PoP"
Last updateNo further issues have been observed or reported and all systems are operational at this time.
We are continuing to monitor for any further issues.
At this time we have rolled back our upgrade and most endpoints have reverted to their closest PoPs. If your device has not reverted to the closest PoP please restart your provisioning service or restart your endpoint to re-provision.
We are aware of an issue where some registered endpoints are not registering to their closest PoP. We have identified this issue is related to an upgrade from today and are in the process of rolling this back shortly.
Report: "Some recent calls are missing from Analytics (analytics.pexip.com)"
Last updateThis issue has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and we are actively working towards a resolution.
We are investigating this issue. Calls can still be seen in Pexip Control Centre - https://control.pexip.io
Report: "Pexip Control Center search functionality is temporarily unavailable"
Last updateThe issue has been resolved.
We have identified an issue where the Search bar does not return any results. The issue is identified and we are currently working on restoring the service.
Report: "Intermittent call failures in locations across the Americas and APAC."
Last updateThis incident has been resolved.
The internet provider has confirmed that the issue is resolved. Call routing is back to normal, and all systems are operational. We will continue to monitor to ensure service stability.
We have observed intermittent call failures at 05:16 AM, 05:20 AM, 06:16 AM, and 06:27 AM UTC due to a network connectivity issue in the QSF (California) PoP. The issue has been identified by a network provider as an error condition on a device in their network, and they are currently working on replacing the affected device. As a precautionary measure, we have rerouted calls away from the affected location.
Report: "Customers using SSO IdPs (other than Microsoft Entra ID) may experience login issues with Pexip services. Microsoft Entra ID users are not affected."
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
Customers using SSO IdPs (other than Microsoft Entra ID) may experience login issues with Pexip services. Microsoft Entra ID users are not affected.
Report: "Subset of CVI calls from North America Region are failing."
Last updateThis incident has been resolved
The issue is now resolved, and mitigation steps have been reverted.
Problem has been mitigated. Pexip is working with the vendor and continue to investigate the problem to get this resolved as soon as possible.
We are currently investigating a problem with some CVI calls from North America Region failing.
Report: "Pexip Web and Desktop app SSO login errors"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Users may experience issues logging in to the Pexip web and desktop apps if using Single Sign On. When encountering the error it will display 'No such relay state' and the user will not be able to proceed. We are currently investigating the cause of this issue.
Report: "MMV Mobile app users unable to login"
Last updateThis incident has been resolved.
The DevOps team has identified and mitigated the issue.
Users on iOS and Android devices are unable to login to the MMV app or join guest meetings with the conference name and numeric IDs. Some users have also reported the app crashing while loading. We are currently investigating the cause of this issue and recommend utilising the web or desktop app on a PC in the meantime.
Report: "Intermittent call connectivity issues with Microsoft Teams CVI in Europe"
Last updateDue to a call routing issue, a small subset of users in Europe may have experienced difficulties connecting to Microsoft Teams meetings via CVI between 08:58 AM UTC and 09:14 AM UTC on the May 21st. The issue has been resolved, and no further problems have been observed or reported.
Report: "Intermittent call connectivity issues with Microsoft Teams CVI in APAC Region"
Last updateDue to a call routing issue, a small subset of users connected via the Singapore or Tokyo PoPs may have experienced difficulties connecting to Microsoft Teams meetings via CVI between 04:02 AM UTC and 07:09 AM UTC on the May 21st. The issue has been resolved, and no further problems have been observed or reported.
Report: "Pexip Control Center is incorrectly reporting Endpoints Offline"
Last updateNo further issues have been observed and the issue is classed as resolved.
A fix has been implemented and we are monitoring the results. Please note that it may take some time for the registration status of all Video Systems to be fully updated. Subscribers to notifications in Pexip Control Center might receive occasional false offline or online alerts, which they can safely ignore.
We have received reports of Video Systems reporting incorrect status within Control Center. We are currently investigating this issue.
Report: "SIP/H323 -> Google Meet calls taking a long time to setup or failing to set up at peak times."
Last updateFurther adjustments were made before today's Peak which were shown to be successful. This issue is now resolved.
Whilst the implemented changes notably reduced the setup delays and failures during today's 08:00 AM UTC peak, there were still a few instances of delays and call failures observed. Our investigation is ongoing, and we aim to provide a complete solution as soon as possible
We have identified what we believe to be the cause of this issue and have deployed a fix. We will continue to monitor at this time
We are currently investigation the issue to establish root cause.
Report: "Potential Connectivity Issues for VMR Users"
Last updateThis incident has been declared resolved.
We have identified the cause and have applied a change which has fixed the issue. We are currently monitoring the situation for any additional adversities.
A fix has been implemented, and we are monitoring
We are investigating reports of failures to join VMR meetings on the service.
Report: "Unscheduled VMR YouTube stream/record is failing to establish"
Last updateThe issue with unscheduled YouTube streaming and recording is functioning properly and is consider resolved.
A fix has been implemented and we are monitoring.
We are currently investigating the issue. As an alternative, scheduled YouTube stream/record is working normally.
Report: "Some MMV calls to VMR may fail to establish Audio / Video connections when UDP is blocked on the network."
Last updateRollout is now complete and calls from affected networks are working normally
We are continuing to work on a fix for this issue.
Rollout is almost complete, with only 3 of 14 media servers remaining in OSL PoP, all other PoPs are complete.
The issue has been identified and a fix is being rolled out.
We are currently investigating the issue. As a workaround in the meantime, allow UDP traffic on port 10000 to our service, as detailed in our firewall page: https://help.pexip.com/service/firewall.htm?Highlight=test%40pexip.me#mmv
Report: "Pexip Analytics missing information."
Last updateThis incident has been resolved.
A fix has been deployed and call data has begun to backfill Analytics. At this time we are continuing to monitor the situation however we expect the backfilling process to take some time to complete.
We are currently investigating a situation where some calls and call data may be missing from Analytics reporting. This has no impact on current or future calls.
Report: "Possible H.323 call issues in the Pexip Tokyo PoP"
Last updateThis incident has been resolved.
A resolution has been created and implemented. The applicable calls are being routed back through the Tokyo Pop. We are currently monitoring the situation for any issues.
We are currently investigating a potential issue with H.323 call drops using the Pexip Tokyo based PoP. We have rerouted some traffic while we look into this.
Report: "H323 CVI, H323 Google Meet, and PSTN audio calls in the APAC region are experiencing connectivity issues, either failing to connect or connecting with no audio."
Last updateThis incident has been resolved.
Incident mitigated. We are monitoring.
The PSTN (Telephone calls) issue has been resolved. All issues have been mitigated.
The Gmeet and CVI calls incident has been successfully mitigated, and we are actively working on resolving the PSTN calls issue.
We are currently investigating a call routing issue affecting H323 and PSTN calls in the APAC region.
Report: "Some Teams CVI calls may fail to connect in East Americas"
Last updateThis incident has been resolved.
We are continuing to monitor this issue, at this time we are not aware of any call failures. We hope to be able to mark this issue as resolved in the next 4 to 6 hours.
We are continuing to monitor this issue, at this time we are not aware of any call failures.
We are currently working on an issue with Teams CVI calls that may fail to connect within the East Americas region. At this time we have not seen these call failures occur yet and are working to prevent failures. On-going calls are not affected.
Report: "SIP call routing issues in Europe"
Last updateResolved: The problem with Incoming SIP calls from unregistered endpoints in the Frankfurt PoP has been identified and successfully addressed. SIP calls may not have been routed correctly, but the issue is now resolved as of 2024-02-21 08:41 AM UTC and all systems are operational at this time.
Report: "MS Teams CVI - Intermittent Call failures"
Last updateWe have not observed any new failures since moving this incident to Monitoring, at this time we are considering the incident resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work with Microsoft and the issue has been identified. We anticipated to have the next update by 13:00 UTC.
We are continuing to work with Microsoft Support regarding this issue. We have also confirmed that this issue is being experienced by Pexip Service CVI users and Pexip Self-Hosted CVI users.
We are continuing to work with Microsoft Support regarding this issue. Customers experiencing this issue will see their CVI call dial but will remain on the "Please Wait, Connecting You" screen. Customers may find redialing the call will allow their call to reconnect.
We continue investigating the issue, and are working with Microsoft support to resolve this issue.
We are currently investigating reports of VTCs failing to receive media in MS Teams meetings.
Report: "Some users may be unable to initiate new calls and use some features on Pexip Portals"
Last updateNo further issues have been observed or reported. All systems are operational at this time.
A fix for The Cloud OTJ issue has been implemented, and Cloud One-Touch Join Providers can now be created. Our team is continuing efforts to enhance stability in response to the incident.
The current status indicates that the Cloud One-Touch Join service is experiencing issues, leading to the temporary inability to set up Calendar Providers.
All issues have been mitigated and the team is continuing to optimize performance.
We have identified a potential problem with Partner Portal users being unable to invite new EUL Users, and we are actively addressing and resolving any remaining issues.
Access to the Analytics portal is restored and we are monitoring the results.
A fix has been implemented and the call connectivity issue is resolved, however, the Analytics portal is not available at that moment.
The issue has been identified and our team is actively working on a solution.
We are currently investigating an issue with occasional call failures on the service.
Report: "Newly created onpexip.com record will fail to create"
Last updateThis incident has been resolved.
We have identified and applied a fix. We continue to monitor this at the moment.
The team has identified an issue with DNS records failing to create when partners create new onpexip.com domains in the portal.
Report: "Some users unable to log into the Pexip Service Partner Portal"
Last updateNo further issues have been observed or reported and all systems are operational at this time.
We have mitigated the issue with users logging into Pexip Service Portal. We are continuing to monitor to ensure no further issues.
It has been observed that some users are not able to log into the Pexip Service Partner Portal. We are currently investigating this issue.
Report: "Users may be unable to connect to Microsoft Teams, Google Meet and VMR meetings"
Last updateThere have been no additional impacts or incidents reported.
A fix has been implemented and we are monitoring the results.
The issue has been identified and our team is actively working on deploying a solution.
We are currently investigating reports of failures to join Microsoft Teams, Google Meet, and VMR meetings on the service.
Report: "Intermittent call failures in locations across South and North America"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented, and we are monitoring the results. Additionally, we have observed that some calls in the Americas may have been disrupted between 13:07 and 13:28 UTC.
We are currently investigating network connectivity issues that may affect a subset of VMR and CVI/Google Meet calls in locations across South and North America.
Report: "Creating new onpexip.com domains is not working"
Last updateThis incident has been resolved.
The issue has been mitigated and currently being monitored.
Creating new onpexip.com domains do not get SRV records auto created. The Operations Team is currently investigating.
Report: "Outbound Calls from VMRs to SIP Destinations Failing"
Last updateNo further issues have been observed or reported. All systems are operational at this time.
The cause of the issue has been identified and the issue mitigated. We are continuing to monitor calling performance to ensure that services have returned to normal operation.
Outbound VMR calls to SIP destinations routing via the Singapore PoP are failing. The Operations teams are currently investigating.