Historical record of incidents for Sharpen
Report: "Insights Dashboards displaying as white screen"
Last updateWe are currently investigating an issue where Insights Dashboards are loading as a white screen in our IZ0 Isolation Zone. Customers accessing dashboards through direct links may encounter a blank white screen and fail to load. Our team is actively working to resolve the issue.
Report: "IZ1 Sharpen Q Disruption"
Last updateThis incident has been resolved.
We have observed stability in Sharpen Q Phone and call connectivity. We will continue to monitor the sustained recovery.
All Sharpen Q Phone connection issues have been resolved. We are investigating potential remaining issues based on other supporting infrastructure telemetry data.
We have identified all appropriate corrective actions and are in progress with executing them. We expect resolution shortly.
We are seeing additional disruption to outbound dialing from desk phones. We have isolated the fault and are working through corrective actions now.
We are executing a fix for the remaining impacted users. Some users may experience a one time disconnect if they are on an active call. While this impact is not desired, it is necessary to fully restore the expected state of the supporting resources.
We have observed some continued impact to Sharpen Q Phone connections for a subset of users. We have corrective actions identified and will be executing them shortly.
A fix has been implemented and we are monitoring the results. If your users are still experiencing Sharpen Q Phone connection issues, it is advised that they log out and back in to re-establish a successful connection.
We have identified the issue and are in progress with executing corrective actions.
We are aware of an issue which may be impacting some customers leveraging the IZ1 (app.iz1.sharpen.cx) environment. Some users may be experiencing disconnects, unexpected dead air, or failed outbound calls. We have all necessary resources engaged and are working to resolve the issue as soon as possible.
Report: "IZ1 Sharpen Q Disruption"
Last updateThis incident has been resolved.
We have observed stability in Sharpen Q Phone and call connectivity. We will continue to monitor the sustained recovery.
All Sharpen Q Phone connection issues have been resolved. We are investigating potential remaining issues based on other supporting infrastructure telemetry data.
We have identified all appropriate corrective actions and are in progress with executing them. We expect resolution shortly.
We are seeing additional disruption to outbound dialing from desk phones. We have isolated the fault and are working through corrective actions now.
We are executing a fix for the remaining impacted users. Some users may experience a one time disconnect if they are on an active call. While this impact is not desired, it is necessary to fully restore the expected state of the supporting resources.
We have observed some continued impact to Sharpen Q Phone connections for a subset of users. We have corrective actions identified and will be executing them shortly.
A fix has been implemented and we are monitoring the results. If your users are still experiencing Sharpen Q Phone connection issues, it is advised that they log out and back in to re-establish a successful connection.
We have identified the issue and are in progress with executing corrective actions.
We are aware of an issue which may be impacting some customers leveraging the IZ1 (app.iz1.sharpen.cx) environment. Some users may be experiencing disconnects, unexpected dead air, or failed outbound calls. We have all necessary resources engaged and are working to resolve the issue as soon as possible.
Report: "Call Connection Issues in IZ1"
Last updateThe issue impacting inbound call handling in the IZ1 region has been fully resolved. Call traffic is operating normally, and no further disruptions are expected. Thank you for your patience while we worked to restore service.
Inbound call handling in the IZ1 region has recovered, and normal service has resumed. We are continuing to monitor the platform to ensure continued stability. No further impact is expected at this time.
Initial efforts have led to a recovery in busy signals for inbound calls in the IZ1 region. However, a smaller percentage of calls may still experience intermittent delays when connecting. Our team is still diligently working to correct the issue.
We are currently investigating an issue impacting inbound call handling in the IZ1 region. Callers may experience intermittent delays connecting or experience a busy signal.
Report: "Call Connection Issues in IZ1"
Last updateThe issue impacting inbound call handling in the IZ1 region has been fully resolved. Call traffic is operating normally, and no further disruptions are expected.Thank you for your patience while we worked to restore service.
Inbound call handling in the IZ1 region has recovered, and normal service has resumed. We are continuing to monitor the platform to ensure continued stability. No further impact is expected at this time.
Initial efforts have led to a recovery in busy signals for inbound calls in the IZ1 region. However, a smaller percentage of calls may still experience intermittent delays when connecting. Our team is still diligently working to correct the issue.
We are currently investigating an issue impacting inbound call handling in the IZ1 region. Callers may experience intermittent delays connecting or experience a busy signal.
Report: "IZ1 Inbound Calling Impact"
Last updateThe primary service incident that began earlier today is resolved. We understand this has been an extended disruption and sincerely apologize for the impact this had on your operations throughout the day. We appreciate your patience and understanding. While the main issue is resolved, there is a remaining defect where some call recordings are not immediately accessible through the Insights user interface or the contact card. Our team is actively working on the final remediation steps for this. We anticipate that all call recordings will be fully accessible by the end of today. Should you require immediate access to a specific call recording before it becomes available in the UI, please open a request with our Care team. They will be able to retrieve the recording for you. Thank you again for your continued patience.
All WebRTC resources are fully restored and confirmed to be working. All previously observed impacts have subsided and we're monitoring recovery.
The majority of WebRTC resources are now functioning as expected and most users should be seeing recovery. We are correcting a small portion of remaining infrastructure and will update when all necessary resources are fully restored.
Outbound calls are now working as expected.
We are now observing a disruption to outbound Sharpen Q initiated calls.
We are performing additional corrective actions on remaining WebRTC hosts to extend recovery. There may be a brief interruption during the change.
We are observing disruptions to outbound calls and inbound call alerts for some agents. All necessary subject matter experts continue to investigate.
We are continuing our investigation. We've been able to identify and correct a portion of the resources supporting the Sharpen Q Phone WebRTC collective and are continuing efforts toward full restoration
Some WebRTC connections are still failing to connect and we're continuing our investigation. If a user has an active connection, it is recommended to not refresh to potentially lose the connection.
Current symptoms recognized are as follows. The investigation continues to correct the last symptom, unexpected drops from Sharpen Q phone. 1. Dead air for a portion of inbound calls (variable rate of failure) <- Corrected as of 1:54 PM ET 2. Unexpected disconnect from Sharpen Q phone 3. Unexpected drop of connected call at 2 minutes. <- Corrected as of 12:30 PM ET
We have identified another corrective action and are coordinating steps to implement.
We are continuing our investigation but have been able to identify that the current expected failure rate is ~20%. All necessary resources are engaged to resolve this issue as soon as possible.
Current symptoms recognized are as follows. The investigation continues. 1. Dead air for a portion of inbound calls (variable rate of failure) 2. Unexpected disconnect from Sharpen Q phone 3. Unexpected drop of connected call at 2 minutes.
We are seeing a resurgence of the symptoms at a low rate. We are resuming investigation to identify additional corrective actions.
A fix has been implemented and we're monitoring recovery.
We have isolated the suspected fault point and are taking corrective action.
We are currently investigating an issue with inbound calls dropping or resulting in dead air.
Report: "IZ1 Intermittent dropped Sharpen Q calls delivered from Logic+ IVR"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
This has been isolated to a very small set of customers. The majority of users are not impacted by this incident. We continue our investigation to identify and correct this issue.
We are currently investigating an issue that's causing between 30 and 45% of inbound calls to be dropped if they have been transferred to Sharpen Q from a Logic+ IVR. All necessary resources are engaged to identify and resolve this issue.
Report: "Reporting and Insights Unavailable in IZ0"
Last updateThis incident has been resolved.
We're currently investigating an issue affecting customers in IZ0 where the Insights page and dashboards are not loading as expected. Users may see partial content or a blank screen when accessing these features.
Report: "Intermittent dropped calls (IZ1)"
Last updateThis incident has been resolved.
We are planning a more comprehensive fix planned to be completed by 9:00 PM EDT. We apologize for the continued disruption.
Unfortunately the fix did not have the intended result and is being rolled back.
We have identified a potential fix and are working on the hotfix now. We will work to release it as soon as it is ready.
We are continuing to work on a fix for this issue.
We are continuing our investigation and are working to restore service as soon as possible.
We are continuing to investigate this issue. At this stage we have isolated the fault zone and can see that, for customers leveraging Logic+ as the toolset for their IVR, between 30% and 50% of inbound calls may unexpectedly disconnect at the 2 minute mark. All necessary resources continue to be engaged to resolve this issue as soon as possible.
While our changes have introduced improvement in the frequency of failure, we are still observing some unexpected disconnected calls at the 2 minute mark for calls delivered to queue from a Logic+ IVR. We continue to have all necessary resources engaged to resolve this issue.
We are continuing to monitor resolution of the issue.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Our initial fix had positive impact, reducing the frequency of the event. However, additional measures were needed to establish full recovery. We have implemented additional changes and are analyzing our recovery.
The issue has been identified and a fix is being implemented.
We have isolated the likely cause and are continuing our investigation. Thank you for your continued patience.
We are continuing to investigate this issue.
We are currently investigating an issue causing intermittent dropped calls for inbound callers to the IZ1 (app.iz1.sharpen.cx) environment. If customers are leveraging Logic+ for their inbound IVR calls, subsequent transfers to Sharpen Q from the IVR, will result in an unexpected call drop at the 2 minute mark. We have all necessary resources engaged to correct the issue as soon as possible.
Report: "Intermittent Dead Air (IZ1)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Callers to the platform of IZ1 customers (those logging into app.iz1.sharpen.cx) may be experiencing dead air or disconnected calls when calling. This may be impacting up to 70% of calls, but is quite intermittent in its impact. We have all necessary resources engaged to resolve the issue and are working on a corrective action to deploy as soon as possible.
Report: "Intermittent dead air calls (IZ1)"
Last updateThis incident has been resolved.
We are seeing initial signs of recovery, and are closely monitoring.
We have identified the source of the problem and are finalizing our corrective actions plans. We're targeting resolution within the next 15 minutes.
We have identified the fault zone for this issue and are working through remediation options to restore service as soon as possible.
Customer in IZ1 (app.iz1.sharpencx.com) may be experiencing intermittent disruption to inbound calls
Report: "Sharpen Q Caller ID Disruption"
Last updateThis incident has been resolved.
We’ve detected a partial disruption to caller ID name lookups in Sharpen Q. Some users might see "Unknown Caller" displayed for the caller ID name on certain incoming voice calls. We’ve pinpointed the problem to our supporting callerID lookup provider and are actively working to fix it as quickly as possible. Though some caller ID names are missing, calls are still routing normally and displaying the correct caller ID number.
Report: "IZ1 Inbound Logic+ IVR transfer to queue disruption"
Last updateThis incident has been resolved.
We have implemented a fix and are assessing recovery.
An issue has been identified that is blocking certain incoming IVR calls processed through Logic+ from successfully connecting to agents in Sharpen Q. We have all necessary resources engaged to restore service.
Report: "IZ1 Sharpen Q Disruption"
Last updateThis incident has been resolved.
We are seeing recovery to all impacted services and will continue to monitor.
We have observed restoration of underlying infrastructure and are re-assessing service availability.
We have isolated the fault zone of the issue and are working with our cloud infrastructure provider, AWS, to identify a solution. Currently, users may be impacting disruption to call controls (hold, transfer, mute, hangup, etc.), but calls can still be handled after initiation. Additionally, some call flows leveraging logic+ may be experiencing impact. We are working diligently to restore service as soon as possible.
We are aware of a potential service disruption to Sharpen Q in our IZ1 isolation zone. We have all necessary resources engaged to restore service. Additional details will be provided soon.
Report: "Sharpen Q Disruption (IZ0)"
Last updateThis incident has been resolved.
The fix has been confirmed. All systems look stable. We will continue to monitor for 15 minutes and then mark this item as resolved.
We have identified the source of the issue and have implemented corrective actions. We're assessing the results of the change and will notify once resolution has been confirmed.
We are aware of an issue impacting Sharpen Q operations in our IZ0 isolation zone. We have all resources engaged to resolve the issue as soon as possible.
Report: "Sharpen Database Availability Issue - IZ1 - 19DEC24"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
In collaboration with AWS we have identified a path to resolution. We expect resolution within 4 hours and will be providing hourly updates as we progress.
Corrective actions/efforts are still underway with our vendor (AWS) -- where the underlying database communication issue has been identified. We'll proceed with validation efforts and post further updates to this status page, as soon as restoration has been achieved.
We are currently investigating an issue where Call Flows using SharpenDB2 for data dips are failing, resulting in inbound interaction failures or routing misbehavior. We have identified the issue and are working with our vendor to resolve.
Report: "IZ0 Application Delays and Connection Issues - 10DEC24"
Last updateRecovery from this misbehavior was established at approximately 3:00PM Eastern, and we have not encountered any further responsiveness/delay issues across the application. Service has been fully restored, but we'll continue to monitor the situation. Please reach out to the Care Team in the event you have end users that are still encountering lingering symptoms.
We've identified an error trend within our infrastructure, and our Development Team has taken initial steps to correct the misbehavior. Our initial validation tests show a return to app responsiveness (being able to make/take calls, change statuses, etc). We're continuing to monitor our infrastructure for any indicators or error trends. Please reach out to the Care Team if you're seeing signs of sustained disruption.
We're aware of an issue for customers hosted within our IZ0 infrastructure - where app behavior is significantly delayed/unresponsive. Symptoms range from seeing a "Connecting..." overlay appear in the Sharpen Q application, to being unable to answer interactions in a timely fashion. End users may also be experiencing problems with modifying their status or logging in/out of the application. We have all technical resources engaged at this time and are actively troubleshooting the issue.
Report: "IZ1 users (app.iz1.sharpen.cx) unable to login"
Last updateWe continue to see sustained recovery. This incident is resolved. Thank you for your patience.
We are observing recovery as of 9:30 AM EST. We are continuing to monitor and identify the specific cause.
We have isolated the issue and can confirm that users are intermittently able to access app.iz1.sharpen.cx, but some are still experiencing failure. We have all necessary resources engaged and will continue to work for the earliest possible service recovery.
We are aware of an issue in IZ1 (app.iz1.sharpen.cx) preventing users from being able to login. All resources are engaged and are working to restore service as soon as possible.
Report: "Insights Application Not Loading - Blank Screen"
Last updateThe underlying issue has been resolved for Insights, and the Care Team has verified all reports/dashboards/agent vues are loading at this time in IZ0 and IZ1.
We've identified the underlying issue that is causing the reporting end-point to be unresponsive/only load a blank page in the application. We're in the process of getting things corrected and will share another update when there is full restoration to Insights.
We're aware of an ongoing issue with the Sharpen Insights application suite - where navigating in-app, loading a saved Dashboard link, etc results in a blank white screen for all users. The issue has been reproduced by our Care Team in both IZ0 and IZ1 portions of our infrastructure. We're working to identify the underlying issue, and will share additional details as more information becomes available.
Report: "Service Interruption - Sharpen Application Suite (IZ1)"
Last updateTransitioning to a resolved status, as all systems are back up and running - and we have full redundancy restored at the infrastructure level. A root-cause analysis report is being compiled at this time by our Development/Operations resources and the vendor (AWS), and will be sent out via cases that were opened with the Care Team when it is finalized.
Our Dev Team verified that all services auto-recovered after the database failover completed - and that full service availability has returned to the application. Any users experiencing continued symptoms with inbound call handling/alerting or outbound calling difficulty should log out and back into the application. Please reach out to the Care Team if continued impact is encountered. We're actively monitoring our infrastructure at this time, and have engaged with Amazon Web Services to identify root-cause of the availability/restart issue.
At approximately 12:45PM Eastern, our infrastructure team received an alert tied to Database availability. Services have automatically failed over to our backup infrastructure, and we're actively investigating the underlying cause - while performing additional process/app validations.
Report: "Application Login Slowness - IZ1"
Last updateThe underlying issue within our Infrastructure was identified and corrected. Login behavior is back to a fully operational status. If any users are still encountering problems at this time - have them fully log out of the application, then log back in (to force their browser session to pull an updated session configuration).
Customers running within our IZ1 infrastructure may notice delays with getting logged into the application. Upon navigating to app.iz1.sharpen.cx, it is taking upwards of 30 seconds for a login attempt to complete. Users may also experience delays with navigating between pages/toolsets in the Admin application.
Report: "Sharpen Service Desk Interruption"
Last updateWorking with our Service Desk vendor (JIRA), the Care Team discovered that our "Support@sharpencx.com" inbox monitoring entered a degraded state on 18OCT24. The underlying issue has been resolved as of 2:05PM Eastern on 29OCT24. If you had submitted a new service request or responded to an existing incident via to the "Support@sharpencx.com" inbox between those dates, your message was unable to trigger the necessary automation for support case generation and updates to have been received. Please reach out to us in the event you are still experiencing any misbehavior associated to use of the Sharpen product suite or are having difficulty getting in contact with our Care Team!
Report: "Reports of agents in IZ1 isolation zone unable to place outbound calls"
Last updateWe've identified the issue and have corrected it. This incident has been resolved.
Thank you for your patience as we work through this impacting event. Investigation continues into the cause of the behavior. You may see different unexpected behaviors with Sharpen Q as we work through mitigating efforts to resolve the issue.
We continue to narrow down on the issue internally and hope to have an update shortly.
We continue to investigate this behavior with our internal development and infrastructure teams. Once an update is available we will post it here.
We have received reports of agents being unable to place outbound calls from the IZ1 isolation zone. We are investigating and will post updates here as more information is available.
Report: "IZ1 Service Disruption"
Last updateThis incident has been resolved.
All corrective actions have been implemented and we are observing recovery. We will continue to monitor for the next 15 minutes.
We have identified there are still some lingering interruptions to functions such as hanging up calls. Our team continues to employ corrective actions to restore service as soon as possible.
We have identified the fault as an underlying AWS service crash. Recovery was automatically established, and we are monitoring for sustained recovery.
Users of the Sharpen Q platform within our IZ1 isolation zone (app.iz1.sharpen.cx) may have experienced intermittent degradation starting at 4:22 PM EDT. All necessary resources are engaged and working to fully restore service as soon as possible
We are aware of a potential service disruption in our IZ1 isolation zone. We have all necessary resources engaged to restore service. Additional details will be provided soon.
Report: ""Unknown Caller" Caller ID name"
Last updateSince 3:39 PM EDT, we have observed about 90% recovery from our CNAM carrier, resulting in a small amount experiencing the original "Unknown caller" symptom. We extended the monitoring period in anticipation of 100% recovery within normal business hours, but it has not yet been observed. We continue to request updates from our carrier in effort to expedite full resolution. Since the observed impact is now below our Severity 4 impact levels, we are resolving this notification and resuming updates with individually impacted customers. Should you have continued concerns regarding this impact, open a support request at support.sharpencx.com
Our vendor has implemented a resolution and is currently monitoring results.
Our vendor has identified the cause of this incident and is working towards a resolution.
We are aware of an issue impacting caller ID names displaying as "Unknown Caller". This is an expected fallback for a name lookup failure with our carrier. We are currently working with our vendor to resolve.
Report: "Sharpen Q Interaction Routing Disruption (IZ0)"
Last updateThis incident has been resolved.
We have implemented a fix and are monitoring recovery.
We are continuing our efforts toward addressing this issue and estimate resolution to be achieved within 15 minutes. In addition, scope of impact has been identified as follows. Sharpen Q inbound and outbound interaction routing operation impacted intermittently at varying levels of disruption including delay through full disruption.
We have identified the issue and are working through corrective actions to restore service as soon as possible.
We are experiencing an issue disrupting some voice interactions for users of the Sharpen Q platform in IZ0. We have all resources engaged and are restoring service as soon as possible.
Report: "Inbound IVR route to queue non-functional for some call flows (IZ1)"
Last updateThis incident has been resolved.
A fix has been implemented as of 9:09 AM EDT and we are monitoring sustained recovery.
Some users of the Sharpen platform are experiencing disruption to their IVR inbound call flows where menu options chosen are not routing interactions to queue. We have isolated the issue and have all necessary resources engaged to resolve this incident as soon as possible.
Report: "Sharpen Q disruption"
Last updateThis incident has been resolved.
We have identified the root cause of this issue. A vendor supporting a portion of our operations entered a degraded state, causing this issue. This vendor has restored service and is monitoring recovery. We will continue to monitor recovery along side our vendor, as well as pursue preventive measures to prevent impact, should a similar event occur in the future.
Our fix validation affirms resolution of this issue. We are monitoring for sustained recovery.
We have identified the issue and are evaluating initial recovery steps.
We are continuing to investigate this issue.
We are aware of a potential service disruption impacting Sharpen Q and an inability to pick up calls. We have all necessary resources engaged to restore service. Additional details will be provided soon.
Report: "Certain Icons not loading within Sharpen application"
Last updateThis incident has been resolved.
This issue has been corrected and we are continuing to monitor. If impact is still observed, please log out of the Sharpen application and log back in.
We are continuing to work with our vendor on a fix to address this issue.
Sharpen is experiencing additional UX impact related to a dependent vendor who is having an outage. We are working with this vendor to re-establish functionality. Certain icons and buttons may not be observed as loading in the UI.
Report: "Certain Icons not loading within Sharpen application"
Last updateThis incident has been resolved
Sharpen is experiencing a UX impact related to a dependent vendor who is having an outage. We are working with this vendor to re-establish functionality. Certain icons and buttons may not be observed as loading in the UI.
Report: "IZ0 Intermittent Sharpen Q Inbound Call Delivery Failure"
Last updateThis issue has been fully resolved.
We have identified the issue and have implemented a fix. We are monitoring recovery now.
We have identified an issue causing intermittent disruption to inbound Sharpen Q voice interactions. We have all necessary resources engaged to resolve the issue as soon as possible.
Report: "Sharpen Q degration (IZ0)"
Last updateThis incident is resolved.
We have identified the issue and have implemented a fix. We are monitoring for sustained recovery for the next 30 minutes.
We are observing an intermittent delay in processing Sharpen Q events within our IZ0 (app.sharpencx.com) isolation zone. Users may not be able to answer, place, or handle interaction. We have all necessary resources engaged to restore service as soon as possible.
Report: "Inbound/Outbound calling in IZ0 inoperable"
Last updateNow that a period of stability has been observed this event is being marked as resolved. As always if you encounter any additional issues don't hesitate to reach out to SharpenCare for assistance.
Setting the status of this impact to Monitoring.
We have successfully completed restorative measures and are seeing success both in logging and in testing. If you continue to experience impaired behavior within Sharpen Q please update your current support case with the SharpenCare team or reach out to the team for additional assistance. The status of the event is now in Monitoring as we maintain hightened visibility to this particular area of our infrastructure and will resolve the event once we perceive sustained a period of stability.
Restorative measures continue as we work to correct the impacting issue. We will continue to post updates as more info is available.
We believe we have located the issue and are in the process of taking corrective actions to recover. Once recovery has been confirmed via logging we will monitor for continued impact.
Customers in our IZ0 isolation zone will be experiencing issues placing and receiving calls within Sharpen Q. We're investigating this behavior and will update in 15 minutes with the current status.
Report: "Interaction assignment degradation (IZ1)"
Last updateThis incident has been resolved
We've identified the source of the disruption and have implemented a fix. We are monitoring for sustained recovery.
Users may be experiencing slowness or the inability to handle some interaction in Sharpen Q. We have all necessary resources engaged and are working to resolve this issue as soon as possible.
Report: "CX projects unable to initiate in IZ0 (app.sharpencx.com)"
Last updateThis inicident is fully resolved.
We have implemented a fix and are monitoring recovery.
Customers leveraging .CX (most commonly used for custom IVR data dips and custom Sharpen Q tabs) may be experiencing failures in the execution of CX projects. This does not impact VCX (Voice CX). All necessary resources are engaged and are working to resolve the problem as soon as possible.
Report: "Inbound call routing impact if leveraging SharpenDB lookup"
Last updateWe have continued to see successful SharpenDB query events since our recovery steps were put in place. If you continue to experience any service impact don't hesitate to reach out to the SharpenCare team for assistance.
Recovery steps are complete for IZ1 and we are now seeing successful querying to SharpenDB from VCX projects. We are moving the status of this impact to monitoring while we validate further. Please reach out to the SharpenCare team via the service portal or at (355) 249-3357, option 2 if you continue to experience impact.
Steps to recover in the still impacted IZ1 isolation zone are in motion. We expect to see recovery of functionality shortly. An update of service restoration will be sent as soon as the change steps are complete. Currently sustained recovery is being seen in the IZ0 isolation zone.
Restoration of functionality has been completed for the IZ0 isolation zone with restoration to IZ1 incoming momentarily.
We are nearing confirmation in the cause of the behavior and are putting next steps in place.
We've found and corrected issues we were seeing with Logic+ ability to query SharpenDB however the ability for VCX projects to query SharpenDB is still impacted. We're continuing to isolate the cause for that functionality to complete full restoration.
We're continuing to narrow down on the area of concern with this impact and are closing in on a potential cause for the behavior. Next steps for recovery are to come shortly.
We're receiving early indications that call routing may be impacted for some customers if they use SharpenDB data lookups to return data for intelligent routing. Our Care and Development teams are reviewing the behavior and will have next steps soon.
Report: "Outbound SMS messaging failure"
Last updateThis issue is resolved.
This issue is now confirmed to be fully resolved and preventive measures are in place.
We are continuing to monitor the recovery of Outbound SMS message delivery. SMS continues to operate as expected. We're working with our carrier network to finalize our preventive measures.
Outbound SMS messages are now being delivered on time.
Our past update indicated messages are failing to send. Recent data analysis reveals that messges are delayed by about 10-15 minutes, rather than being failures. We are continuing to work to resolve fully.
Outbound SMS messages sent from the Sharpen platform are currently failing. We've identified the fault and are working with our carrier network to resolve.
Report: "Outbound dialing impact"
Last updateThis incident has been resolved.
The fix has been released and validated. Users will no longer experience this issue. We will monitor for the next 15 minutes.
We have identified the issue and are working on a fix.
We've received communication outlining some issues being experienced by agents placing outbound calls in Sharpen Q via the 'Place Call' button on an interaction or switching between interactions. This is currently under investigation and we'll update shortly.
Report: "VCX Inbound Routing Availability Issues - IZ1"
Last updateThis issue has been resolved.
The fix has been deployed and we are observing recovery. We will monitor resolution for the next 15 minutes. If you have changed your call flow to workaround this issue, you are safe to restore to previous configuration.
Our Dev Team has identified the underlying issue and are in the process of validating a fix. We are actively working to implement their corrective actions to our Production system and begin additional testing measures to confirm recovery.
Our development/infrastructure teams continue to work through various troubleshooting tasks, in an effort to directly resolve the underlying issue or identify a way to workaround the communication problem impacting VCX functionality.
We are still actively troubleshooting/testing various changes in our infrastructure - to attempt to restore functionality or divert traffic around the portion of the system that is having an issue interacting with VCX processing functionality.
While our removal of 2 anomalous servers has reduced the volume error tracing in our system logs, we're still seeing continued impact on the portion of our infrastructure that is responsible for integrating call traffic with VCX functionality. Our Dev Team is currently pursuing options to standup new infrastructure or find ways to redirect traffic around the failing communication path, in order to reach the portion of our network hosting VCX projects.
In our efforts to drive recovery, we're removing 2 specific servers that present a different behavior pattern than others in our infrastructure - indicating they could be consuming resourcing and locking up communication paths with the servers/processes that host VCX functionality (denying their use to other services, calls, etc that are trying to interact with a VCX project). We're continuing to work towards resolution, and are pursuing all available options to correct this misbehavior.
Our Dev Team is continuing to health-check various system services and devices within our network, in order to systematically rule out items that could be contributing to the communication/port-traversal issue that is preventing call traffic from processing through to a VCX project. Additional infrastructure is being stood up in an attempt to rule out any sort of corrupted device/service from the equation. Further updates will be posted after we're able to divert traffic within our network and run diagnostic tests.
Our Dev Operations team has identified an area of contention; where network communication isn't happening in the expected fashion over certain port ranges that are necessary for VCX functionality. We are still actively investigating the issue and have a specific area of misbehavior we're focus our troubleshooting efforts on. Further information will be shared when we take action to correct the network communication block/availability issue.
In an attempt to restore functionality, our Dev Operations/Infrastructure team has been restarting impacted services aligned with VCX functionality (in parallel with reviewing examples, system logging, etc to identify the underlying cause). Further information will be posted to this status page - regarding our continued efforts to correct the misbehavior as quickly as possible for IZ1-based customers.
We're aware of a misbehavior that surfaced at approximately 12:30PM Eastern for IZ1-based customers; where Inbound Routing -- that is dependent on VCX project availability -- is failing to properly hand-off call traffic. Our Development Operations team has been engaged and we're actively investigating the issue.
Report: "Inbound/Outbound Call Failures in Sharpen Q (IZ0)"
Last updateThis incident has been resolved.
We have identified the root cause impacting Sharpen Q call operations and have restored service as of 12:38 PM Eastern. If users are still experiencing issues placing or receiving calls in Sharpen Q, please have them log out and back in. If issues continue to persist, please reach out to Sharpen Customer Care.
We are aware of a potential service disruption that is impcating the ability to process calls in Sharpen Q. We have all necessary resources engaged to restore service. Additional details will be provided soon.
Report: "Agents experiencing long load times for interactions in Sharpen Q"
Last updateFurther monitoring has continued to show resolution to the issue. This incident is now resolved.
An infrastructure issue has been identified and corrected linked to a third party logging vendor. A configuration change has been made to remove the dependency on this service to restore functionality. Currently we're seeing recovery and are in a monitoring state. As always if you continue to experience issues with this behavior please reach out to the SharpenCare team for further assistance.
We're receiving reports of agents experiencing long load times and intermittent issues with inbound interactions becoming stuck while loading for agents working in Sharpen Q. This behavior is under investigation. We'll post updates as they become available.
Report: "Intermittent Sharpen Q interruption (IZ0 - app.sharpencx.com)"
Last updateThis issue has been resolved.
We have identified the source of the issue and implemented corrective actions. Sharpen Q is now working as expected. Some users, if they were handling interactions at the time of the event, may be stuck in a status, or have a stuck interaction. If agents are experiencing this issue, please have them log out and back in to clear their state. If there is a sustained stuck state, please contact the SharpenCare team to help correct. We will continue to monitor for sustained recovery.
Users of the IZ0 Sharpen Q instance may be experiencing intermittent interruptions to Sharpen Q functionality. Our team is investigating and will work to identify and correct the issue.
Report: "Intermittent dead air IZ1 isolation zone (app.iz1.sharpen.cx)."
Last updateThis issue has been resolved.
A fix has been implemented as of 4:52 PM EDT. Testing has confirmed restoration for 100% of test calls. We will continue to monitor for sustained recovery.
We are continuing to investigate the issue. We have narrowed the scope of the troubleshooting and expect confirmation of the root cause shortly.
We are continuing to investigate the failure. Intermittently, some phone interactions may result in dead air. We have isolated the fault zone and are working to restore service as soon as possible.,
We are currently investigating this issue.
Report: "Agents experiencing issues handling interactions within Sharpen Q"
Last updateThis incident is fully resolved.
We are now seeing some inbound email interactions routing. They appear to be processing through a backlog. Realtime delivery is not yet functioning as expected. We will update this page when realtime delivery is confirmed.
We are continuing to see sustained recovery for most Sharpen Q operations. Inbound email continues to be impacted. Outbound email works as expected. We will continue to work towards full recovery.
A fix has been introduced by AWS and we're seeing initial recovery. We are validating full recovery and will post another update when we are confident all service has been restored.
AWS investigation continues with a confirmed root cause, but recovery is not yet established. We are continuing our pursuit of resolution with AWS.
A path to service restoration has been confirmed by AWS and recovery processes are underway. We continue to be in direct contact with our representatives at AWS and will update with a timeline as they provide more details.
AWS has confirmed that the root cause for this event has been found. Corrective measures are being discussed and we expect an ETA on service restoration shortly. More info to come.
AWS has confirmed an issue with at least the Lamda service. Sharpen Q operations depend on Lamda. As a result, the majority of Sharpen Q interaction related activities are subject to failure. Inbound IVR calls, ring groups, external transfers, and calls to and from Connect licensed users are not impacted by this service disruption. These functions may be used as workarounds as this investigation continues. We continue to have all necessary resources engaged and are pursuing all feasible recovery options.
Agents may be experiencing several varying behaviors within Sharpen Q. We have the fault zone isolated in correlation to events within the AWS indicating a likely failure point within the AWS infrastructure. We are engaged with all necessary Sharpen and AWS resources to restore service as soon as possible.
Agents may be experiencing several varying behaviors within Sharpen Q. This behavior is currently under investigation. We'll post an update shortly on our findings.
Report: "Sharpen Q Disruption"
Last updateThis issue is resolved
Additional validation testing has confirmed that all Inbound and Outbound call routing functionality has been restored throughout the application (In our IZ1-specific infrastructure that was seeing impact). Please reach out to the Care team if you're continuing to see impact for certain end-users.
A service/sub-system contention issue was identified, and corrective actions were taken by our Dev Operations team to restore call routing functionality in the application. We're continuing to monitor the situation and perform additional validation tests.
We are aware of a potential service disruption to our IZ1 isolation zone. We have all necessary resources engaged to restore service. Additional details will be provided soon.
Report: "Sharpen Q Service Disruption"
Last updateThis issue is fully resolved.
This issue is now corrected. We will continuing to monitor for the next 30 minutes to confirm sustained recovery.
We have isolated the problem and have identified steps to correct. We are working through steps to correct now.
We are aware of a service disruption in our IZ0 isolation zone (app.sharpencx.com). We have all necessary resources engaged to restore service. Additional details will be provided soon.
Report: "IZ1 Sharpen Q Interruption"
Last updateService has been fully restored and sustained.
Since 11:33 ET, we have observed stability but will continue to montitor for sustained recovery. If agents are continuing to experience issues, it is recommended to logout and back in.
We have implemented a change to restore service. Telemetry shows partial recovery, but we are continuing efforts to fully restore service.
We are currently aware of an issue intermittently impacting some users of Sharpen Q in the IZ1 isolation zone (app.iz1.sharpen.cx). We have isolated the issue and all necessary resources are engaged to restore service as soon as possible.
Report: ".cx project functionality impairment"
Last updateAt this time the final remaining impacted .cx projects have been restored to full functionality. We will remain in monitoring until 10:30 a.m. EST and resolve this event at that time. As always please reach out to the SharpenCare team with any lingering issues for further investigation.
At this time the final remaining impacted .cx projects have been restored to full functionality. We will remain in monitoring until 10:30 a.m. EST and resolve this event at that time. As always please reach out to the SharpenCare team with any lingering issues for further investigation.
The cause of the behavior has been isolated and recovery is in progress. If you are still experiencing behavior issues please reach out to the SharpenCare team and provide the impact details for us to review.
We are investigating an ongoing issue with .cx functionality and have a lead on the cause of the behavior. Restoration efforts are in progress and we will have an update on estimated time for restoration shortly.
Report: "Potential Service Disruption"
Last updateThis event has been resolved.
Sharpen's monitoring caught an event with an app host falling into a bad state. Agents in the IZ0 isolation zone may have experienced issues placing outbound calls from Sharpen Q causing them to fall into a stuck loading state on the outbound interaction. If agents are still stuck in this state a logout and subsequent log back in will recover their status. Service has been restored to normal operation however if you have agents that continue to experience any impact please reach out to the SharpenCare team for assistance.
We are aware of a potential service disruption in our IZ0 isolation zone. We have all necessary resources engaged to restore service. Additional details will be provided soon.
Report: "Routine Incident Messaging Test"
Last updateThis is a routine test of the Sharpen incident notification mechanism. All systems are operational. This test is complete
This is a routine test of the Sharpen incident notification mechanism. All systems are operational. Please disregard the message.
Report: "Sharpen Q (IZ1) Service Disruption"
Last updateThis issue is resolved.
We have identified the failure point and performed initial recovery steps. Test interactions, both inbound and outbound from Sharpen Q, are working as expected.
We have continued our path to narrowing in on the issue, but have no identified the cause. All necessary resources continue to be engaged until this is resolved.
We have narrowed in on the fault zone of the issue and are working to identify the source of the issue. All necessary resources continue to be engaged until this is resolved.
We are aware of a service disruption to our IZ1 (app.iz1.sharpen.cx) environment. We have all necessary resources engaged to restore service. Additional details will be provided soon.
We are aware of a potential service disruption. We have all necessary resources engaged to restore service. Additional details will be provided soon.
Report: "Agents working within Sharpen Q may experience unexpected impact"
Last updateThis incident is confirmed to be resolved.
We are currently seeing recovery from impact in our testing however we have not confirmed the root cause of the event. Invesigation continues into causation and next steps for future avoidance.
We have received reports from customers in our IZ0 isolation zone indicating users working within Sharpen Q could experience issues with calls disconnecting, extended ringing events, unexpected status changes, becoming stuck in Alerting or calls not routing to agents in a timely manner. We are investigating these reports and will update as soon as possible.
Report: "Outbound calls intermittently failing/disconnecting unexpectedly"
Last updateFurther monitoring has shown sustained system health and full return to operability. If further behavior is experienced please reach out to the SharpenCare team for assistance.
The cause of this behavior has been identified and corrected. We are continuing to monitor for any additional issues.
In both isolation zones calls placed outbound may drop unexpectedly or fail to load/connect.