Historical record of incidents for Pixlee
Report: "Elevated Response Times and Error Rates"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are aware of and currently investigating increased error rates and elevated response times when doing inbox search.
Report: "UGC content not loading"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
The system has returned to an operational state for the vast majority customers.
We have identified the issue and are currently working on a remediation. We are continuing to see elevated error rates and response times.
We're aware that some UGC features as widgets, API requests and some content are not loading, currently investigating the situation.
Report: "UGC features not loading"
Last updateThis incident has been resolved.
A fix has been implemented and we're currently monitoring the performance.
The issue was identified and a fix is being implemented.
We're aware that some UGC features as widgets, API requests and some content are not loading, currently investigating the situation.
Report: "Elevated Response Times and Error Rates"
Last updateThis incident has been resolved.
We are aware of and currently investigating increased error rates and elevated response times
Report: "Some albums and widgets are not loading"
Last updateAlbums and widgets have returned to its normal state as of around December 4th, 20:30 UTC.
Albums and widgets have returned to its normal state as of around December 4th, 20:30 UTC.
The issue has been identified and the team is working to implement a fix.
Some users are experiencing errors when loading album contents and the associated widget. We are currently investigating this issue.
Report: "Slowness across the application"
Last updateWe've identified the cause of the issue. Application performance has now returned to regular levels, and we will continue monitoring to ensure it stays that way.
There is an ongoing issue causing degraded performance in the dashboard and API. We've been alerted and are currently working to address it.
Report: "Gallery display issues"
Last updateEarlier today we deployed an internal change intended to improve our internal monitoring. No production-facing impact was anticipated or detected during development, as this change was internal. An unexpected interaction with our production configuration resulted in gallery CSS not loading for some customers, which caused their galleries to be visually broken. Our technical team was made aware of this issue at 11:30AM Eastern Time, and identified the cause at 11:39AM Eastern Time. A fix was live by 11:58AM Eastern Time. We are reviewing this deployment, including our processes and controls, to ensure similar issues cannot recur in future. We're very sorry for the trouble.
Report: "Slowness retrieving photos"
Last updateWe have resolved the performance issue and are seeing photos loading in albums now, as well seeing a large drop in HTTP response errors.
We are continuing to work on a fix for this issue.
We are currently investigating an issue with photos not appearing in albums. We first identified this around 7:00AM EST.
Report: "Widget analytics data collection showing lower-than-expected data"
Last updateWe deployed some backend changes for our website displays around 19:30UTC on February 13th. Unfortunately, these changes prevented some display analytics events from being collected – most notably daily widget views and daily navigation clicks. Lightbox-related analytics (such as lightbox opens and call-to-action link clicks) were unaffected and continued to be collected. We were made aware of this issue on the 27th February, and deployed a fix the same day. However, analytics reports for February will show lower-than-expected widget views as a result of the data gap. We're very sorry for this mistake, and are taking steps to prevent a similar issue from recurring in future.
Report: "Slow loading times in albums"
Last updateAlbum performance has returned to its normal state as of around December 22nd, 19:00 UTC.
Some users with large albums may see delays or timeouts when loading album content. We're currently investigating this issue.
Report: "Instagram comment visibility for some clients"
Last updateAfter conducting a thorough review, we have determined that the cause of some clients’ comments getting hidden stems from Instagram itself and is unrelated to our platform. Should the problem continue, we suggest reaching out directly to your Meta representative to resolve it on Instagram’s end. Please feel free to include your Customer Success Manager in the correspondence if you would like our help providing supporting details to Meta. Rest assured we remain committed to delivering an optimal experience through our system. We apologize for any inconvenience caused by third-party filters beyond our control but will aid however possible in expediting a resolution. Please reach out with any other questions.
We're aware that some clients are experiencing problems with the visibility of their comments on Instagram. Our team believes this is connected to Instagram's spam prevention measures, such as throttling of comments. We understand the importance of Instagram commenting and are working with Instagram to better understand the source of the problem. We will keep you updated as we learn more. If you have any questions, please reach out to support or your Customer Success Manager.
Report: "X (Twitter) collection temporarily unavailable"
Last updateTwitter (X) functionality has completely been restored and upgraded to v2 API. To ensure that your Twitter (X) functionality is restored, please log out and log back in to your Twitter (X) account in Pixlee if you have not logged in within the past 2 weeks. Thank you all for your patience.
Our code is deployed and ready, and we are awaiting the proper access tokens for X (Twitter) so that we can resume collecting and permissioning tweets.
We will be deploying new authentication code for Twitter v2 this week. When we are ready, we will instruct customers to log back in to Twitter to enable v2 collection.
In the progress of deprecating v1.1 endpoints, X (Twitter) has unexpectedly removed endpoints related to content collection among others ahead of their reported schedule. As a result, we are currently unable to collect X (Twitter) content. Our upgrade to v2 migration is coming very shortly, and we will be announcing the information as soon as we have concrete timelines. See schedule for reference: https://twittercommunity.com/t/x-api-v2-migration/203391
Report: "Meta (Facebook) Logins temporarily degraded"
Last updatePermissions have been approved and facebook is back to normal. If you have never logged in to Facebook on Pixlee or have only newly logged in the last 7 days, please log out and log back in to resume functionality.
We're in back and forth app review with Meta and hoping to get a resolution by the end of the week at the latest. Thanks all for your patience.
Meta now requires a new permission (business_management) in order to access Instagram Business Accounts. This was an unexpected change, and as a result we are in the process of obtaining the new permission, which we expect to do so very shortly. In the meantime, logging in to Meta (Facebook) in social logins will not provide adequate permissions needed to retrieve Facebook and Instagram content. We will update as soon as there is more information.
Report: "Facebook / IG collection degraded"
Last updateIt appears the issue has been rectified and collections and permissions are back to normal.
Meta APIs are currently experiencing an issue in which certain content is incorrectly being marked as not collectable due to permissions. This is a pervasive issue that Meta is currently working to resolve. This can affect the collection and permissioning of Facebook and IG content. More information can be found here. https://developers.facebook.com/support/bugs/296153916339672/?join_id=fce9caa2d495a We will update when we receive further updates from Meta.
Report: "Missing permalinks from some Instagram posts"
Last updateNo further issues have been detected, although some subset of collected content during this section may not have their links populated until they are re-@ mentioned.
Meta reports that this problem is fixed. Our team is evaluating Facebook API responses to confirm the fix and the feasibility of backfilling any newly-available permalinks. All freshly-collected content should have the expected permalinks.
Due to an upstream Facebook API bug, a small number of collected Instagram posts are missing permalinks back to Instagram. As a result, permission responses may not be detected on those photos. We are awaiting Meta's resolution of this problem, which is tracked here: https://developers.facebook.com/support/bugs/1225579374820629/
Report: "Database maintenance"
Last updateTransition is complete and control panel access is regranted
Database performance is improving, but we won't be releasing control panel out of maintenance mode until we've confirmed that it is performing optimally
DB maintenance is taking longer than expected, will continue to update as progress is made.
Report: "Widgets down due to database updates"
Last updateToday around 3AM EDT, to 5:45 AM EDT, our widgets experienced downtime. This was eventually traced to a one-off, mandatory update within our backend Amazon powered data stores, which caused some routing issues to our distributed servers. Engineers eventually rectified this issue and confirmed that traffic resumed normally. We apologize for the inconvenience.
Report: "Pixlee Assistant inconsistencies identified"
Last updateThis incident has been confirmed as resolved.
We have released v0.40 of the Pixlee Assistant and re-enabled assistant permissioning in the Pixlee Dashboard. Internal testing indicates this version potentially resolves the reported issues. We're working with customers to verify the fix. Thank you for your patience.
We have identified reports of some customers having issues with the Pixlee Assistant. While we pinpoint this issue, we will be shutting off the Pixlee Assistant and switching customers to manual permissioning, to avoid confusion in the meantime. Thank you for your patience.
Report: "Pixlee Assistant not posting comments"
Last updateThe Pixlee Assistant chrome extension has been updated and will now work properly. Please restart your chrome browser to ensure updates to the extensions.
We have identified the issue affecting the Pixlee Assistant. We’re continuing to work on a fix in response to some changes made by our partners at Instagram, and will update when we have a clearer timeline for resolution.
We’ve identified an issue with the Pixlee Assistant not being able to post comments on Instagram posts. On Monday, December 26, 2022, we will put the Assistant into maintenance mode as we work on a fix. Permisisoning will not be available through the assistant at the time. We apologize for the inconvenience and will share an update as soon as the issue has been resolved.
Report: "Pixlee Assistant Chrome extension outage"
Last updateThe Pixlee Assistant chrome extension has been updated and will now work properly. Please restart your chrome browser to ensure updates to the extensions.
We have submitted an update to the Google app store and are awaiting review
We are continuing to investigate this issue, and are working towards a fix.
We are aware of an issue that stops the Pixlee Assistant leaving comments, and are working to fix it.
Report: "Issue with Heroku causing API degradation"
Last updateThe upstream provider has returned to normal responsiveness.
We are noticing issues with our upstream provider Heroku and is causing issue with our API and widgets. We are in communication and will update as soon as we can
Report: "Data inconsistencies in upstream analytics provider"
Last updateThe incident has been resolved and analytics values have been fully restored. Thank you all for your patience.
We note that the issue primarily affects Interaction analytics We're working actively with our upstream provider who is making progress. Will update on next milestone
Our upstream data provider is experiencing some data inconsistency https://status.keen.io/incidents/rm4cmynj61fw And this can affect numbers in analytics charts We're monitoring the situation and will update when we receive new information
Report: "Issue with redis provider"
Last updateThe upstream issue has been resolved.
Our upstream redis provider is having some status issues https://status.redis.com/ and we are currently affected. This may mean that there is degraded performance of widgets and the dashboard. We are actively monitoring the situation and will provide updates as they come.
Report: "Degraded Analytics performance"
Last updateThe incident has been resolved and analytics have returned to full operation.
We're aware that some Analytics features are not loading data, due to an upstream service provider outage. We're monitoring the situation, but expect it to be resolved shortly.
Report: "Email service impacted by heroku outage"
Last updateEmail appears to be back, we will be monitoring for further incident.
The email widget service is currently impacted by a Heroku outage. We are tracking and communicating with the heroku team.
Report: "Cannot approve photos on Pixlee Dashboard"
Last updateThe fix for this issue has been deployed and verified, so this incident is now resolved.
We are aware of the issue in which users cannot approve photos on the dashboard. We have pushed a fix and will inform everyone when the fix is deployed.
Report: "Displays outage"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
Thank you for your patience. At 8:35am pacific, the issue was resolved, and had to do with server scaling ratios which we have properly adjusted.
The service has been partially restored, and we are monitoring for further issues
We’re aware of some issues with displays and investigating
Report: "Pixlee Assistant Not Working"
Last updateWe have released a new version of the Pixlee Assistant which resolves this issue. Version 0.35 restores all Assistant functionality. Thank you for your patience!
We’re aware of reports that the Pixlee Assistant isn’t working. We’re investigating the problem.
Report: "Tiktok Collections halted"
Last updateWe've confirmed that TikTok collection has resumed, so this incident is resolved.
The upstream errors appear to have been resolved. Tiktok collection will resume on Pixlee shortly. We'll update this incident when we've confirmed it's fully resolved.
Due to some upstream errors with our TikTok integration, We are currently unable to collect content from TikTok. We are investigating the issue in collaboration with TikTok, and will post updates as they arise.
Report: "Pixlee Assistant investigations"
Last updateThe Pixlee Assistant (0.34) has been confirmed to be functioning with the latest Instagram changes. Please ensure your Pixlee Assistant Chrome extension is updated to the latest version!
A new release of Pixlee Assistant (0.34) has been published and we are testing to ensure that it has resolved the problem.
We are aware of compatibility issues with Instagram and the Pixlee Assistant at the moment and are in the process of updating the Assistant.
Report: "Issues with Email Uploader"
Last updateEmail uploader issues have been resolved.
We are currently investigating issues with our email uploader tool that is causing attached content in Pixlee album emails to not upload.
Report: "Facebook and Instagram Collection Issue"
Last updateAs of 12:18PM EST Thu Mar 3, we were informed by Meta/Facebook that our Verification had passed. We have assessed the access tokens of all users and confirmed that all previous permissions have been restored for Facebook logins pre-outage. However, if you logged at some point during the duration of the outage, you will need to re-login to retain permissions. All Instagram and Creator accounts have been restored to full functionality. At this time, non-mention FB/IG collections are resuming, and will automatically cover the time since collection was halted. Unfortunately, there is no way to recover @mention IG content, or IG hashtagged content older than 1 week, during this time period. We offer our sincere thanks to everyone who endured this process with us. We understand it was not easy. We will continue to monitor progress, and work with the Meta team to ensure that such interruption does not reoccur.
Our Facebook verification has been restored. We’re working to restore Facebook and Instagram functionality, and verify stability of our systems.
Our original contacts are still awaiting inquiry RE: the business verification results. We've simultaneously added further contacts who are aiding us in investigating the issue. Moreover, we've begun to receive confirmation that other brands and organizations have encountered the same verification issues with their own Facebook applications, which strengthens our suspicion that this is a logistical Meta issue. We'll check back in with further results as we receive them.
We have submitted a host of new verification details to the Facebook team and are awaiting confirmation.
We're continuing conversations with our Facebook partners to navigate the business verification details.
As of this morning, from our latest communication with the business verification team. we are in process of submitting additional documents for the verification protocol.
We are now engaged in active communication with the business verification team. We're working through some interface and logistics challenges to ensure that the right information is returned to the FB BV team.
We are continuing to awaiting for Facebook verification. We have made contact with further Facebook contacts who are aiding us in expediting the process.
We are still awaiting Facebook completion of Business Verification at this time.
We are still awaiting completion of re-verification by Facebook.
We are in contact with our Facebook representatives who are helping the verification process along. Thank you for your continued patience.
Thank you everyone for your patience. As it's been a long weekend, Facebook offices have taken a bit longer with the verification efforts. According to Facebook, it can take a few business days to get to. We have full confidence that this will be resolved expediently, and again, we'll update as soon as we have new information.
We are still awaiting the Facebook team's verification review. We will inform everyone ASAP when review is completed and functionality is restored.
API calls to Facebook and Instagram have been malfunctioning since approximately 5AM EST on February 18th. Our team investigated and identified that our API access has been limited by Facebook due to business verification updates. Unfortunately, Facebook did not notify us in advance about any required changes. We’ve already updated our verification, and are awaiting Facebook to process it. In the meantime, Facebook and Instagram collections are halted, and Instagram Business Accounts will be not be visible in social settings. Once Facebook has updated its verification records, all permissions for collection and other functionality will be restored. We’re working with our Facebook contacts to ensure that this happens as quickly as possible. Thank you for your patience, and we’re sorry for the trouble.
Report: "Heroku issues causing email widget downtime"
Last updateWe are seeing improved results across all of our widgets, as Heroku has mentioned in their reports as well.
Heroku has acknowledged the issues and are working on a fix. We will update when any changes occur.
Issues with our upstream provider Heroku has affected widgets and API and may cause degraded performance. We've notified Heroku of such issues and are awaiting reply.
Report: "Facebook and Instagram Collection Issue"
Last updateWe are pleased to announce that our last remaining permission, - Submitting Facebook videos via widget uploaders Has been restored. Thanks to everyone for their patience and cooperation in these times.
We are almost fully operational again, the only capability remaining is - Submitting Facebook videos via widget uploaders A reminder that to obtain the latest updates, which include restoration of - Collecting @mentioned content - Collecting photo tagged content - Permissioning Instagram Content - Scheduling content to Facebook Pages - Sending content to Facebook Ad accounts You must have re-authenticated with Facebook AFTER Nov 29 12:34PM Eastern. Reauthentication instructions: 1) Navigate to https://app.pixlee.com/app#settings/social_profiles 2) Log out of all Facebook accounts 3) For each Facebook account, go to https://www.facebook.com/settings?tab=business_tools&ref=settings 4) If there is a “Pixlee Auth” app, remove it. You may not see it if you are doing this for the first time. 5) Go back to Pixlee, log back into the same Facebook accounts while making sure to select the desired Facebook page(s) and Instagram business account(s) that you will be connecting and ensure that all permissions are accepted. That’s it. Each account that is logged back in will regain access. For more information on the login process, please see: https://pixleeteam.zendesk.com/hc/en-us/articles/4410186266263-2021-10-19-Update-Instagram-Facebook-Login-Flow
We are continuing to resolve the last 2 permissions: - Submitting Facebook videos via widget uploaders - Sending content to Facebook Ad accounts A reminder that to obtain the latest updates, which include restoration of - Collecting @mentioned content - Collecting photo tagged content - Permissioning Instagram Content - Scheduling content to Facebook Pages You must have re-authenticated with Facebook AFTER Nov 29 12:34PM Eastern. Reauthentication instructions: 1) Navigate to https://app.pixlee.com/app#settings/social_profiles 2) Log out of all Facebook accounts 3) For each Facebook account, go to https://www.facebook.com/settings?tab=business_tools&ref=settings 4) If there is a “Pixlee Auth” app, remove it. You may not see it if you are doing this for the first time. 5) Go back to Pixlee, log back into the same Facebook accounts while making sure to select the desired Facebook page(s) and Instagram business account(s) that you will be connecting and ensure that all permissions are accepted. That’s it. Each account that is logged back in will regain access. For more information on the login process, please see: https://pixleeteam.zendesk.com/hc/en-us/articles/4410186266263-2021-10-19-Update-Instagram-Facebook-Login-Flow
We are pleased to announce that at this moment, the following functionality has been restored: - Collecting @mentioned content - Collecting photo tagged content - Permissioning Instagram Content - Scheduling content to Facebook Pages We are working on obtaining the final missing components: - Submitting Facebook videos via widget uploaders - Sending content to Facebook Ad accounts In order to obtain such restored functionality, a re-authorization is necessary. 1) Navigate to https://app.pixlee.com/app#settings/social_profiles 2) Log out of all Facebook accounts 3) For each Facebook account, go to https://www.facebook.com/settings?tab=business_tools&ref=settings 4) If there is a “Pixlee Auth” app, remove it. You may not see it if you are doing this for the first time. 5) Go back to Pixlee, log back into the same Facebook accounts while making sure to select the desired Facebook page(s) and Instagram business account(s) that you will be connecting and ensure that all permissions are accepted. That’s it. Each account that is logged back in will regain access. For more information on the login process, please see: https://pixleeteam.zendesk.com/hc/en-us/articles/4410186266263-2021-10-19-Update-Instagram-Facebook-Login-Flow Please note that all Creators who have signed in via our Creator platform (https://pixlee.me/) will also want to re-authenticate to continue their connection with Pixlee. They can follow these instructions: https://www.pixlee.com/pixlee-for-creators-faq We appreciated everyone's patience on this matter and we wish everyone a happy Thanksgiving weekend!
After working with Facebook, we have an update for all Pixlee customers. As of this moment, a segment of Facebook and Instagram functionality have been restored. To finalize access to functionality, you will want to: 1) Navigate to https://app.pixlee.com/app#settings/social_profiles 2) Log out of all Facebook accounts 3) For each Facebook account, go to https://www.facebook.com/settings?tab=business_tools&ref=settings 4) If there is a “Pixlee Auth” app, remove it. You may not see it if you are doing this for the first time. 5) Go back to Pixlee, log back into the same Facebook accounts while making sure to select the desired Facebook page(s) and Instagram business account(s) that you will be connecting and ensure that all permissions are accepted. That’s it. Each account that is logged back in will regain access. For more information on the login process, please see: https://pixleeteam.zendesk.com/hc/en-us/articles/4410186266263-2021-10-19-Update-Instagram-Facebook-Login-Flow What functionality has been restored? - Hashtagged IG content, IG own-user content and Facebook page collections have been restored. We will retroactively collect all of the above. - All forms of content scheduling to Instagram - Permissioning of Facebook content - Scheduling of content to Instagram - Submitting Facebook photos via widget uploaders What functionality has not yet been restored? - Collecting @mentioned content - Collecting photo tagged content - Permissioning Instagram Content - Submitting Facebook videos via widget uploaders - Sending content to Facebook Ad accounts - Scheduling content to Facebook Pages We are continuing to work with Facebook to restore these remaining features, and expect this to be done within the week. Once such features are restored we will send another message to all customers. To regain such access, customers will simply need to reauthenticate their Facebook accounts again. Please note that all Creators who have signed in via our Creator platform (https://pixlee.me/) will also want to re-authenticate to continue their connection with Pixlee. They can follow these instructions: https://www.pixlee.com/pixlee-for-creators-faq Thank you all for your continued patience.
We have made significant headway in congruence with Facebook, and have strong confidence in functionality restoration this week. We'll have more news tomorrow.
Facebook has informed us that they are continuing to dive into the issue and should have an update for us within 24 hours. Thanks everyone for your continued patience.
Our partners at Facebook have identified the issue and are working within their teams to resolve. When there is an update, we will update here accordingly. We appreciate everyone's patience and understanding.
We are continuing to work with facebook to resolve FB/IG related app issues. We thank you for your patience.
We have identified an issue with our Facebook / Instagram authentication app that is causing collection, permissioning and scheduling to be temporarily unavailable. We are actively working with the Facebook team to remediate.
Report: "Dashboard down Due to Heroku and AWS outages"
Last updateAs AWS is slowly recovering, we are seeing full operation across all of our systems. We will mark this resolved and monitor the situation if needed.
We are continuing to work on a fix for this issue.
Our upstream providers AWS: https://status.aws.amazon.com/ and Heroku: https://status.heroku.com/ Have identified issues today that have rippled and have caused Pixlee Dashboards to be temporarily unavailable for small measures of time. Widgets and API are so far unaffected. We are following along to the resolutions of the above status issues and will update as soon as we have any changes.
Report: "Permissioning issues"
Last updateWe have resolved issues with permissioning, thank you for your patience.
We are aware of some issues that may occur when using the platform to permission Instagram content. We're working on it and will update everyone when we have an update.
Report: "Slower-than-usual Response times"
Last updateOur monitoring detected slow backend response times, which were also reported by customers. Some areas of the control panel did not load data, and galleries on your website may not have appeared. This is now fully resolved.
Report: "Heroku issues causing widget downtime"
Last updateWe are seeing a resolution of previous heroku issues and our widgets are now back up.
We are continuing to await updates from Heroku
Heroku, our third party provider, has reported issues that are causing issues with our photos widget: https://status.heroku.com/incidents/2362 We are monitoring and will update accordingly.
Report: "Product tagging degraded"
Last updateWe have resolved this issue with our third party provider and all is back to normal.
An issue with a third party service is affecting our product tagging service and it is currently degraded. We are working with that service for updates and will update ASAP.
Report: "Pixlee Dashboard down"
Last updateThe issue has been resolved and we apologize for the inconvenience.
We have resolved an issue with rate limiting on a third party provider and are monitoring to ensure it is resolved
We are investigating an issue with the pixlee dashboard
Report: "Pixlee Assistant investigations"
Last updateWe have released the new version of Pixlee Assistant which resolves this issue https://chrome.google.com/webstore/detail/pixlee-assistant/dimboijfhompnhhpaglbaagdhefimipk?hl=en&authuser=1
We have implemented a fix and submitted to the chrome app store.
We are investigating an issue with Pixlee Assistant which is causing it not to permission Instagram content properly
Report: "Instagram Follower Growth counts"
Last updateWe have ensured that changes in follower counts are now being tracked daily.
Late last week, we identified an issue with the Insights endpoint of the Instagram Graph API that had been in effect since mid December 2020. This affected the way daily changes in follower counts were reported in the insights API. As a result, our Instagram Analytics graphs did not properly report the daily gain/losses in followers correctly (total follower count changes were unaffected) Changes were made late last week to rectify this situation for the past 30 days (as far back as the IG API allows) and moving forward. We will complete such changes this week and update this issue.
Report: "Fastly CDN down"
Last updateWhile we are continuing to monitor https://status.fastly.com/incidents/vpk0ssybt3bj, our services seem to have returned to normal
Our services seem back up, but we will continue to monitor fastly's status here: https://status.fastly.com/incidents/vpk0ssybt3bj
Our CDN provider, Fastly, is experiencing a major outage. This affects a massive chunk of the internet. We will continue to monitor and deliver updates as the situation progresses.
Report: "Slowness with loading albums in Control Panel"
Last updateLatency issues have subsided.
We have upgraded to stronger dynos in the meantime to reduce latency of album / product requests
We are monitoring this issue https://status.heroku.com/incidents/2232 as we believe it has something to do with this current issue
We have identified a slowness in loading albums in control panel and are looking into it.
Report: "Email Widgets Momentarily Unresponsive"
Last updateOn Apr 27 2021 from 9:34am to 11:28am Eastern time, Pixlee email widgets did not display images when loaded. This was due to a TLS issue that we rectified immediately upon discovery. We have since added additional monitoring to prevent such cases in the future. Apologize for the inconvenience.
Report: "Instagram API failing to return content"
Last updateFacebook has declared the issue resolved. https://developers.facebook.com/status/issues/471343314132946/ While stories and mentions missed in this period cannot be recollected, other collections like hashtags, user feeds and photo tags will be retroactively collected.
There have been multiple issues filed to Facebook since last week and the Facebook team is looking into the situation
Since around 9PM Eastern time Friday April 23, the Instagram API has failed to return valid content from webhooks, hashtag collections, user collections, and photo tag collections. This appears to be an issue with the IG/FB API. We have filed a high priority issue with FB and have confirmed that others are seeing the issue as well. Will keep everyone up to date on this ongoing issue as we receive a response.
Report: "Widgets downtime"
Last updateOn Apr 12, 11:20 PDT, a faulty deploy was delivered which caused Pixlee email and web widgets to become inoperational and hide themselves from site embeds. Diagnostics were made as to determine the root of the issue, and the issue was resolved on 13:18 PDT. We are implementing further guardrails to ensure this issue does not reoccur. We apologize for the inconvenience.
Report: "Delayed Analytics Reporting"
Last updateReports queue is complete and we are back to normal. Thank you for your patience.
Optimizations have been made today to 1) Expedite reporting 2) Prevent live analytics from being disturbed while reports are being generated We are still running through reporting backlog and will resolve this issue when complete
Given the new month and an unusual amount of analytics reports being generated, we are experiencing slowdowns within our analytics platform. We're working with our data provider to get these reports to you as soon as we can.
Report: "Pixlee Dashboard outage"
Last updateThis incident has been resolved.
We have fixed the issue, and all systems have returned to normal operation.
We are currently investigating the issue.
Report: "Dashboard analytics down"
Last updateWe have resolved the issue and are back to normal.
As part of the Amazon outage today (https://status.aws.amazon.com/), our backend analytics display systems have been affected. We currently cannot display analytics within the Pixlee dashboard, but Amazon is well on their way to repairing the issue. We will check in periodically on the issue as well. Widgets are unaffected.