Historical record of incidents for AlsoEnergy
Report: "Ingest Delay in Power Track US"
Last updateWe are continuing to monitor for any further issues.
The issue has been identified and resolved. We will continue to monitoring this until all device data has been backfilled.
We are currently investigating this issue.
Report: "Ingest Delay in Power Track US"
Last updateWe are continuing to monitor for any further issues.
The issue has been identified and resolved. We will continue to monitoring this until all device data has been backfilled.
We are currently investigating this issue.
Report: "PowerTrack may be experiencing loading issues."
Last updateThe scheduled maintenance has been completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
PowerTrack may be experiencing loading issues.
Report: "Ingest Delay"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix has been implemented.
We are currently investigating this issue.
Report: "Ingest Delay"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix has been implemented.
We are currently investigating this issue.
Report: "Delay in ingest."
Last updateThe issue has been identified and a fix has been implemented.
There is an ingest issue that is causing a delay in processing which is affecting a subset of sites. We are working to resolve the ingest issue.
Report: "Delay in ingest."
Last updateThe issue has been identified and a fix has been implemented.
There is an ingest issue that is causing a delay in processing which is affecting a subset of sites. We are working to resolve the ingest issue.
Report: "Delay in solaredge api ingests"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Delay in Ingestion"
Last updateThis incident has been resolved.
There is an ingest issue that is causing a delay in processing which is affecting a subset of sites. We are working to resolve the ingest issue
Report: "Delays in processing."
Last updateThis incident has been resolved.
There is an ingest issue that is causing a delay in processing which is affecting a subset of sites. We are working to resolve the ingest issue.
There is an ingest issue that is causing a delay in processing which is affecting a subset of sites. We are working to resolve the ingest issue.
Report: "PTWeb Reporting degraded performance"
Last updateAll reporting issues have been resolved.
Report Queue is currently very high. Any user who tries to generate a report will have the request pending in a queue, and it will be completed once the queue is cleared.
Report: "AWS Draker VPN de-graded Health"
Last updateThe tunnels are since bounced and healthy. No customer or application impact noticed.
Currently AWS Draker VPN de-graded Health.
Report: "PowerTrack Maintenance"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
We will be performing some additional maintenance to improve our the performance of our PowerTrack Systems. Start Time: India: 04/01/2025, 09:30:00 GMT+5:30 Eastern: 04/01/2025, 24:00:00 EDT Pacific: 03/31/2025, 21:00:00 PDT
We continue to see degraded performance with some of our services. Customers would experience: - Delays in Cell Modems heartbeats. - Heartbeat alerts - Draker sites may have a delay in ingest along with SMAFTP and custom FTP sites. - Data Backlog and delay in processing - Delays with OPC sites Impacted Systems: - PowerTrackWeb - Draker
Report: "Agency Reports issues"
Last updateThe issue is isolated to PJM and changes to PJM side. Closing this status and will monitor the progress directly with the customer.
We have determined the issue is isolated to one customer due to changes made on the customer side. We are working with the customer to resolve. the issue.
We have observed a decline in the performance of our agency reporting services. Our team is actively engaged and assessing the issue. This impacts: PTWeb Agency Reports
Report: "AWS Draker VPN de-graded Health"
Last updateDraker AWS VPN tunnel started showing un-healthy. The tunnels are since bounced and healthy. No customer or application impact noticed.
Report: "A small number of residential devices not communicating"
Last updateAll known infrastructural issues are resolved. Any additional meters that have issues identified will be addressed through dedicated case tickets.
Following the scheduled maintenance, a small number of customer residential devices are experiencing communication issues with the platform. We are actively collaborating with our third-party partner to investigate the problem and will provide updates as we have more information.
Report: "Ingest Delay on Powertrack EU and US"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "Data ingest delays"
Last updateThis incident has been resolved
We have expanded our infrastructure resources and are actively monitoring system performance. Our primary focus remains on resolving the backlog issue, and we are diligently working towards its resolution.
We are currently experiencing data delays and have identified the issue. We will provide more information as soon as possible.
We encountered an issue with a process that resulted in a backlog of data. The issue has been identified and resolved, and we are currently monitoring the backlog to ensure smooth processing.
We are currently experiencing data delays and have identified the issue. We will provide more information as soon as possible.
Report: "Delays in processing certain sites"
Last updateThis incident has been resolved.
The issue was identified and a fix has been rolled out and put in place. Backfill operations will be a part of the next steps.
We are currently investigating an issue were certain devices are showing behind in reporting
Report: "Delays in processing certain sites"
Last updateThis incident has been resolved.
Fixes were implemented and all sites reducing backlogs. Expect some latency during this phase. We are continuing to monitor as we recover into normal operations.
The scheduled maintenance has been successfully completed. However, please note that there may still be a brief delay in data ingestion for a short period of time. Thank you for your patience
Report: "Ingest Delay on Powertrack EU"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Report: "Issues connecting to the web portal."
Last updateThe issue has not reoccurred, we are closing the incident.
We identified and issue with an application and system is stable, we are continuing to monitor this issue.
We are investigating an issue with connections to the interface. We will update this message once the investigation has been completed.
Report: "EU API Service endpoint"
Last updateClosing this issue after monitoring the services.
A service connector for other services was found to be in a bad state. Once restarted and the supporting service the issue resolved. We are continuing to monitor the services and consider this issue resolved.
We are currently investing an issue with API services on EU environments
Report: "PowerTrack mobile notifications blocked"
Last updateThis has been resolved
An issue has been identified where third party transmission to carrier SMS gateway have been blocked. If possible, Stem advises affected customers update/add an email for monitoring impactful alerts until a fix can be determined and implemened.
Report: "Delays in processing certain sites"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We are seeing delays in data ingests for a small number of sites, we have identified the issue and data is processing and sites are catching up.
Report: "Installing PowerTrack Mobile"
Last updateWe have successfully resolved the authentication issue with Google and the Android OS Version of the app is now available within the play store.
There is an issue with installing the PowerTrack Android application. We are aware and currently working with Google to re-authenticate the Android OS Version of the app
Report: "Enphase fleet reporting delay"
Last updateFleet reporting at 95%
Stable with 81% fleet reporting. We are continuing to monitor for further issues into full recovery.
We have seen fleet reporting now at 54% entering night quiescence and are monitoring for full fleet reporting
The service issue has been identified, fixes implemented, and data ingest requests are being accepted.
We are currently investigating a service interruption with Enphase Energy which has delayed enphase device readings data ingestion.
Report: "Delays in processing certain sites"
Last updateAll services are operating under normal conditions.
The issue has been identified and data streams will be catching up shortly, may take hours to complete.
We are currently investigating a slowdown in processing for a small number of sites.
Report: "CatM fleet reporting delay"
Last updateThe underlying condition was found and resolved on a separate incident. Fleet reporting fully restored. This incident is resolved.
Readings are delayed for <10% of sites. We are monitoring progress towards fleet recovery for return to normal operations.
The issue was isolated to a nonresponsive service on the main server in a cluster for which a restart restored that primary service. We are continuing to investigate/inspect cluster service applications to restore nominal operations.
We are currently investigating an issue with ingest/reporting delays for CatM meters
Report: "FTP service interruption delays ingest for SMA SolarLog ITron CatM fleets"
Last updateThe backlog has cleared for the affected fleets, restoring fleets to operation status. The incident is resolved.
Backlog processing prior to night quiescent period achieved Fleet OK: SMA SolarLog CatM as we continue to monitor backlog and fleet reporting. will review services during morning insolation for progress towards incident resolution.
The service interruption has introduced a backlog in data ingest for the affected fleets. We are experiencing a delay in loading data into database(s) used by our API and LocusNoc for the most recent data. Additional updates as we resolve data delay due to this backlog.
Report: "CatM fleet disconnects"
Last updateFleet reporting has returned to 98%. Squawks open with vendor for recurrence control. This incident is resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "High alert email count"
Last updateThis incident has been resolved.
An issue has been identified with excessive alert emails, specifically backlogged notifications. We are working to restore proper device communication status. We recommend using powertrack to verify device status until the alert notification backlog is cleared.
Report: "Locus email reports"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
An issue has been identified with Reports that are not being received via email after getting generated from LocusNOC. We are investigating and are working to restore functionality.
Report: "Anomalous readings reporting for CatM meters"
Last updateAll systems are in nominal reporting. This incident is resolved.
Anomalous data for devices affected by the incident has been identified and corrected. We continue to monitor the system and are currently stable.
We are now moving forward with data corrections on the affected devices and will be assessing propagation to ensure dependent aggregation reporting is nominal
We have fixed the nonconformance which led to data discrepancy and containment has been verified. We are presently preparing a fix for data correction on the affected devices.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Maps are experiencing and outage due to 3rd party API."
Last updateIssue appears to have cleared we will now close this issue.
The map provider is currently experiencing and API issue on their end. This incident is for visibility to our customers experiencing this issue.
Report: "Enphase fleet reporting delay"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Enphase fleet reporting delay"
Last updateThis incident has been resolved.
A fix has been implemented and data Backfill is in progress
We have identified delayed enphase device readings data ingestion due to a service interruption with this provider. We will provide updates as we resolve this interruption.
Report: "Data ingest delay"
Last updateAn issue with one of our data ingests had a negotiation mismatch and has now been corrected. We will be running backfill operations.
An issue with one of our data ingests had a negotiation mismatch and has now been corrected. We will be running backfill operations.
We are currently investigating data ingest delays for a subset of sites.
Report: "Virtual Irradiance"
Last updateThis incident has been resolved.
We are investigating a delayed ingest into the locus platform for VI data
There was an issue with Virtual Irradiance, the issue has been identified and the data is currently backfilling and should complete by 02/17/24 mid morning MST.
Report: "L-Gate devices Locus"
Last updateFleet reporting is back to nominal daily levels. This incident is resolved.
A fix has been implemented and we are continuing to monitor fleet reporting closely.
Device ingestion has crossed 80% reporting in. We are continuing to monitor the situation closely.
The issue has been identified and the impacted service reverted. We are beginning to observe LGate data flowing back into our systems.
We are currently investigating issues with the L-gate devices, this is not the same issue as experienced in earlier communications issues over the last 2 weeks.
Report: "Enphase fleet reporting delay"
Last updateIssue is now resolved
Service access is restored and we are monitoring results.
We have identified delayed enphase device readings data ingestion due to a service interruption with this provider. We will provide updates as we resolve this interruption.
Report: "Data acquisition and Ingest delay"
Last updateWe are now closing this incident. Thank you for your patience.
We have completed the hardware and network operations for this incident. We are keeping this in monitoring for 24 more hours to allow all changes over the last week to settle across the platform. At the end of this monitoring period we will be closing this incident as resolved. Thank you for your patience.
We made several changes last night and this morning to move traffic to new endpoints. We will have more maintenance windows in the coming days to move more traffic and increase throughput. Until then we will continue to monitor these changes. Most devices should now be up to date with minor delays to portal reporting.
We have made the switchover with our software and hardware updates and are monitoring the issue closely. We will update this message 3/12/24 9:00 AM PST.
We have completed the upgrade and cutover of our network devices. We see an improved infrastructure performance. The system is still processing the data backlog from the original issue, and we expect this to continue through the weekend. We will continue to evaluate the system through and decide on any other further action once the backlog clears.
We are continuing to make progress after doing additional testing and are actively upgrading specific infrastructure with our provider. That action requires swapping of equipment and other configuration information which has been going forward with an urgent level of support over the weekend between Stem and our provider. We will provide daily updates as we go forward.
STEM / Also Energy support and engineering teams have discovered network congestion in one of our data pipelines. This is causing delayed data ingestion for a subset of our fleet. We are working on steps to reduce congestion and increase bandwidth in the coming days. These steps should reduce network latency and will help with timely ingestion of data across our fleet.
STEM / AlsoEnergy support and engineering is diligently working on troubleshooting a delay in data ingestion affecting a subset of sites with Acquisuite loggers uploading data to our monitoring platform. We have identified the issue and are working closely with our 3rd party vendors to resolve it as soon as possible.
We are currently investigating this issue further.
We are continuing to monitor for any further issues after the scale-out undertaken late Friday evening
The underlying service contributing to latency is scheduled for scale out today, and we are continuing to monitor.
Backlogged devices are checking in while in night quiescence for generation. We are continuing to monitor latency reduction.
The issue has been identifed, a workaround implemented, with good initial results for device upload cadence.
We are seeing increase delay in some site data acquisition and ingest into the system . We are currently investigating the affected lagging sites.
Report: "Modem heartbeats"
Last updateWe have resolved this issue, no recurrence of alerts.
We identified a service that was not communicating after a network operation, this issue is resolved however it may take some time for the portal to update. This notification is in resolve of modem heartbeat alerts. We will continue to monitor this over the next 8 hours. Thank you for your patience during this time.
We are currently investigating an issue with modem heartbeats for a subset of devices.
Report: "LocusNOC Reporting anomalies"
Last updateWe have reviewed customer account data for issues of data latency, energy adjustments, and aggregate reporting and can ensure that the platform operation is nominal. The corrective actions taken were successful, and this issue is resolved. We will continue to work with individual customer readings reporting concerns as required.
The same backend service had a recurrence on 2-Sep, and required a redeployment, as it had blocked other service schedules. Synchronization completed on 4-Sep. We are continuing to monitor for recurrence.
The fix was implemented and anomalies within reports are now providing expected results. We are continuing to monitor for any energy report issues
A backend service issue has been identified and a fix implemented. Synchronization is underway, and we will update upon completion.
We are experiencing an issue with the LocusNOC Reporting feature. Canned reports (Historical Energy Report, Interval Energy Report, etc) are returning incorrect results. We are investigating this incident and will provide updates as we identify the issues and work towards resolution.
Report: "DECK sites not showing uploads in portal."
Last updateAll data has been backfilled and systems are normal.
Deck - There was an issue with a running service that has been corrected and data is now backfilling. We will continue to monitor and close this status once backfill operations have completed.
We are investigating an issue with DECK services, initial investigation.
Report: "Data delay for sites in locusnoc"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified another delay in readings ingestion, with update interruption commencing between 10am-11am MT today, which we are investigating.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "reporting delays for subset of devices"
Last updateThis incident has been resolved.
We are continuing to investigate this issue. Legacy deck processing was found to be in a non-optimal state, and remedial actions were performed to return to nominal operations.
We are currently investigating a delay within data acquisition for acquisuite/obvius devices, where some devices have a lag over 12 hours.
Report: "Alerts investigation"
Last updateWe have monitored the changes for 20 hours with no addition issues, I will now close this status alert.
We have identified an issue preventing the display of alerts on the website. A fix is currently being deployed. We will continue to monitor this issue and update messaging as needed.
We are currently investigating an alerting issue in our AE environment We will update this messaging once we identify the issue.
Report: "Portfolio calculated power value anomaly"
Last updateThis incident has been resolved.
Confirmed that most recent statistical calculations are reflecting expected power values. We are continuing to monitor for further issues as sites/portfolios are refreshed.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Data delays in a small subset of sites"
Last updateCustomers utilizing the software solution that wished to continue with the solution were given assistance to reconfigure SSL/TLS. This incident is resolved.
We are currently investigating this issue.
Report: "Solectria site data unavailable"
Last updateThis incident has been resolved.
Solectria Fleet is reporting at nominal levels. We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
An investigation identified data retrieval from Solectria is unavailable and we are awaiting a response from Solectria.
Report: "Degraded data acquisition for LGate LTE (and some CatM) meters"
Last updateThis incident has been resolved.
Fleets are reporting at nominal levels. We are continuing to monitor for any further issues.
Reporting achieved 82.3% of nominal (expected 87.8%) entering the night quiescence period. We be continuing to monitor progress on Thursday.
We are presently running at 73.2% of nominal as the backlog continues thru data ingest. We are continuing to monitor progress and will continue to update as the backlog resolves.
The carrier has corrected the underlying network problem about nine hours ago, and impacted device reporting has recovered to 56% of nominal. Estimated time to normal operations is by close of business today. We are continuing to monitor progress, with additional status update at COB.
Our engineers have identified an issue with a wireless carrier. Our Network as a Service provider has contacted the carrier who is actively working to fix the network problem. Expect delay in LGate 120 LTE meter readings ingestion until connectivity restored. Note that this also affects a subset of CatM meters placed on this IoT plan.
Report: "PVGuard platform is unavailable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue with site connectivity. Customers with sites on PT web can view up-to-date device data.
Report: "Delays in ingest services"
Last updateThe site data latency is nominal and we consider this incident resolved.
We are currently experiencing a delay in some site data which is now ingesting into the system. The delay is varied between 1 and 5 hours. We will update this messaging once all back logged data is consumed.