Historical record of incidents for Kickserv
Report: "Dispatch view"
Last updateThis incident has been resolved.
We've received reports that the dispatch view is not loading properly. We're looking into it now. Until this is resolved, please use a different view (Schedule, Calendar, Resources, or Map) to view your jobs for the day. We'll update this post as soon as we have more information.
Report: "Unformatted links in text messages"
Last updateThis incident has been resolved.
Normally, links in our text messages are shortened by a third-party service (Rebrandly) to fit and look similar to the following: https://kck.sv/mrz2 However, Rebrandly's API is currently reporting an error, and our links are going unshortened: https://app.kickserv.com/servicechampions/self_service/jobs/11240?ksrc=sms&token=xH-4tCHXbcx39KBZPvkEAoSlh56LHnNQua1Q2lHsdmU This does not impact the behavior of text messages or prevent customers from visiting links in text messages. In rare cases (mostly on older devices), the text message may be split into multiple parts if it is too long. We are working with Rebrandly to resolve the issue and get our link shortening working properly again as soon as possible.
Report: "Date Formatting Error"
Last updateThis incident has been resolved.
Both iOS and Android should be working now, once users download the latest update.
Update: The issue has been identified and tested. We've pushed new versions with the fix to both major app stores. Please keep the app updated to make sure you have all the latest stability enhancements and bug fixes. Thank you for your patience and have a good week.
In our efforts to modernize our mobile app, we updated our date-formatting code. We missed one place where the old method is still called. Users will find that saving new or existing tasks on any job record will generate an error with a 'Try Again' page. This error is purely visual and the task is successfully saved to the job record. When users encounter this, it's recommended to simply tap the 'Try Again' button a few times and the error will resolve and take you back to the job record. We have identified the issue, we are working on the fix and we'll have a new build submitted for approval to both major app stores shortly. Depending on approval times, the update should be publicly available this coming week. We're sorry for any inconvenience, and we thank you for your patience.
Report: "Redirect Issue/Degraded State"
Last updateAll systems are operational.
Navigating and searching the Kickserv web application is resulting in an issue where our users are getting directed to our main sign in page or showing an invalid customer with an error referencing degraded state. Our team is investigating the cause.
Report: "Emails from Kickserv free accounts to Yahoo and AOL may be sent to spam folders"
Last updateThis incident has been resolved.
Some emails sent from Kickserv free accounts to Yahoo or AOL email addresses are being sent to customers' junk mail folders. This is a result of spammers attempting to use Kickserv to send unsolicited email, raising our spam score at certain email providers. We have added measures to eliminate spam, and are working with Yahoo and AOL to resolve this problem more quickly, but spam scores take time to correct themselves. Please be patient and let your Yahoo and AOL customers know to check their junk mail folders for emails from Kickserv. You can also send notifications via text message instead of (or in addition to) email. This gets your customers notified more quickly without the need to check email at all. Check your Messaging settings in Kickserv to enable text messages.
Report: "Mobile app "Share in Customer Center" when uploading files"
Last updateThis incident has been resolved.
We have released the fix for this issue in version 3.10.11. It is available on the Google Play Store for Android devices and in review in the App Store for Apple devices. We'll let you know as soon as Apple releases to the App Store.
The issue has been identified and a fix is being implemented.
With the latest mobile app release (version 3.10.10), we discovered an issue when adding files to a job or customer. If you check the "Share in Customer Center" option, and do not enter a description, the app may crash. We are investigating this now, but there's an easy workaround. Just make sure you enter a description. You can even enter some text and then delete it before uploading. We'll let you know as soon as we have a fix available.
Report: "Server issue with latest version of mobile app"
Last updateVersion 3.10.14 is now available on the iOS App Store. This version contains the fix. If you have version 3.10.13 with the data error, please update your app now. Once you've updated, you will no longer see the "MOBILE_STAGING_SERVER_URL" message.
While we work to get a fix approved and released on the iOS App Store, we've made a configuration change on our servers to allow the mobile app to use production data. You'll still see the message "MOBILE_STAGING_SERVER_URL" at the bottom left of your screen, but your Kickserv account's jobs, customers, and employees should all be correct now. We recommend force-closing and relaunching the app for good measure. We'll update this space when a new version with the permanent fix is available for download. Thanks for bearing with us.
Our latest iOS app release (3.10.13) is incorrectly using test data. If you use an iOS device, please do not update your app just yet. We’ll get that fixed today.
Report: "Issues displaying new opportunities"
Last updateOn Monday afternoon, we deployed a new option to the filters you see at the top of the Jobs lists \(Opportunities, Jobs, and Invoices\). The new option allows you to choose a custom date range \(from any date, to any date\) rather than being limited to a specific timeframe \(this month, last year, etc\). Unfortunately, there was a bug in the code for the new option that went undiscovered during QA testing. Our product and success team members test every fix and new feature before approving it for deployment. This means our software is tested by the people who are most familiar with the ways that real-world users use it. However, it also means our QA process is less formalized and there is room for improvement. We’re sorry this happened and we know you depend on our software to run your business. Reliability and trust are of prime importance to us. We will guard against this in the future by adding specific steps and hints to testers before releasing code changes to QA, and making sure that the code review process includes verifying that QA documentation is in place.
This incident has been resolved.
We identified the issue as a bug introduced with some recent changes to our opportunities filter. We rolled back those changes and you should once again see your new opportunities back where they belong. We're sorry for the inconvenience.
We are continuing to investigate this issue.
We've received reports that the New Opportunities tab is not showing opportunities. We're looking into the issue now.
Report: "Delivery issues with some email notifications"
Last updateTowards the end of last week, the success team received a few reports of undelivered tasks and agendas. The engineering team investigated and found that the part of our system that queues up background tasks—like emails and texts—occasionally backed up to the extent that a task timed out after a number of retries. There are several queues in the system that share a finite set of resources, and things like QuickBooks syncs and search indexing sometimes generate a large number of background tasks. The problem went away on its own with no obvious cause, and troubleshooting became more challenging without the ability to see the system under load. Engineers focused on other pressing support and maintenance issues. Unfortunately, the condition degraded over the long weekend until it affected many accounts on Monday, May 29th. This was probably exacerbated by higher-than-normal, post-Memorial Day weekend activity. For example: everyone arriving at the office on Tuesday and running a big QuickBooks sync at the same time. We’re addressing this specific issue, and guarding against future occurrences, by doing the following: * Adding more monitoring for task notifications and other essential background tasks in the system. In the event that the notification queue starts to enter a retry situation, we will alert engineering and reprioritize the notification queue higher so that tasks do not get dropped. * Prioritizing updates to infrastructure software. The part of our system that runs background tasks is starting to get out of date, and newer versions offer improved performance and stability. We are sorry this happened and are grateful to you for bearing with us.
This incident has been resolved.
We are continuing to monitor for any further issues.
We have identified this as an issue with our email sending queue. We updated some settings on the queue and emails are sending normally. Please let us know if you experience any further delivery issues. Thanks!
We've received reports that email notifications are not being delivered in some cases. We're looking into it now.
Report: "Credit card payment issues this past weekend"
Last updateOver the weekend, we deployed a minor update to our credit card processing. The update passed QA, but once in production, the new code triggered an unexpected error condition at Stripe (Kickserv's credit card processor). This caused some credit card payments to fail with an error message. We rolled back the update Monday morning and all payments since have processed without issues. If you or your customers processed a credit card payment over the weekend (from May 19 to the morning of May 22), check to make sure those payments went through, and run them again if necessary. We're sorry for the inconvenience.
Report: "Text message issues with T-Mobile"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
Text message deliveries to T-Mobile subscribers are back to normal this morning. We are continuing to work with Twilio to identify the root cause of the previous outage.
We are continuing to work with Twilio (our text message provider) to identify and resolve this. We'll keep you posted. In the meantime, you can always use Kickserv's built-in messaging to communicate with T-Mobile customers.
SMS text messages from Kickserv are currently not being received by T-mobile subscribers. We're investigating with our SMS provider now.
Report: "Errors in the latest version of the iOS app"
Last updateWe have released iOS version 3.10.5 to the App Store. Please update as soon as you can.
We'll be releasing a new build to the App Store as soon as possible. This will be version 3.10.4.1 and it will be a required update. In the meantime, we have made some server changes to allow version 3.10.4 to operate more or less normally. If your iOS device is not set to automatically update to the latest versions of apps, and you have not yet updated to 3.10.4, you should wait until 3.10.4.1 is released before updating the Kickserv app.
Our latest iOS app update contains a configuration error. We are correcting the issue and will release a fix ASAP. In the meantime, we recommend you do not update to the latest version of Kickserv on iOS (version 3.10.4). We'll let you know as soon as the App Store is updated with the fix.
Report: "Problems logging manual payments"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The fix for logging non-credit-card payments has proven more tricky than we anticipated. In the meantime, a page refresh or two should clear up the issue and allow you to log those payments.
We've identified the issue with payment logging and are pushing a fix now.
We've received reports that some accounts are unable to log manual (non-credit card) payments in Kickserv. We're currently looking into the issue and will update this page as soon as we have a resolution.
Report: "Issues adding notes and attachments"
Last updateWe have adjusted our autoscaling to take into account periods of higher S3 latency. If you continue to experience issues when adding notes or files, please let our support team know.
We've traced the problem to unexpected latency issues with Amazon Web Services' Simple Storage Service (S3). We've added additional compute resources to compensate and are monitoring the situation in case we need to take further action.
We're receiving reports of problems adding notes and attachments to a job. Both web and mobile applications are affected. We're investigating now and will update this page as soon as we have identified the issue.
Report: "Inbound email replies are not being received"
Last updateThis incident has been resolved.
Postmark has let us know that many of the missing email replies can still be processed on a delayed basis. They're running those now. Keep an eye on your message inbox.
We are once again receiving inbound email replies. It appears that Postmark (our email provider) had a misconfigured setting on their end. Thanks for bearing with us.
We are investigating an issue which is preventing email replies from being received by the Kickserv web application. We hope to have it resolved shortly. Until we do, if you are awaiting a reply from a customer, you may want to follow up with them by phone or separate email. We'll update you when we have more information.
Report: "kck.sv links are not resolving"
Last updateThis incident has been resolved.
Service has been restored for the domain kck.sv, so existing short links for payment requests and to the Customer Center are once again working. As previously mentioned, though, we will be switching to kck.im very soon to guard against reoccurrence of the issue. We hope to switch back to kck.sv at some point in the future, if and when it becomes possible.
The issue has been identified and a fix is being implemented.
Due to severe technical limitations at the registrar for our kck.sv domain, we will switching to kck.im for our text message links. We'll update this incident when that's done. Any payment requests will need to be resent, as the old kck.sv links will no longer work. Note that you can always send your customer a link to their Customer Center--where they can view and pay invoices and requests--by visiting the customer's screen in Kickserv, selecting the Overview tab, and scrolling to the Customer Center section.
We are seeing DNS problems resolving web URLs created by our link shortening service. These URLs start with kck.sv and are used by our text messaging service and a few other areas of our web application. We are investigating now. In the meantime, if your customers are having problems viewing payment requests, you can send them a link to the Customer Center by visiting the customer's screen in Kickserv, selecting the Overview tab, and scrolling to the Customer Center section. Thanks for bearing with us while we sort this out.
Report: "Search is currently unavailable"
Last updateThe search index is now up to date and everything should once again be functioning normally. Please let our support team know if you encounter more issues.
Elastic has restored our search service. It will take some time for our servers to get the search index caught up with recent data changes, and this may affect certain aspects of Kickserv's performance--for instance, some event reminders may be slightly delayed. We are working to get everything completely back to normal, and we'll update this page as soon as we're there. Thanks for bearing with us.
Amazon Web Services, the hosting provider for Kickserv and much of the rest of the Internet, is having issues with one of their server availability zones. The Kickserv application itself is hosted in multiple zones and is unaffected, but Elastic Cloud (our search engine service) has been affected by the outage. We are working with them to try and resolve the situation. In the meantime, instead of searching for jobs or customers, we recommend using the Customers list, finding your customer by name, and selecting the job from the customer’s Jobs tab. We’ll post updates as we have them.
Report: "Amazon Web Services degraded performance"
Last updateAWS has resolved their issues, so Kickserv should be operating normally once again.
Amazon Web Services (AWS), the hosting provider for Kickserv and much of the Internet, is reporting various issues today in its us-east-1 region. Kickserv is hosted in multiple regions, so our performance has so far seen little impact, but this could change at any point. Many services we use also rely on AWS, so you may encounter issues that are out of our control. We'll update this page as events warrant. Thanks for bearing with us. For more details, watch the AWS status page at https://status.aws.amazon.com
Report: "Reports of T-Mobile users not receiving Kickserv texts"
Last updateT-Mobile users are once again receiving texts from Kickserv. There was a configuration problem between Twilio (our text messaging service provider) and T-Mobile that caused a number of texts to be mistakenly flagged as spam by T-Mobile. The problem has been resolved. Thanks for bearing with us.
We've received reports that Kickserv text messages are not being delivered to T-Mobile subscribers. We're looking into it with our SMS provider and will update this incident as soon as we know more. In the meantime, we advise using email notifications, Kickserv's built-in messaging, and the Kickserv mobile app wherever possible as alternatives to text messaging.
Report: "Investigating reports of text messages not being received"
Last updateThis incident has been resolved.
We are continuing to look for the root cause of the missing text messages, but in the meantime have made some changes that should help ensure they do not get stuck.
We are continuing to investigate this issue.
We've received a few reports of employees not receiving text notifications. We've been unable to reproduce the issue so far, but it may have to do with a problem at our text messaging service. We're continuing to look into it. Please let us know if you have encountered this problem.
Report: "Planned Search Re-index maintenance"
Last updateWe've paused the reindex during business hours today to prevent the extra load on application resources. The reindex is complete except for a stretch of jobs created between 5 and 9 years ago. We'll close this incident now and reopen a new one for the final batch this evening.
Over the weekend, we made some improvements to the way Kickserv's search works. Custom fields are once again searchable, and you can even search within notes and attachment names. These changes required us to regenerate our entire search index, which we did over the weekend (read the details here). As of Sunday evening US Central time, the customers index is complete and the jobs reindex is still in progress. Until it completes (expected in under 24 hours), some older jobs may be missing from search results. We reindexed all jobs beginning with the most recent, so unless you search for a very old job, chances are you won't notice any missing results. If you do happen to be looking for an older job that does not show up in a search, you can always find that job by searching for (or navigating to) the job's customer, clicking on the Jobs tab, and looking there. We'll update this page as soon as the reindex is complete.
We are continuing to monitor the re-indexing of the database. As always ping us at support@kickserv.com or from within the app with questions. Thank you for understanding as we progress through this update.
Kickserv is performing database maintenance. It began Friday, Aug 13th, at 6pm Pacific Time. This maintenance is related to the customer feedback regarding the search feature and capabilities. The estimate was that the work would be completed early this morning, but due to the size of the data the indexing has continued into most of Saturday. You may find that your search capabilities are spotty today. We are monitoring the progress and you should see a steady increase in search speed results throughout the day. Thank you for working with us. Ping us in support if you have any questions.
Report: "Jobs page read traffic alert"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
At approximately 10:50 am PDT / 12:50 pm CDT Tuesday Aug 10, 2021 for 12 minutes the following pages were unreachable: Jobs page Dashboard page Customers page We have installed logic that will speed-up database calls from the main database to read or view data. What is supposed to happen is when a request comes into the database the software will redistribute that load to multiple servers so that the main database is not overly burdened. We needed to run this during high traffic hours to locate any issues. A conflict was found between our logic and our hosted AWS load balancers which is inherent with our servers. We quickly figured out what was causing the high error rates and dialed the load back onto the main database. This brought the pages back. We are going to continue to test while fully monitoring the situation and hopefully we will not have any further issues with this enhancement. We apologize for the inconvenience this has caused your workflow today. Once fully implemented you will see an enhanced experience with your customer and job data appearing more quickly on the pages.
Report: "Errors when adding job charges"
Last updateOn Tuesday, July 27, our engineers deployed an update to permissions on our items API. This particular API call is is normally used to list products and services, and should only be accessible to users who have a role of Owner or Admin. However, after deployment, we received reports of problems adding job charges. After investigating, we found that the same API call used in the item list view was also being used to search for an item while adding new job charges. This meant that Staff and Tech users were no longer able to add a job charge without running into an error. As usual, the remedy for this sort of bug involves two steps: 1\) stop the immediate symptoms; and 2\) address the underlying architectural issue. We took care of step one by putting permissions back the way they were, and job charges are now functioning properly again. For step two, we’ll need to make the API work the way it always should’ve worked: use a different API call when searching for an item to add as a job charge. This is a very different use case than showing the main list of items: one is an administrative function, and the other is used in the normal job workflow. They should be treated differently. We’ll also be reviewing our internal system documentation to come up with a more reliable way of tracking what API calls are used in what places. The Kickserv codebase is large and constantly changing, and some parts of it are close to ten years old—which is a lot of lines of code for a developer to have to keep track of! We’re always looking for ways to improve quality of life for our engineers as well as our customers. We regret that this happened, and thanks for bearing with us.
Staff and technician users are once again able to add job charges in the web and mobile apps. Thanks for your patience.
A fix has been implemented and we are monitoring the results.
This issue currently affects only users in the staff or tech roles. Admins and owners are unaffected. The fix is minor and is in progress.
We are currently seeing errors attempting to add charges to a job. This is related to a minor permissions change we just added to our API. The fix is in progress and shouldn't take long. Thanks for bearing with us.
Report: "Problems sending text messages to AT&T customers"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Twilio has posted a status update for the AT&T issue. Read about it here: https://status.twilio.com/incidents/n18x4t9qvw71
Only AT&T users are affected by this issue, which accounts for about 7% of all the text messages Kickserv sends.
We have identified the issue as a configuration setting on our account with Twilio (our SMS provider). We have opened an urgent support ticket with Twilio asking them to make the change. If you suspect your consumers are not receiving texts from Kickserv, please continue to reach out to them via email or phone.
We are currently seeing problems delivering text messages to AT&T customers. We’re looking into it with our SMS provider. We’ll let you know when we have more information. Until then, we recommend contacting consumers via email or telephone.
Report: "Possible text messaging issues earlier today"
Last updateBetween 7 am and 9 am US Central Time, Twilio (Kickserv's text messaging service) reported connection issues. Kickserv's own application log showed some errors sending text messages. It's likely that some text messages sent by the Kickserv application encountered the Twilio error and were unable to be sent. By 9:55 US Central Time, the Twilio service was once again operating normally. If you have reason to believe that any text messages (estimates, invoices, or notifications) did not make it to their destination, you can resend them now. More information about the Twilio outage: https://status.twilio.com/incidents/zmz5c9prz8mm
Report: "Possible delays in sending email notifications"
Last updateEarlier today, Postmark, our email provider, noticed that a Kickserv trial account was being used to send spam emails. We shut down the offending account and are investigating ways to improve our signup process to better keep spammers out. In the meantime, email sending is back to normal. Thanks for bearing with us.
Sending of email notifications has resumed.
We are currently investigating reports from our automated email service that email notifications are being delayed. Thanks for bearing with us while we look into the cause. In the meantime, we suggest you use text notifications to your customers whenever possible.
Report: "Brief unexpected downtime"
Last updateAt 5:55 pm US Central Time on January 25, Kickserv’s locations database \(used for tracking technician movements in the field\) began running automated daily cleanup operations. At nearly the same time, one or more mobile devices running the Kickserv application reestablished contact with the network, and began uploading a large backlog of locations stored throughout the day while the devices were offline. The combination of these two events \(a large database insert operation and a maintenance process which momentarily locks data\) degraded Kickserv database performance over the next several minutes, eventually becoming deadlocked at 6:13 and causing the main application to become unavailable. Kickserv engineers were automatically alerted and switched over to the backup database, a process which took one minute _\(no data is lost during the switchover process\)_. Degraded database performance continued for a few more minutes while engineers reset the web application to clear the database locks. Full availability was restored by 6:23 US Central Time. The downtime was caused by unanticipated database load from a combination of two things: a spike in location insert operations and a regularly-scheduled maintenance procedure. The engineering team is pursuing two courses of action to eliminate this confluence of events: 1. The maintenance process has already been rescheduled for later in the evening to avoid close-of-business time on the US West Coast \(but not so late as to affect start-of-business in Australia\). 2. Location storage may be moved out of the main Kickserv database and into Amazon’s DynamoDB service \(a lower-latency, high-performance serverless database service\) so as to avoid placing excessive load on the rest of the system. Kickserv apologizes for any inconvenience caused by the downtime.
At 6:13 pm US Central Time on Monday, January 25, the Kickserv application was briefly unavailable due to a spike in operations on our mobile location tracking database. The downtime was limited to brief periods between 4:13 and 4:23, at which time the database had completed its regular failover operation. We have identified the cause of the surge in database operations and will post details to this page shortly.
Report: "Logos missing for some accounts"
Last updateThis incident has been resolved.
Any missing logos should now be restored. Please let our support team know if this is not the case for you.
We've received a few reports of logo images missing on the dashboard page and on generated estimates or invoices. It's a side effect of a change we recently made to the way we store uploaded files. This should only affect a handful of accounts, but if you happen to be one of them, we're working on restoring those files today. If you'd rather not wait on the fix, you can upload your logo again and everything should go right back to normal. We're sorry for the inconvenience.
Report: "Infrastructure issues"
Last update# Kickserv 11/25/20 Outage ## Summary On November 25, 2020, Amazon Web Services' Kinesis service suffered an extended outage, which impacted availability of Kickserv's server infrastructure \(along with many other applications on the Web\). During the outage, Kickserv engineers built and transitioned to a backup production web environment that did not depend on the affected services. Total Kickserv application downtime was approximately 2 hours 25 minutes. ## Situation \(all times EST\) **At 8:14 am**, Amazon's Kinesis service began experiencing an outage. However, since Amazon's [Service Health Dashboard]() also relies on Kinesis, most AWS tenants were unaware of the problem until later. **At 10:54 am**, Kickserv's alerting systems began indicating an application outage. Engineers discovered that Elastic Container Service \(ECS\) was affected, but Elastic Compute Cloud \(EC2\) was not. Over the next 2 hours, engineers built a new, temporary production web application environment based on EC2 servers. All application data was preserved. **By 1:20 pm**, main Kickserv functionality was once again available. Over the next two hours, engineers restored search indexing, calendar live updates, and QuickBooks™ Desktop sync. **By 6:58 am** **on November 26**, ECS was once again functional, with all other services affecting Kickserv restored by 2:24 pm that afternoon. As of the time of this document, Kickserv is once again running on the regular ECS production stack. ## Remedy AWS outages that affect an entire application are very rare; only 2 such events in the past seven years have caused Kickserv to be down for an hour or more. However, they do happen, and they represent a single point of failure that should be eliminated. During the next few weeks, Kickserv engineers will build a failover production environment in a separate AWS availability zone. The backup environment will maintain a replica of all application data \(database records and user-uploaded files\), and with some improvements to our DNS architecture, we expect to be able to switch over in 30 minutes or less. We expect to have the new stack in place, along with protocols and training, by the end of January. We realize that our customers depend on Kickserv for the minute-by-minute smooth operation of their business, and regret the inconvenience this event caused. Thank you for bearing with us. ## For further reading [Summary of the Amazon Kinesis Event in the Northern Virginia \(US-EAST-1\) Region](https://aws.amazon.com/message/11201/)
This incident has been resolved.
QuickBooks™ Desktop sync has been restored to service. Kickserv should be fully operational once again. We are still using a backup server infrastructure, so you may occasionally notice longer than usual page load times. Don't hesitate to reach out if you have questions or experience problems.
Calendar views should now be operating normally. We are still working to restore QuickBooks™ Desktop sync.
QuickBooks/Xero sync, search, and emails are once again operating normally on our backup infrastructure. We're still working on the calendar views. We'll post another update when those are restored.
While most areas of Kickserv are up and running, we are still working on a few areas that are not yet functioning properly on our backup infrastructure: - Calendar screens - Live search results - Adding notes and files We want to stress again that no data has been lost. For instance, if an event is missing from your calendar, you should find it on the Schedule screen. We’re working on restoring full operation and we’ll let you know if we encounter any more difficulties. Thanks for your continued patience.
Display of the calendar pages continues to be affected by the AWS outage. While we're sorting that out, you can use the Schedule page or the Jobs page to keep track of today's events.
AWS is still having trouble, but in the meantime, we have restored service. Please let us know of any ongoing availability issues. Your data is safe: https://www.kickserv.com/security/
We're continuing to work on our contingency plan for getting around the AWS outage. We will update this page the second we have service restored.
We are still working with AWS and their team on timing of their systems getting back up. We are still working through alternative solutions to get our service restored.
We're currently experiencing infrastructure issues with Amazon Web Services. The issue seems to be affecting many parts of the Internet. Thanks for bearing with us while we investigate further.
Report: "In-app support issues"
Last updateThis incident has been resolved.
The support chat widget should once again be operating normally. Please let us know if you continue to have issues with it.
The issue with the support chat widget has been identified and we're working on the fix.
If the support widget is not loading for you, you can go to https://help.kickserv.com and try sending a support message from there.
We are continuing to investigate this issue.
We've received reports of the in-app support chat widget not loading properly. We're investigating and will keep this page updated. In the meantime, if you need help, you can check out https://help.kickserv.com or email support@kickserv.com.
Report: "Unexpected downtime"
Last updateWe traced the brief downtime to a user error that occurred while deploying a code change to our testing environment. We have updated our deploy procedures to add an extra check for the proper environment before deploying. We apologize for the inconvenience.
Kickserv is once again operating normally. We're looking into the cause of the incident and will post our findings. Thanks for bearing with us.
We’re investigating an issue with the availability of the Kickserv web application. Watch this space for updates as soon as they are available.
Report: "Employee reminders in the mobile app"
Last updateVersion 3.8.1 of the app, which fixes the blank screen issue, is now available in the iOS App Store and Google Play Store.
Version 3.8.1 of the mobile app is beginning to roll out now. Check your App Store over the next couple of days for an update.
The issue has been identified and we are preparing the fix now. Version 3.8.1 will be available for download as soon as it has been tested and approved by the app stores.
We're looking into reports of the mobile app (version 3.8) hanging when an employee reminder or time off event is scheduled for the current day. We've paused rollout of 3.8 and will have a fix out as soon as we can. Until this is resolved, we recommend you either refrain from updating to 3.8, or avoid scheduling those types of events.
Report: "Degraded performance on calendar, map, and resources"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We're still working on improving the performance of live updates for Map, Calendar, Dispatch, and Resources screens. Most people should not see any degraded performance. However, if you have many recurring events or many techs being tracked in the field, you may see a pause in live updating on one of those pages. If this happens to you, just refresh the browser and you'll see your changes. Thanks for bearing with us.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We're investigating instances of slower-than-usual load times on parts of the Kickserv web application that feature live-updating content. This includes (but may not be limited to) the Map, Calendar, and Resources screens. If you are experiencing unusual slowness in any of these areas, please let our support team know. Thanks!
Report: "Google Calendar connection issues"
Last updateThis incident has been resolved.
Our Google Calendar integration has been reviewed by Google and should be working normally again. Please let us know if you experience any problems connecting Kickserv to your Google Calendar.
We've discovered an issue in our integration with Google Calendar that is preventing most new connections at this time. If you are already connected to Google Calendar, you should not be affected. We're working with Google to resolve the issue.
Report: "Degraded search and sync performance"
Last updateThis incident has been resolved.
We have identified and addressed a problem briefly affecting search, QuickBooks/Xero sync, and job and customer views. We will continue monitoring performance, but it appears to be back to normal.
Report: "Service outage in some locations"
Last updateWe have not received any reports of availability issues for a couple of hours, so we're considering this issue resolved. Please let us know if you have problems accessing Kickserv. Thanks!
The network outages appear to have abated for now, but we'll continue watching the situation. Thanks for your patience. Pingdom has a helpful map of network outages for reference: https://livemap.pingdom.com/
The Kickserv web application is currently inaccessible from some geographic locations. This appears to be an Internet-wide availability issue. We're investigating now.
Report: "jonathan@kickserv.com"
Last updateThis incident has been resolved.
You can continue to view help articles and send support requests by chat, but Kickserv support representatives may be delayed in reading them.
Support and user guide are undergoing emergency maintenance. Follow Intercom's status page for updates: https://www.intercomstatus.com/
Report: "Live mobile tracking"
Last updateThis incident has been resolved.
The errant server has been disciplined and mobile GPS tracking should once again be working properly. Thanks for your patience.
The location tracking issue appears to be related to a misconfigured server. Stand by.
Tracking of mobile app users is currently not working as expected. We're looking into the issue now.
Report: "Estimates & invoice CC recipients"
Last updateThis incident has been resolved.
We've released a fix for the invoice & estimate CC recipients. Please contact support if the problem persists for you.
We've identified the issue with CC recipients and will be deploying a fix shortly.
We're looking into reports that estimate and invoice emails are not being sent properly to recipients in the CC field. We'll let you know what we find. In the meantime, you can either send a separate email to a CC recipient, or download the PDF copy and email it directly.
Report: "Google API connectivity issues in contact address field"
Last updateThis incident has been resolved.
The Google API issues appear to have been cleared up. We'll keep this incident open a little longer to make sure it's resolved.
We have received reports of the contact address field not working properly for some West Coast customers. This appears to be an issue with the Google location API that Kickserv uses to look up addresses. We're currently investigating and will post an update when we have more information.
Report: "Android app v3.7.1 crash issue"
Last updateThe issue that was causing the Android app to crash when running in the background has been resolved, and location tracking should be working as expected in version 3.7.1.1.
Version 3.7.1.1 is now available in the Google Play Store and should correct the background crash issue.
We've determined that the Android crash only occurs when the app is running in the background and should not affect app usage, although location tracking may be impacted. Regardless, an app update containing the fix should be released soon.
We've found an issue that can cause crashes in the latest release of the Android app (3.7.1) and are working on an expedited fix now. If you have not yet upgraded from the previous version (3.7), we highly recommend not updating the app until we release the fix.
Report: "Error when viewing job lists"
Last updateThe jobs lists and all custom reports should once again be functioning normally. Thanks for your patience.
As a side effect of the fix for this incident, custom jobs reports that depend on a job completed date will be unavailable for a brief period of time. We're working to get that functionality restored as quickly as possible. Thanks for bearing with us.
A fix has been implemented and we are monitoring the results.
We've received reports of error pages when viewing the jobs list. We've identified the problem and will be deploying a fix very soon. Meanwhile, as a workaround, we suggest using the Schedule, Calendar, Resources, Dispatch, and Check-ins pages instead, if possible. We're sorry for the inconvenience.
Report: "iOS app temporarily removed from the App Store"
Last updateThe Kickserv app is once again available for download in the iOS App Store. We'll have a full writeup of the situation shortly.
Apple recently changed their App Store requirements involving collection of usage metrics. This resulted in the removal (with very little notice) of the Kickserv app from the App Store. We are submitting an updated app version today without usage tracking, and it will be available as soon as Apple approves it for release. If you currently have the Kickserv app installed on your iOS device, there is no impact to you at all, but we strongly recommend you do not uninstall it until the new version is available for download. Users of the Android app are unaffected, of course.
Report: "Attachment upload problems in Internet Explorer"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Some IE users are currently reporting problems uploading attachments. We should have it resolved momentarily. Please try a different browser in the meantime.
Report: "Event and charge display on mobile"
Last updateThis incident has been resolved.
Mobile version 3.5.3, which contains the fix, is available in the iOS App Store. If you're affected by this issue, we recommend you update your app right away.
This is fixed in version 3.5.3, which is available now for Android and will be available for iOS as soon as it has been reviewed by the App Store (usually 24 hours or less).
We’ve received reports that some jobs are displaying events and charges incorrectly in the mobile app. We’re testing the fix now and will post an update when it’s available for download. This applies to the mobile app only—the web application should function as expected. If you encounter these issues with the mobile app, try quitting and relaunching it.
Report: "Event and charge display issues on mobile app"
Last updateThis incident has been resolved.
We just released version 3.5.2 of our app to the iTunes App Store, which should fix the issue. If you have an iOS device and were experiencing this issue, please update the Kickserv app and try again. Thanks!
We’ve received reports that some jobs are displaying events and charges incorrectly in the mobile app. We’re working on a fix now and will post an update when it’s available for download. This applies to the mobile app only—the web application should function as expected. If you encounter these issues with the mobile app, try quitting and relaunching it.
Report: "Quickbooks Desktop sync server update"
Last updateThis incident has been resolved.
This evening, we changed our Quickbooks Desktop sync servers to new infrastructure. Our server's IP address changed during the update. If your DNS cache has not been updated, you may have connection issues during Quickbooks Desktop syncs. This issue typically happens because a local machine needs to update its DNS cache. So if it's happening to you, try flushing your DNS cache: https://www.noip.com/support/knowledgebase/how-do-i-flush-my-windows-dns-cache/ If that doesn't resolve the error, you can also try rebooting your wireless modem or router.
Report: "DNS caching post infrastructure update"
Last updateThis incident has been resolved.
After a maintenance update on Monday, we had reports from some users of "502 Gateway" errors. Our application server's IP address changed during the update. This issue typically happens because a local machine needs to update its DNS cache. So if it's happening to you, try flushing your DNS cache: https://www.noip.com/support/knowledgebase/how-do-i-flush-my-windows-dns-cache/ If that doesn't resolve the error, you can also try rebooting your wireless modem or router.
Report: "Search is resulting in Ah Nuts page"
Last updateThis incident has been resolved.
We are temporarily rolling back some of our maintenance upgrades while we implement the permanent solution to the search issues. Search should be functioning normally again. Thanks for your patience.
We're continuing to investigate the search issues. If you get an "Ah nuts" page when attempting to search, please refresh the page. That may give you your search results.
We have reports that after our upgrade we are getting errors on searching. To find records use the Opps, Jobs and Invoices tabs and to locate contacts use the contacts tab until this is resolved.
Report: "Custom template error"
Last updateThis incident has been resolved.
We are temporarily rolling back some of our maintenance upgrades while we implement the permanent solution. Custom templates should be functioning normally again. Thanks for your patience.
Estimates, invoices, and work orders printed or sent using custom templates will currently have missing information or errors. This is upgrade-related and we're looking into it. In the meantime, if you use custom templates and you need to print or send documents, we recommend you first copy and paste the template code into a separate file and then temporarily disable custom templates. This will allow you to send documents with the built-in templates. Thanks for your patience.