Is Noko Time Tracking Down Right Now? Check if there is a current outage ongoing.

This incident has been resolved.

We've moved background task processing from the affected servers temporarily to other servers while we work on fixing the underlying issue.

We're seeing a delay on some background tasks due to a network issue in our data center and are working on fixing the issue.

Some low-priority background tasks are still catching up (which has no impact on using Noko). Everything will be completely back to normal in several hours!

We've identified an issue with a stuck background worker and have restarted it. Noko is currently catching up; there should be no longer an impact on performance. We will continue monitoring.

We are currently investigating this issue.

Noko's Back up, sorry about the hiccup!

It looks like the fix has stuck; but we're still going to be monitoring it

We've implemented a fix and we're monitoring the response times to make sure everything's squared away. Thanks so much for your patience!

We're looking into Noko being very slow to load or timing out completely.

We're still monitoring the situation, and will open a new one if needed

Our monitoring systems are showing green again, but we're keeping a close eye and will keep you updated.

Noko's hosting provider is encountering an issue: https://status.digitalocean.com/incidents/xkpsn9hbnnlz. We're monitoring their status updates and will keep you posted.

It looks like our hosting provider was able to re-route traffic, but we're keeping an eye on it!

We're aware of Noko being very slow to load or timing out completely, due to an outage from a major Internet backbone provider, which is affecting our hosting. We're monitoring the situation, and will keep you updated

This incident has been resolved.

Our hosting provider is looking into networking issues in their datacenter; there may be some intermittent packet loss and network latency. We'll update this incident as soon as they give the all-clear.

Sorry about the timeouts! The servers look like they're back to normal, but we're monitoring them to make sure everything's working.

We're looking into Noko being very slow to load or timing out completely.

Sorry about the timeouts! A fix has been implemented and we're monitoring the servers to make sure everything's working.

We're looking into Noko being very slow to load or timing out completely.

This incident has been resolved.

We've restarted the hanging processes causing this issue. We're keeping an eye on our metrics and logs.

We're seeing hanging processes and 500 errors and are investigating.

This incident has been resolved.

All systems go. We've identified the underlying problem and will take measures to prevent this from happening again in the future.

We're looking into elevated errors rates.

This incident has been resolved.

We've fixed the stuck processing of background tasks and are waiting for queues to clear out.

We're seeing in issue with background processing not working correctly. This affects some parts of Freckle, like sending email, cache invalidation and PDF downloads. Logging time is working normally.

All background processing has been restored to normal.

We're seeing issues with background tasks (like email sending and imports) not being processed. We're investigating the cause.

Issues with Amazon's CloudFlare DNS not resolving have been fixed by Amazon. Happy Thanksgiving!

Freckle is back—we've temporarily disabled Amazon CloudFront. Loading times will be a bit slower and we're seeing some Avatars not being loading, but everything else should work. We'll restore CDN functionality once Amazon CloudFront gets fixed.

CloudFront DNS seems to be down again. We're preparing a workaround should this situation not be resolved by Amazon in 10-15 minutes.

For some users, Freckle has returned to loading normally. The root cause of this outage was that Amazon Cloudfront's DNS entries couldn't be resolved. It appears that DNS is coming back online for it. There's no word from Amazon, so we're watching the situation. It might take a while to work for you as DNS changes propagate.

Note: Freckle is down, because JavaScript can't be served as well. We're monitoring and have a fallback strategy in case Amazon CloudFront is not coming back up soon.

Our CDN (Amazon CloudFront) appears to be offline, unfortunately affecting serving of stylesheets and graphics for Freckle. We're looking into this problem right now.

We have increased capacity and tweaked a few things here and there for smoother loading.

We're seeing some 500 errors and slow loading problems and are looking into it.

We've tuned some background processing to lighten the load and haven't seen 500 errors in several hours. We're marking this issue as solved, but will continue to keep an eye on it.

Load is back to normal levels. We'll continue to monitor.

We still see occasional errors and are working on fixing these.

We've rolled out a fix for the main cause of the 500 errors. Certain types of reports could cause our Unicorn server processes to become saturated with new request being denied. We'll monitor load and error occurrences and will update as necessary, though we don't expect any more issues.

We're again seeing 500 errors & are looking into it.

All systems go! :)

After a deployment Freckle is not available, we're looking into the root cause.

This incident has been resolved.

We're seeing some emails and other background processing tasks being delayed due to a stuck processing queue. We have fixed the cause and queues are now processed normally, however it will take a little while to catch up with all the tasks that have accumulated.

All queues are processing normally. There's some backlog on syncing with Basecamp, but we're catching up. If that is the case for your account it should be resolved within the next hour or so.

Background processing is working normally, we'll update when all queued work has been processed.

A few queued tasks got stuck when processing due to a timeout when talking to external services not kicking in. We're isolating the affected background worker queues so that further problems with them won't affect other tasks (like mail sending).

A few of our background processing queues are not processing tasks, and work is queuing up. We are investigating the cause.

This incident has been resolved.

Due to memory exhaustion some background tasks are not processing. We're looking into resolving this quickly.

This incident has been resolved.

PDF downloads of invoices and reports have been failing on and off today. Right now, PDF downloads should work again for all customers. We've identified the root cause as background workers running low on memory and are working on resolving this problem permanently.

Support chat is available again!

Our support chat software is currently not working, please send us an email at support@letsfreckle.com if you need support! Sorry about this!

This incident has been resolved.

Freckle is back up. We're monitoring our fix.

Freckle is currently not available, we're investigating and keep you updated.

This incident has been resolved.

Due to an issue of a 3rd-party service we use, when using timers you may not see the latest version when you switch devices or computers. You can workaround by reloading the timer window. Starting/pausing and logging timers works normally. (See https://status.pusher.com/incidents/2llsqlg9rlmy for details on the issue with the Pusher 3rd-party service.)

Today, March 5, 2015, the **Freckle app experienced an outage of approximately 2:45 hours**. First of all, this sucks—and we're sorry that it happened. Freckle is supposed to save you time—not make you bite your fingernails in anxiety and wear out your ⌘R keys to see if it's back up again.  Here's the sequence of things as they happened: * There was a **scheduled emergency server maintenance from our hosting provider Rackspace**, which happened from about 1am EST to 5am EST. They installed a critical, as yet not publicly available security update to the Xen virtualization software used in their hosting environment. (Yay, security! Boo, timing in the middle of the workweek!) * After this maintenance was complete, the Freckle servers, as expected were automatically booted back up. However, **one of the servers, while booting normally, did not start the nginx service**, which, as our main web server software is critical. * The underlying reason why nginx didn't start was that DNS was not working properly—nginx checks SSL certificates and it couldn't resolve one of the hosts needed to verify our main SSL certificate. We don't know why DNS didn't resolve, but it's likely that to the large number of booted servers in the Rackspace datacenter there was a temporary problem with DNS resolution requests.) * This in turn made the **Freckle web app not load, with requests timing out**. (Same for related apps, like Mac app and iPhone app. * Our monitoring alerted us immediately—however **there was a window of time this morning were no one was reacting to the alerts** (as a small business, we mostly have one person on on-call duty, and in this case, while alerts did go through, the CTO simply failed to wake up [Thanks, allergies!]). _This is 100% our fault, and we'll make changes so someone is actively monitoring any maintenance that happens by our hosting provider._ * As soon as alerts were received, we immediately announced the outage and investigated the cause. A few reboots later, and with working DNS resolution, **Freckle, and the Freckle apps are now fully up and running again**. Again, this sucks, and it's our fault. We'll make changes so it won't happen again. **_Thomas Fuchs, Partner & CTO Freckle Time Tracking_** _Note that the landing page and help/support site were not affected by this problem._

This incident has been resolved.

Freckle is back!

We're looking into why Freckle is not available after the scheduled maintenance of our hosting provider today

Freckle is all back in action. The root cause of the errors was a temporary traffic spike on our API. Our emergency systems we have in place to recover from an overload like this worked as expected and Freckle came back after a downtime of about 4 minutes. Obviously this still sucks, and we're taking steps to prevent the same thing from happening again in the future so that temporary traffic spikes will no longer cause Freckle to become unavailable.

Freckle is back to normal! We're monitoring Freckle for the time being and are further investigating the root cause.

We're getting reports that Freckle is not loading for some customers and are looking into it.

This incident has been resolved.

We've fixed the underlying problem and Freckle is all back to normal! Whew!

We've identified the problem and are working on a fix that should be deployed within minutes.

We're seeing a high rate of 500 errors and are investigating.

This incident has been resolved.

Freckle briefly experienced some downtime, but we're back to normal! We're monitoring Freckle for the time being and are further investigating the root cause.

This incident has been resolved.

Due to issues with our upstream DNS provider the http://developer.letsfreckle.com is currently not available at least for some customers. See also https://www.cloudflarestatus.com/incidents/m8hf8q8vs81d

Today, October 28, 2015, **Freckle experienced an outage of approximately 5 hours and 20 minutes**, between 5:54am UTC and 11:14am UTC. First of all, **this sucks and we’re extremely sorry that it happened.** Freckle is supposed to save you time—not make you bite your fingernails in anxiety and frantically refresh to see if it’s back up again.  Here’s the sequence of events as they happened: 1. **There was a scheduled emergency reboot for one of our servers** from our hosting provider Rackspace, which was scheduled to occur between 3AM UTC and 5AM UTC. We've been informed about this by Rackspace a few days ago, and set up a scheduled maintenance notice on our status page (this reboot was to apply a critical security patch to Rackspace's hosting infrastructure). **Part of the Freckle Ops team was actively monitoring the servers to make sure the reboot was performed successfully and Freckle continued to work as expected.** _The server that was scheduled to be rebooted did so at approximately 3:30AM UTC without a problem._ 2. **At approximately 5:54 AM UTC, a different server was rebooted as part of a different scheduled infrastructure maintenance.** Rackspace did not notify us that this server would be rebooted at all, and it was rebooted outside of the previously scheduled maintenance window. 3. Unfortunately, **this reboot did not properly configure the networking interfaces for the server**, causing the server to be up and running but unreachable by our other servers. **This is turn caused Freckle to be unavailable and show an error page.** 4. Our monitoring alerted us immediately—however our ops team was not actively monitoring the server because they were unaware of the additional scheduled reboot. _In other words, our on-call staff was actively prepared for the given maintenance schedule, but a different server was rebooted outside of that window (two hours laters when the ops team was asleep)._ **However, the main issue is that we did not receive the alerts that the second server was down immediately. This is 100% our fault, and we’re going to make sure that our ops team is notified of any major server outages via multiple methods.** 6. **As soon as alerts were received by our ops team, we immediately contacted Rackspace's support to investigate and resolve the outage.** They were able to restore network connectivity for the server, and Freckle, and the Freckle apps are now fully up and running again. **Rackspace has committed to investigate why we weren't informed of the reboot.** The have confimed that we should have been informed (_"…the fact of the matter is that you were not made aware that this server was going to be impacted by the reboots today in the ORD datacenter"_). We will update this post-mortem when we hear back from them. **Again, this sucks, and we’re extremely sorry about the downtime.** We're proud of being able to deliver high availability of our hosted software to our customers, with most months at 100% or 99.99% availability—but **we completely failed to do so today.** We’re actively addressing how our ops team is notified of server outages, to ensure we will respond as quickly as possible. Thomas Fuchs Co-Founder and CTO, Freckle Time Tracking

Freckle was down and not loading from 5:54am UTC to 11:14am UTC. First of all, we're extremely sorry that this happened. This sucks, and it definitely should not happen. Freckle is back to working order and we've identified and resolved the issue. We will provide a detailed post-mortem analysis.

This incident has been resolved.

We need to restart parts of Freckle in order to solve some residual issues from this morning's outage. This will cause a short outage of no more than a few minutes.

This incident has been resolved.

We've temporarily added more capacity to handle the integration queues and all Basecamp data is now current. We're continuing working on a fix for the root cause that causes the synchronization to run slower than expected in some cases.

Some accounts are seeing delays in syncing projects from Basecamp. We're seeing that sometimes calling the Basecamp APIs is taking a long time, which in order causes syncing to be queued up and longer than normal waiting times for queue processing. We're looking into the cause for the delays.

Freckle is running smoothly again, and our ops team will continue to keep an eye on the servers.

It looks like Freckle is back up, we are continuing to monitor the app and investigating the root cause.

Freckle is down and not loading. We're identifying the root cause right now and will keep you posted.

This incident has been resolved.

After our hosting provider's scheduled maintenance some network connections weren't restored correctly leading to background processing queues being stuck. We've restored connectivity and the stuck queues are now being processed normally and are catching up.

The underlying reason of API latency was an API client issuing too many requests for large sets of entries. As a temporary fix we've reached out to the affected customer and they've reduced the amount of requests issued. In the longer term, we're working on better caching for the more popular API methods; this will all but eliminate latency issues like this one.

We're seeing some slow-downs in processing API requests. The API is still functioning fully, but response times are slow for some types of API calls.

This incident has been resolved.

The slowdowns and timeouts appear to have stopped, but we're continuing to monitor Freckle.

We're looking into Freckle being very slow to load or timing out completely.

Freckle is back up after our hosting provider experienced a temporary network outage. Our ops team will continue to monitor the app.

Freckle is back up, but we're continuing to investigate the root cause and monitor the app.

Freckle is down and not loading. We're identifying the root cause right now and will keep you posted.

Freckle was unavailable for about a minute today shortly before 10am EST. This was due to a configuration issue that has been resolved. Sorry about this! All systems are go now!

Everything is back to normal! If you still have any issues downloading reports or think you're seeing out of date statistics, please let us know at support@letsfreckle.com so we can have a look. Thanks for your patience while we fixed this—we know the first day of the month is a busy day for invoicing and payroll and it's not helpful when your time tracking software is having trouble generating the Excel files and PDFs you need. :(

We're deploying an update that will fix the underlying cause of the issue and will allow our background processing to catch up on report statistics and make sure all those numbers you see on the reports page are the latest and greatest. We'll report back once everything is back to normal.

A secondary impact from the earlier issue is that summaries of reports on the reports page (e.g. total sum, etc.) are in some cases slightly out of date. The sums will be correct if you run the report. We're working on getting this up-to-date again.

Download are working again. Yay! We're monitoring the fix and the server performance and will update this incident as needed.

Downloads are not working right now. We've identified the problem and are working on bringing everything back up.

PDF generation is working again! We'll be keeping an eye on it, though!

Due to an outage with DocRaptor, the 3rd party service we use to generate PDFs, PDF exports of reports and invoices are unavailable. We are actively monitoring the 3rd party service and will notify you when the outage is over.

We've been given the A-OK from our hosting provider that the hardware issue has been fixed. All systems are working normally now. We'll of course continue to monitor (as we always do!). Happy weekend!

And Freckle is back! Yay! We'll keep monitoring and post updates as needed.

Our hosting provider are right now replacing defective hardware on the web host—we'll keep you posted on the progress and update here as soon as we know more (and hopefully have an ETA for when Freckle will be back up soon!).

Our hosting provider's engineering team is continuing to work on fixing the issue. We'll update here as soon as we get more information.

We're in contact with our hosting provider's engineering team and working to identify and resolve the problem. The issue specifically affects our web servers only; the database/backend is functioning normally (but can't be accessed without the web front-end).

We've been informed by our hosting provider that there's an issue with their hosting environment and they're working on resolving it. We'll keep posting updates here.

We just got alerted by our monitoring that Freckle is down and we're investigating.

This incident has been resolved.

Our hosting provider has resolved the underlying issue. We'll keep monitoring just in case.

Our hosting provider still shows the outage, but our monitoring systems show Freckle as online and our engineering team is actively monitoring Freckle's status. Thanks to everyone for your patience as we've been dealing with this outage, and we're continuing to monitor Freckle (as always!)

We're still waiting for an update from our hosting provider, but our monitoring tools show the web servers are online and accepting requests. However, the database is still in a degraded state and we're waiting for it to return to normal/actively monitoring it.

We're still waiting for an update from our hosting provider, and will update our status page as soon as possible.

Our hosting provider's engineering team is continuing to work on fixing the issue. We'll update here as soon as we get more information.

We've been informed by our hosting provider that there's an issue with their hosting environment and they're working on resolving it. We'll keep posting updates here.

Freckle is down and not loading. We're identifying the root cause right now and will keep you posted.

This incident has been resolved.

We've implemented a fix to improve Freckle's overall performance, and are monitoring the results.

We're looking into Freckle being very slow to load or timing out completely.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We're looking into Freckle being very slow to load or timing out completely.

This incident has been resolved.

Freckle's server infrastructure will be going through some upgrades. Freckle should be generally available, but will be offline for a short period of time (up to 10 minutes)

This incident has been resolved.

Our hosting provider has finished maintenance. Freckle is all up and running, we will monitor.

Freckle is down due to emergency hardware maintenance of our hosting provider. We will post updates as we receive them.

Freckle is currently down, we're investigating.

This incident has been resolved.

Unfortunately our hosting provider is doing further emergency maintenance which affects connectivity to our main database, which in turn causes 500 errors in the web app.

We're seeing 500 errors and are investigating.

This incident has been resolved.

Freckle's hosting provider is currently experiencing network connectivity issues. They have implemented a fix, and we are monitoring the status of their fix. We'll let you know as soon as the issue has been resolved!

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

Freckle is down and not loading. We're identifying the root cause right now and will keep you posted.

We've solved the timeouts and slowdowns, and will continue to monitor the app.

We're looking into Freckle being very slow to load or timing out completely.

Sorry about the timeouts! A fix has been implemented and we're monitoring the servers to make sure everything's working.

We're looking into Freckle being very slow to load or timing out completely.