We appreciate your patience while our developers investigated the cause of this incident.

On Nov 8, 2017 at 06:57 PST performance degradation was detected which escalated to excessive load on our database environment.

During the preliminary incident investigation, emergency procedures were instigated, every element of IT Glue restarted, and geographical failover initiated to our backup availability zone. This ruled out any possible hosting environment-specific issue, but symptoms continued.

After deeper investigation we found that a specific database table was the root cause. Emergency code was written and reviewed to disable Recent Items and Activity Feed functionality, both of which required access to this table.

At 07:50 PST these changes were pushed to production, and the processing of queued requests resumed. Database load began its return to normal levels.

At 08:10 PST full performance was restored and the incident was confirmed closed.

Activity Feeds were restored to normal operation by Nov 18th. Our team of developers has been working hard on bringing back Recent Items functionality, while taking the opportunity to ensure our environment becomes even more resilient and scalable.

We are confident that this incident was isolated to this specific table, and there is no other IT Glue architecture currently affected. Our sincerest apologies go out to all our partners, as we fully understand how critical IT Glue is to your day-to-day operations. We will continue working tirelessly to drive the best availability possible.