IT Glue is currently unavailable
Incident Report for IT Glue
Postmortem

We appreciate your patience while our developers investigated the cause of this incident.

On Nov 8, 2017 at 06:57 PST performance degradation was detected which escalated to excessive load on our database environment.

During the preliminary incident investigation, emergency procedures were instigated, every element of IT Glue restarted, and geographical failover initiated to our backup availability zone. This ruled out any possible hosting environment-specific issue, but symptoms continued.

After deeper investigation we found that a specific database table was the root cause. Emergency code was written and reviewed to disable Recent Items and Activity Feed functionality, both of which required access to this table.

At 07:50 PST these changes were pushed to production, and the processing of queued requests resumed. Database load began its return to normal levels.

At 08:10 PST full performance was restored and the incident was confirmed closed.

Activity Feeds were restored to normal operation by Nov 18th. Our team of developers has been working hard on bringing back Recent Items functionality, while taking the opportunity to ensure our environment becomes even more resilient and scalable.

We are confident that this incident was isolated to this specific table, and there is no other IT Glue architecture currently affected. Our sincerest apologies go out to all our partners, as we fully understand how critical IT Glue is to your day-to-day operations. We will continue working tirelessly to drive the best availability possible.

Posted over 1 year ago. Dec 14, 2017 - 11:31 PST

Resolved
Normal IT Glue performance has resumed. Activity feeds and Recent Items are not being shown throughout most of the App. This is a temporary situation. Activity Logs will operate normally and contain complete information for Admins.
Posted over 1 year ago. Nov 08, 2017 - 08:41 PST
Update
Normal operation is resuming. We expect to confirm full resolution here presently.
Posted over 1 year ago. Nov 08, 2017 - 08:20 PST
Update
We are still establishing root cause. We love documentation. We hate letting you down and affecting your business.
Posted over 1 year ago. Nov 08, 2017 - 07:52 PST
Monitoring
IT Glue is still unavailable. We have our top team working to resolution as quickly as possible. We appreciate your patience.
Posted over 1 year ago. Nov 08, 2017 - 07:28 PST
Identified
We are investigating some performance issues that are preventing access to IT Glue for all our partners. Please subscribe here for updates as we have them.
Posted over 1 year ago. Nov 08, 2017 - 07:14 PST