Solidsoft Reply have been providing the pharmaceutical industry with bespoke Track and Trace systems for over 15 years with a dedicated Operational Team, covering 12 hours per day, 6 days per week.

A new project required a global support coverage for 24/7 system availability to be monitored via a single dashboard. Initially Solidsoft Reply partnered with a call handling company to provide the ‘eyes on’ monitoring outside of the Solidsoft Reply business hours.

After a period of 2 years Solidsoft Reply had found that their robust system had only suffered 3 instances of potential high priority incidents, resulting in a ‘red alert’ scenario.

The Challenge

The challenge was to create a solution within the IT Service Management system that would result in an On Call technical specialist to be phoned during the out of hours period, but only in an instance of the ‘red alert’ event being triggered.

The monitoring system used by Solidsoft Reply was a bespoke dashboard, developed in house, that would pass synthetic transactions through their backend system in order to judge success or failure. Any single failed transaction would result in an ‘amber alert’ due to factors such as network latency that may impact the transaction timings. Only when two consecutive failures were encountered would a ‘red alert’ event be triggered.

The process to call the On Call agent would only need to be implemented outside of the Operational hours, and not during working hours. Further calls would need to be made to the On Call person in the event that they were not initially awoken by the first call. 

Other on call staff would also need to be notified by text message, and an escalation phone call would be required for the On Call Manager if the event was not reacted to within a 10 minute period, since first being raised.

The Solution

Elements of Freshservice needed to be merged to create a working solution, using the below functions and features:

• On Call Schedules - In order to remove the 'eyes on' monitoring the initial on call schedule was required to align Agents to shift patterns with 3 levels of responsibility, including a High Priority Incident Manager.
• Shift pattern - being that the on call requirement was configured for outside of business hours and weekends the shifts would ensure that agents were only part of a shift rota during those 'out of hours' timeframes.
• Escalation Policy - An escalation policy was required to manage the notification types for the agents, depending upon the Priority of the ticket. Any ticket that required immediate attention would be configured to place a phone call to the primary technical agent and notify other on call team members via text.
• Priority settings - Using the content of the Alerts and Events that were raised by the Monitoring system and logged in Freshservice the priority of the event tickets was determined.
• Workflow Automation - The configuration of the priority settings was then fed into the Workflow Automation process to then ensure that the correct configuration was automatically entered into the ticket when logged.
• On Call Notification - Any ticket that adhered to the fields and priority for an out of hours Priority incident would then follow the process and notify the primary agent via phone, and only during the shift hours of coverage.

 As the customer service did also include a requirement to respond and resolve individual user issues outside of hours then a further set of automation was needed to ensure that such end user issues were not disregarded, but also not resulting in an on call resource being contacted without it being a necessity, as the individual SLAs did not require reaction for hours.

For this we added Supervisor rules and Scenario Automation. By using these features we could provide the customer with an initial response and then allow a certain time period before any reaction was required. Should that period of time fall into the business hours then an active agent would pick up the ticket, but if the allocated time frame for reaction fall outside of the business hours then a notification would be made to the On Call agent, ensuring that all SLAs were met.

An example of the above single user process would be that if a customer logged a ticket at 01:00 on a Friday then the on call notification would not be required, as an agent would pick up the ticket at 07:00. However should the ticket be logged at 01:00 on a Sunday morning then the on call agent would be notified at 09:00 on the Sunday, since the operational support hours do not cover a Sunday.

The full implementation of the On Call notifications resulted in a large cost saving to the customer, with no reduction in the service provision.