The Challenge
The Public Utility Commission (PUC) of Texas’ has implemented a requirement that utility providers provide redundancy for Advanced Metering System (AMS) and Market Operations applications in the event that the utility’s primary data center becomes inoperable. In response, a Fortune 500 utility company established an Enterprise Disaster Recovery program committed to enabling a solution that would ensure fewer than 59 minutes of downtime.
The company quickly realized that its existing data center was significantly underprepared for integration into a multi-data center environment. Not only did a second data center need to be created to mirror the first, but the first needed to be completely upgraded to allow for new capabilities including storage, network, security and platform improvements.
The creation of a second data center created other challenges within the IT organization as a whole: First, the company would have to staff and maintain two data centers, and second, the company would need to be capable of shifting all operations to the secondary data center in the event of disaster. These challenges meant introducing new technologies, new processes, and new responsibilities to the IT team.
Our Approach
The company engaged Sendero to manage the entire life cycle of this program. The effort started with the building of the second data center facility, and ended with delivering the final product that enabled applications to be orchestrated and moved between data centers. The solution had to meet the Recovery Time Objective (RTO) of 59 minutes. Working with the company’s stakeholders to manage the software, hardware, and infrastructure technology, Sendero actively led project cutovers, documentation gathering efforts, and strategic planning sessions to ensure an effective solution.
Sendero led many change management efforts including training, demonstrations, knowledge transfer sessions, and documentation buildout. To ensure the IT organization had a lasting artifact it could reference to understand the Disaster Recovery solution, Sendero created a single document that detailed the step-by-step processes to follow in the event of a disaster, specific to the roles within the IT organization. The documentation also covered the application and infrastructure steps required to ensure minimal downtime during the failover process. This playbook provided each IT group with an understanding of how the groups were interrelated, with references to detailed vendor and project documentation.
