Chapter 13

Section 13 - Testing the Plan and Evaluating Results

13.13.1 Purpose

Plan testing is designed to determine:

1. The state of readiness to respond to and cope with a disaster involving the Corporate Data Center.
   

2. Whether the recovery inventories stored off site are adequate to support the recovery of computer processing.
   

3. Whether the Disaster Recovery Plan has been properly updated and maintained to reflect the State's actual recovery needs.
   
   
   

13.13.2 Implementation

The test type will be chosen from one of the following:

1. Structured Walk-Through:  A disaster recovery role-play requiring participation of at least the team leaders and alternates.  The test scenario will be made available in advance of the test to allow team members to review their recovery actions in response to the test scenario.  The walk-through requires 2-4 hours to conduct.
   

2. Unannounced Alternate Site Test:  A surprise test to recover computer processing at the alternate computer facility.  Production processing continues in parallel and is not interrupted.  This type of test generally involves only a small portion of the operations and technical staff, along with selected users and does not require the participation of the entire recovery organization.
   

3. Announced Alternate Site Test:  A scheduled test involving actual recovery of computer processing at the alternate computer facility.  Production processing is not interrupted.  This type of test generally involves only a small portion of the operations and technical staff, along with selected users, and does not require the participation of the entire recovery organization.
   

4. Tactical Exercise:  A simulated recovery exercise, conducted in "war game" format.  All members of the recovery organization are required to participate and perform their recovery actions under surprise conditions and with information that is made available over a period of time (much as it will be during an actual disaster), thus requiring the recovery organization members to respond to the scenario information in real-time.  An 8-hour exercise will usually simulate 48-72 hours of recovery activity.

13.13.3 When to Test

The alternate site tests should be conducted at least once a year with increasing scope as the technical staff becomes proficient in executing the programs and libraries on the test configuration.  After the first 2 or 3 site tests, the tests should be broad enough in scope to include bringing up and operating at least 50% of the critical applications during each test.  This permits the testing of each critical application once per 2 calendar years.

The Structured Walk-Through or Tactical Exercise type tests should be held at least annually to keep the plan current and keep the training level of the Recovery Organization at a high level of operating efficiency.

13.13.4 Developing the Test Scenario

The test scenario is normally developed to accommodate the outcomes and objectives to be derived from testing.  Some of the considerations in the development of the test scenario are:

1. Re-testing the plan segments, which were determined to be deficient in past tests.

2. Testing critical applications that have never been tested or have not been tested recently.

3. Involving those Disaster Recovery Organization Team members that need more involvement to sustain familiarity with their respective functions.
   

4. Ensuring that the test will specify various extents of damage or conditions.  (What files, documents, personnel, etc. are available to the teams to support recovery?)
   

5. Ensuring that testing involves the use of only off-site inventory items to ensure the completeness and accuracy of the off-site inventory.
   

6. Deciding whether the test and the testing parameters will be openly announced or will be a surprise at the discretion of the Director of Contingency Planning.
   
   
   

13.13.4.1 Test Evaluation

In all plan testing, the effectiveness of the test should be evaluated by the Team Leaders, who are defined in the plan, and the Steering Committee. They are charged with the following responsibilities:

1. Familiarization with the overall test plan.
   

2. Understanding thoroughly the objectives of the test to be conducted.
   

3. Monitoring and observing all the activities of the Recovery Teams involved in the test.
   

4. Ensuring that the test objectives were met from the data processing and user points of view.
   

5. Documenting findings related to the strengths and weaknesses observed during the test.
   
   
   

13.13.4.2 Review of Test Results

The Team Leaders, the Steering Committee, and the Disaster Recovery Administrator will document test results as soon as possible, subsequent to the test.  Selected members of the Disaster Recovery Organization will review the test results and resolve weaknesses and problems.  The Director of Contingency Planning will chair the review and coordinate appropriate changes/updates to the Plan.

13.13.5 Plan Compliance Audit

The DR Administrator will fully instruct State EDP Auditing in the interpretation of the Disaster Recovery Plan and will provide a current plan volume to EDP Auditing for use in performing scheduled compliance audits.  EDP Auditing should schedule the independent appraisal of the adequacy of the Disaster Recovery Plan and provide analyses and recommendations to fulfill BIT responsibility.