Your Systems Management Lysine Contingency (or Abort! Abort! Abort!)

In the course of our duties as Sys Admins I'm sure we've all "pushed the wrong button" a few times. If you haven't yet, you probably haven't been a Sys Admin very long.

About 10 years ago while working on a project which was wrought with political and technical paranoia I decided to prepare my own "Lysine Contingency" in the event that the political hailstorm became too much for me to handle. I first heard the term Lysine Contingency (LC) from Jurassic Park. I was always intrigued by the idea that Samuel L.'s character introduced. Certain destruction to the park's dinosaurs if a problem occurred that was simply unmanageable.

A Lysine Contingency is not your normal Change Management Rollback plan. It is basically you screaming "MISSION COMPROMISED! ABORT ABORT".

So, I want to offer some things I have learned about Lysine Contingency plans.

Points to consider:

1. How was the change implemented?
   • Were there multiple execution points such as Group Policy, Software Deployment, Login Scripts, Manual Installations, cron jobs or other local schedulers?
   • Which execution point can be used to produce the greatest yield on your rollback? If you can hit 80% of your computers in 20 minutes with Software Deployment vs. 1 day with Group Policies then it's obvious where you should place your focus.
2. Can the change be rolled back or otherwise mitigated without affecting other production systems, users or applications?
   • Does your change require a reboot of the target system or application?
   • What, if any, are the dependent services or applications?
   • Will any data be lost? Are there other methods of removing the change that may take longer but save the data?
3. Do you need the cooperation of any external entity?
   
   • Are you allowed to deploy software but you're not allowed to modify GPOs? Are the GPOs part of your LC?
     
   • The best Lysine solutions are the solutions that don't require anyone else but you. In a serious do-or-die situation you don't want to rely on the guy who's always on a smoking break or the lady who has more drama in her life than the evening lineup on Lifetime.

I want to stress: I'm not talking about proper Change Management procedures here. You should always have a rollback plan for major changes in your computing environment. I have had to rollback many changes to my various environments over the years, however, I have only had to enact a Lysine Contingency once and this came over two years AFTER the changes were introduced! Remember, the issues that often require your Lysine Contingency are just as easily going to be political as they are technical.

In the case I mentioned earlier I had my entire LC whittled down to one script. When the alarm sounded I knew that I had prepared for this scenario and there was no need to script some uninstallation routine or quickly write a new GPO or even to hold a meeting. This script set one flag which my verification utility read as "get the hell out of here!". With this flag, no more new verifications or installations would take place via GPO. The next section of my script deployed a Tivoli Software Package which immediately uninstalled the utility on all Windows computers. The next section added some log files to a central server. These files detailed the actions taken and would, ultimately, record the success rate of the Lysine Contingency.

Within 30 minutes the only traces of the original changes were contained in logs which I promptly gave to my manager. The Lysine Contingency went off without a hitch so that when the musical chairs music stopped I wasn't the one left standing.

Follow me on Twitter @ShaneCorellian