Change Control and Being Difficult to your Customers

“I may not be the project manager the customer wants, but I’m the project manager the customer needs.”

OK, so most PMs aren’t exactly Batman, but maybe they should be more like Batman.  Think about it, the customer wants their functionality in Production, bug free, as cheaply and quickly as possible.  But if any of those requirements I just mentioned are taken even one iota to the extreme, the entire project could easily fail.

This particular, rather balmy, Dallas winter’s day I want to speak about Change Control.  That’s right, the big CC.  Friend of the developer and troubleshooter, enemy of customer relations.  “Hold on” you might say.  How is Change Control the enemy of customer relations? Isn’t Change Control designed to increase quality?  Doesn’t it assuredly mitigate against inadvertant rollbacks?  Doesn’t it essentially guarantee that the client’s environment is like a well maintained library with every items catalogued, indexed and easily locatable and traceable throughout its lifetime?  OK, time to wipe my drool from the keyboard, but I do love a well organized environment (well, at least in computers).

All that aside, Change Control has one flaw, one weakness, one Achillean heel (Is that an adjective? well, I guess it is now).  Small changes are no quicker to deploy than massive overhauls that optimize every line of code in the system (assuming your deployment system handles all the dirty work for you).  And I don’t mean that the single file change on the one server that handles every millionth transaction and sends the end user a virtual poodle sticker takes exactly as much time as the database reorg that requires every index on every table to be rebuilt.  That would be silly.  What I mean is that in real world terms the time between finishing the development and QA work on a change and it being in production is likely the same for both of those silly examples I just mentioned.  A maintenance window will still need to be scheduled; that window will likely not vary in length due to SLA requirements and the amount of lead time required to schedule such a window will also likely not vary for the virtual poodle change as compared to the big reindex change.

That lack of variance will likely upset the customer.  This should be expected as it is unfair to expect that the customer should understand why all of this red tape is keeping their virtual poodles from congratulating their millionth users’ trasactions.  I mean after all, it is likely just a configuration change to change the 10,000,000 to 1,000,000.  Why oh why do we need to schedule 6.5 business days in advance and why do we need to tell the end users that the system will be down for maintenance for up to 8 hours between 1 AM and 9 AM Sunday morning.  How are the night owl end users supports to order their faux fur lined boots (faux fur optional) at 3:28 AM Sunday morning?

But Change Control is the only thing keeping the dark gods of Chaos at bay.  Without change control, that configuration change could be made at 3 PM on a Tueday and the system would likely give zero airborne murine rumps.  However, how about the next time a deployment package is executed?  Would that configuration change stick or would it roll back to the clearly inferior ten million value?

Silly example aside, any change that might be rolled back accidentally and which cannot be identified and traced through the lifetime of the system should not be allowed onto a Staging or Production system since such changes introduce chaos into the system.  Such changes might allow a customer to see functionality faster, but they form the foundation for a process that eventually becomes unmanageable without subject matter experts.  And since those SMEs are humans who quit, are fired, retire, move on to other projects and more importantly forget why the 14th bit on that permissions mask is a 1 and not a zero, those SMEs should not be part of the process.

The next time your clients complain that a one line change takes three weeks to make its way into production, ask them if they are fans of the gods of Chaos.  Then after they either laugh or look quizically at you, explain the situation and the general reason why Change Control is a double capital term and needs to be followed (oh, and keep working at getting those maintenance window times lower and lower).