Tuesday, May 08, 2012

When did I become the "policy" and "process discipline" guy?

When I started in this business (information technology in general, and information security in particular) I was the guy irritated by all the policy and process. The one who just wanted to get things done. The one who thought policy, and process, mostly got in the way.

In many ways I still think that.

All too often, organizations fall back on policy and process rather than thinking, or accomplishing something. People become so captured by the process, that they forget that there is a mission, and a goal, and a task that the process is intended to service; and the process becomes the mission itself.

Then all of a sudden, you're measuring your compliance to the process, and your progress in completing the process; rather than your success in meeting your goals, and furthering the mission; and your organization is failing, but you can't tell why, because your metrics indicate you're extremely successful...

...And you are extremely successful... in following the process.

It is a universal maxim of professional analysis (whether it be business analysis, process analysis, or security analysis) that you will get what you measure.

People will respond to their incentives, and if their incentives depend on meeting a metric, they WILL make sure they meet that metric (if it's possible anyway).

Even if there are no formal incentives, goals, targets etc... When something is being reported on, and someone who matters is looking at those reports, people will make sure those reports look good.

You get what you measure.

So, I have spent a large part of my career warning people, and organizations, against letting the process become the mission; and particularly against managing to metrics.

I believe that entirely.


Policy is important.

Process is important.

Process discipline is important.

Controls are important.

Documentation is REALLY important.

The further I get in my career, the more I realize how true this is... While never diminishing what I said earlier about process and metric capture being a problem; I see more and more that without process, and policy, and  documentation, and metrics...

You simply have no idea what you are doing, or why

Maybe being a pilot has something to do with it. Pilots are among the ultimate in process people, and in process discipline. At least good pilots are.

If you intend to survive very long as a pilot, you learn to do the same thing, the same way, first time, every time (at least once the process is set); leaving room only for changes necessary to your environment, or to improve your processes after EXTENSIVE validation, testing, and training on them.

Doctors are the same way.

There are many things in the military that are the same way.

Anything that presents a strong degree of stress, and a major risk that must be controlled, will exhibit the same pattern.


Because when you are put under stress, you WILL revert to what you have ingrained (or trained, or repeated, or become used to, depending on how formal you are and how much repetition there has been).

So, you better make sure you have something to revert to, or you will be paralyzed.

And you better make sure that something you revert to works, and is kept working.

And that as your situation changes, what you do is updated and you and your people are trained to account for it.

And that it's all documented, and reported on, so you can figure out what the hell you are did and why, when the time comes to return to normal.

Those elements, are the core of process management, process analysis, process improvement, and process discipline.

Ok, so.... what does that have to do with my title?

In the information security business, we have two major... let's call them process drivers.

The first is of course, the reason why you have the process in the first place; risk mitigation.

Our job is to manage risk, document it, place controls and mitigation around it, and develop policy, process, and tools for how to respond to a risk.

You need these, so that when that risk becomes a reality, and you experience an incident, or a compromise... or when you have to take immediate action to avoid one; you know what to do, why your doing it, and what to do if things continue to go wrong.

The second is something that can be an even stronger business motivator, that big scary pair of words: AUDIT, and COMPLIANCE.

Particularly in a regulatorily sensitive, privacy sensitive, financially sensitive, or physically security sensitive (or national security, or communications security, or operational security sensitive for that matter) operational regimes.

In my case, I've spent almost all my career working in banking, finance, medical, military, law enforcement, and government.

Every one of those sectors is sensitive to... basically all of those factors.

Now, I'm in a utility company; which is considered a "critical infrastructure" element by Homeland Security. VERY sensitive to all those factors.

In many environments, operational security grew organically, as an element of each of the operational and functional areas. Server operations took care of server security. Network operations took care of network security. Desktop support took care of desktop security.Often, there has never BEEN a telecom security function, or a document and information lifecycle management and security function, or  an operational risk analysis function, or any kind of cross domain operational security responsibility.

There IS however a heavy audit and compliance burden placed on the operational security functions served by the organization.

In our organization for example have to deal with federal critical infrastructure regulations, federal trade practice regulations, federal market regulations for our market, DHS (we are a "critical infrastructure element"), Sarbanes Oxley, PCI (we process credit cards, handle huge accounts of various kinds etc...), various other federal regulations (and regulators and auditors, and compliance officers), various interstate compact requirements, state regulations (and regulators, and auditors, and compliance officers) in three states; and of course the general privacy and information protection.

All of which must be AUDITABLE.

Audit and Compliance are cross domain.

Aggressively cross domain.

Audit and Compliance people like things to be EASY.

Like everyone else, they don't want to make any more work for themselves than they have to.

Audit folks really do have fairly simple needs from you. No matter how complex the language surrounding what they do can be, or how politically sensitive it can be; it's all logical underneath (at least if they're any good, and your organization has any sanity).

They want everything well documented. For every identified compliance requirement, identified audit risk, identified structural risk, or identified operational risk; they want to see a corresponding policy, a control for each policy, a process for each control, and a reporting and verification mechanism for each process.

Although it's not always absolutely necessary (frequently it is, but not always)... it's STRONGLY preferred that reporting and verification mechanisms include a persistent tracking function (in fact, it's usually called an audit function), localizing the performance of a task, or for that matter any change to an auditable datum, to a specific accountable individual or group.

Then they want at least documentation, and frequently training to support these policies and processes; ensuring that accountable individuals have signed off on their accountability, and have been given the tools necessary to ensure that they maintain compliance.

When security people talk about something being "auditable", that is what we mean.

In my experience, many organizations tend to view audit and compliance folks as unreasonable, a hindrance, or imposing lots of unnecessary or silly or irritating, or just plain stupid and useless work, or process on them.

And sometimes, there is some truth to that. A lot depends on the attitude and training of the auditor, the culture of your organization; and whether the people in question are audit and compliance professionals, vs. people from other fields, or with other responsibilities, who have audit and compliance dropped on them.

Professional auditors and compliance specialists, are business and technical professionals who have the training, tools, and understanding to know that their job is about loss prevention, risk benefit analysis, and risk mitigation; not risk elimination or perfection, which are impossible.

Funny enough, that's a very large part of my definition of what makes a good information security professional.

For people who just have compliance dropped on them... This is something they didn't want to do in the first place, and now they are PERSONALLY responsible for making sure everyone in the organization follows the policies, and documents them, and doesn't screw up, and doesn't send the organization down in flames and OH MY GOD I CAN BE SUED PERSONALLY FOR THIS!!!!!

No wonder they get paranoid, and protective, and overly anal, and can be unreasonable, and adversarial. To these people, everything you do risks their entire lives and careers; multiplied by every person in your organization who has permissions to effectively hold a gun to their head if they don't do their job ABSOLUTELY PERFECTLY.

Which, of course, is impossible.

In my own personal experience, most PROFESSIONAL audit folks are competent, reasonable people, who are more than willing to work with you, providing you don't try to lie, cheat, get away with something, avoid them, or otherwise make their life, and their mission more difficult.

Don't make it "us vs. them", and they won't; and you can all get your job done more effectively.

Audit and compliance can be your best friend, or your worst enemy. As a security professional, or any person of responsibility (system administrator, manager, application owner, DBA etc...), you should look at audit and compliance as a major tool in your arsenal of justification. When you need to spend money, or make politically difficult changes, audit and compliance can be your strongest ally; because believe me, if your management has any brains whatsoever, they LISTEN to what their audit and compliance people say.

On the other side of things, I can't tell you how often I hear "we'd love to do that, but audit and compliance won't let us".

If you have professional audit and compliance people, and work with them, give them the information they need, understand how they do their job, and how important their job is...

..You may not get what you wanted at first, but if you are flexible and mission oriented rather than fixed on a particular tool or technique...

But it's very rare that your audit people won't work with you to help you find a way to meet the business need you are trying to meet, solve the problem you are trying to solve, or otherwise get the job done.

A good, competent, professional auditor, will recognize that in the real world, there are always exceptions.

As IT professionals we know that... sometimes it seems our entire jobs are nothing but dealing with the exceptions (in fact, if you're doing your job well, and have the tools you need to do so intelligently... it should be); and auditors understand that too...

But exceptions raise the final big issue for audit and compliance...

When it all comes down to it though, the absolute most important thing...

The thing, audit and compliance folks want to... in fact need to (because it is the very definition of their job, and they are accountable for it) make sure of...

No matter what exactly you've written down, as your policy, your control, and your process...

What they need to make sure of... Is that what you've written down...

... Is what you ACTUALLY DO...

You might be surprised at how often it isn't... or maybe not.

When there are exceptions to policy, and process, or there are spots where they break, or conflict... These things need to be approved by a person of responsibility, and they need to be documented.

That documentation needs to include an explanation of why the exception is necessary, how long it's necessary for (exceptions should either be temporary, with a definite expiration date, and a person responsible for making sure it's either renewed if necessary... and making sure that "renewed" doesn't become "permanent"... or removed; or they should be written into the core policy), what additional risk may be induced or exacerbated by exception, and how it's going to be managed; and if the primary process needs to be reviewed or changed, you need to include the actions necessary to make sure that will happen, and who is responsible for that.

Which brings me back to what I said way up above...

If you don't have documented policy, and process to support it, you have no idea what you are doing, or why.

Or rather, YOU may know what you are doing, and why... and the people you work directly with might know... But no-one else does.

What happens when, six months later, someone makes a rule in another department that interacts with an exception  made in your department, for an application managed by a fourth department, that interacts with a rule in a fifth department, that ends up breaking everything...

And no-one but Alan knows why the second rule is there, but they have to make everything work RIGHT NOW... And Alan is backpacking in Belize for six weeks...

At which point your senior management go insane, because they're losing money, and customers, and there's bad press... and they don't know who can fix anything, but they do know YOU are right in front of them, and it's YOUR fault if you can't fix it.

Then, when the fire is put out, the REAL carnage begins.

Because I guarantee you, whether you have well documented policy and processes or not, your management is going to hold you accountable as if you did; because even if they told you specifically NOT to have policies and processes or to ignore them etc... When things go wrong, you WILL be held accountable for everything you did, or did not do, and why.

...And when he gets back from Belize, Alan, and Alans boss, and his backup; are all going to be looking for new jobs.

Or, to be more positive about it, take it from the other side.

You are a contractor, brought in from the vendor that supports the computer system that everyone is blaming Alan for screwing up.

Your job is to figure out what went on, and how to fix it; without breaking things work, or breaking things that won't be obvious until they break something else a few weeks later.

If you don't have a full set of policies and processes, that are validated and updated regularly; and which match what staff actually DO; you won't be able to understand what is wrong, never mind how to fix it. You may even have to rebuild a system entirely... what if you don't know the full process, and its 4am on a saturday?

Scoped, managed, documented, and maintained properly; policy and process are your friend.

There is a principle in incident response, disaster recovery, and business continuance; called Continuity of Operations; with the strongly related ideas of Dependency Reduction, and Single Point of Failure Mitigation.

COO is the concept that when things go bad, no matter what, someone will be able to pick up the pieces, and get your organization back and operating at least at some minimum level, within a reasonable period of time.

You maintain continuity of operations, by making sure you have redundant systems, backups, tools, resources, links etc... and the policies, processes, and documentation  necessary; so that any responsible and privileged individual can follow your documented processes, and return to operations from them.

Along with that is dependency reductions. That's just what it sounds like; reducing the number of dependencies in your operations, so that there are fewer things to fail, and fewer things preventing or slowing down recovery in the event of failure.

Single Point of Failure mitigation, or SPOF mitigation; is about finding, understanding, and documenting a particular class of critical dependencies in your operation. The class of dependency which, if a single piece of the operation fails or is missing or unavailable, will crash the whole operation, or cause a major failure, induce a major risk, or prevent a timely recovery.

The point of understanding SPOFs, is to reduce them to as few as possible, then to mitigate the failure modes of the rest which can't be eliminated; usually by providing additional backups and alternates for each function, and a process to cut over to these backups in the event of a failure.

If you don't have good processes, which are properly documented; then you cannot reduce your points of failure. You need to know exactly what you are doing, how, and why; to understand how your operation can fail; and therefore how to mitigate these failure modes.

Unfortunately, all of these things are time consuming; generally the time of skilled employees, many of whom will have no time to complete their normal workstream, never mind deal with extra work around security.

Which is where many people reading this will find themselves right now.

I'm responsible, a the delegate of a person of responsibility; to make sure that our processes all work, and that documentation is created or updated for it; at least those pieces for which we are responsible, and have an audit and compliance requirement for.

So, I find myself being the guy going up to folks, and asking to sit with them to see what they do, and saying "and is there a process for this?" and "is the process documented?" and "who owns the process and the documentation, and can I get access to that?

I'm THAT irritating guy whining about process discipline, and irritating people, and poking my nose in where I'd really rather not have to.

Good god, when did I get old?