Wednesday, June 18, 2008

Trust, but Verify

I've said this before, and I'll say it again: The most intelligent thing Ronald Reagan ever said was
"Trust, but verify".

Expanding on that, it is silly to just say "never assume anything" because we're human beings, we have to make assumptions to get by. If I could give you just one piece of advice in this life though, it would be this:

Never trust your in your assumptions; always verify them.

What follows is a classic example of Making an Ass of U and ME... or at Least ME Anyway.

I've lost my keyring.

No, not my car and house keys (securely 'binered to my belt loops thankyouverymuch); my PGP keyring, along with my private key files.

I have a lot of encrypted communications and other files archived, and generaly, I conduct a fair amount of encrypted communications; though I haven't been doing much of it recently (which is part of the problem).

Flat out, I don't send anything containing PII, PCI, PHI, PLI, or PFI over public networks without encrypting it; and I won't do business or communicate with anyone who doesn't follow the same policy (those stand for Private "X" Information - "X" being Identity/Identifiable, Confidential, Health, Legal, and Financial respectively, in the acronyms above. You may also see them as CII, CPI, CHI, CLI, and CFI; meaning Confidential "X" Information, depending on what auditing and security standards are in use in an organization)

Also, as a matter of security, and in some cases personal preference and/or eccentricity; I customarily exchange correspondence with certain people in an encrypted manner.

This is a pretty big deal, because it means that I can't decrypt my old archived files or communications that use those keys; and because people might try and send me new stuff that I can't decrypt either.

So, how did that happen? You'd think after all that I'd take care to make sure something that important was never lost right?

Well, I've had to completely replace all of my computers over the past few months; starting first with my primary desktop, then my secondary desktop, and most recently my laptop. In the process, I hadn't bothered installing PGP yet on any of the new boxes; until today that is. I was copying a bunch of my archived files over to my laptop, and though "Hey, I should install PGP again".

Thus begins my story of assumptions creating failures.

So, before I stop using a computer (presuming the hard drives are functional of course) I copy all of my files off the drives, onto a backup drive. Usually I also make a backup image of the drive just to be safe, but because I ASSUMED I HAD GOOD COPIES OF ALL MY FILES, I didn't bother this time.

I assumed this, because I did a bulk copy, didn't see any failures, and verified a couple of my files. I ASSUMED THIS WAS SUFFICIENT.

So I wiped my old computers, one by one; each time ASSUMING I HAD A VALID COPY of my keys, on my backup drive, and on my NAS box.

Then, because they were my old computers, no longer in use, I also wiped my backups of them; and destroyed the backup media; because I ASSUMED I HAD GOOD COPIES OF ALL THE FILES, on my backup drive and on my NAS.


I installed GPG on my new laptop, and went to import my keyring... and it wasn't there on the backup drive. Oh the directory was there, but my secring, pubring, seckey, and pubkey files were not.

Ummm... ok don't panic, I've got backups. I ASSUME THE NAS COPY IS GOOD.

NAS box, same problem.

Ok, don't panic, I alway keep an encrypted archive file of my keyrings and keyfiles; and I ASSUME THE FILE IS GOOD, AND UNCORRUPTED.

Uhhh oh.... The archive file on the backup drive is damaged and unrecoverable.

Uhh oh two, the NAS copy is the same file.

Not to worry, I also keep a copy of the archive file on my thumb drive; and I know those keys are good because I used them a few months ago...

Only I replaced my thumb drive a little while after I had last used the crypto stuff on it; and didn't bother copying the file over at the time, because I had copies of it on my laptop, my desktop, my backup drive, and my NAS drive; and I ASSUMED THEY WOULD ALL BE GOOD, AND I'D BE ABLE TO COPY THINGS OVER LATER.

Now remember, I had every reason to believe everything was good. My PGP was functioning on my laptop and the other computers, right up to the day I wiped them. My problem was that I assumed that keeping a straight backup copy from one of those computers was sufficient, and I never verified the validity of my assumption.

See you have to remember, that a copy is a copy; and it may be imperfect. Just because the source is good doesn't mean the copy is.


What I presume happened is that my backup copy didn't copy those files, because they were in use and locked at the time; and I probably just suppressed the error (some bulk copy command lines don't output error messages).

I verified several of the files from the copy, but I didn't do a file for file verify, because I was just copying, not using backup software.

Then I wiped my completely functioning PGP keyrings off those boxes when I decommissioned them.; without ever verifying that I had a valid copy.

Then I compounded my error, and overwrote the copy on my NAS box with the bad copy from the backup drive; without ever verifying.

Then I wiped my old thumb drive, because I assumed all the other copies were good, without ever verifying.

Now some might ask, "why did you wipe all those files. Even the backups"

That's simple, every copy you have increases the chance of compromise; so keeping as few copies as possible is just good security practice. Four is a standard; one online (the active copy), one in online backup (the backup drive), one in near line (the NAS copy) and one in physical archive...

Wait a sec, what about physical archive?

Unfortunately, no; all my physical archives for my working set files (the non machine specific files that I share across all the machines I work on) are done off the NAS or nearline backup, and rotated out periodically; or are of full system images, which I destroyed after I wiped those machines.

I checked them anyway, and finally found an OOOOLD key archive that has the private key... only one problem...

I've changed the passphrase a couple times in the last few years; and I forgot the old passphrase that I originally used when I generated that key. I've cycled through my usual metric for creating passphrases, and it isn't any of them; so I can import my keyring, but I can't use my own private key, and since I never set a revoker, I can't revoke it either (you can't revoke an invalid key without the passphrase).

So my friends, remember, an unverified backup, is WORSE than no backup at all; because when you have no backups, you are careful about making sure copies of critical data exist before you delete things; whereas with a bad backup, you might be just a bit careless, as I was, and end up losing your data.