Wednesday, October 30, 2013

Metagaming, Fandom, and Gaming Theory: Acquisition, Engagement, and Retention

Boston Sports 
Since Y2K the Patriots have won 3 Super Bowls, the Red Sox have won 2 World Series, the Celtics won the NBA Championship, and the Bruins won the Stanley Cup. In addition since 2000 Boston College won 4 NCAA Hockey Championships and Boston University added another. 
The above doesn't even count this year's Red Sox team which is currently leading the World Series 3 games to 2 with the Series returning to Fenway Park. 
...since Y2K every major sports team in Boston except the Red Sox has also lost in the finals of their sport. If the Red Sox end up losing you won't hear me complain. The team has already delivered way more than any fan had a right to expect his season. 
Besides - given all that has happened in recent Boston sports who would have any sympathy for us? When it comes to sports opulence - we has it.
The title of the post reflects the three primary success vectors of a game, and thus the primary missions of a game designer: player acquisition, engagement, and retention.

While they are important to any game, player acquisition is the most important success vector for "one shot" games (also called "standalone" games). These are games where you pay for the game up front, and then don't pay again, so it doesn't matter how long, or how often you play.

Of course, in these games, engagement is still important; because having a "good game" with an engaged user population means that you get good reviews and word of mouth, for a longer period of time; increasing your player acquisition (and thus sales).

With very good engagement, you may be able to create a franchise; thus increasing your success with other installments in the franchise (sequels, expansions etc...). However, in creating a franchise, you effectively change your standalone game into a persistent game.

Persistent games, are a somewhat different story. These are games where the player is expected to play many times, for an extended period, or both; and whose success depends on having a large player population These would include casino gaming (particular slot machines), free to play games (which earn money by either advertising or small in game premium purchases), "many replay" casual games (candy crush anyone?), and persistent world games like MMORPGs.

Both acquisition and retention are particularly critical to these games; and retention is achieved through engagement.

The way game designers accomplish these missions are with spectacle, and reward psychology (positive and negative reinforcement through anticipation, reward and penalty; with a very strong bias towards reward, leavened by the occasional penalty), particularly competitive reward psychology.

Something spectacular engages you for the duration of the spectacle. You are a passive participant. It attracts you, and fascinates you; but only for that moment. Retention requires maintaining engagement over time... becoming an active participant, either directly or as a metaparticipant.

So... what does that have to do with sports? Or with spectator sports fans in particular?

Simple... Sports fans are players in a metagame.

Spectator sport fandom, although passively received (the fan isn't an active participant in the games they are watching); isn't a passive, receptive, entertainment experience (like a movie or television).

However, much as television shows retain viewers by emotional engagement in the story (thus making them metaparticipants in the narrative); spectator sports retain fans by persistent emotional engagement with the sport, and particularly with their team (making them metaplayers in the game).

Sports fandom, is a kind of play by proxy; much as horse racing, and other betting games (roulette for example) where the players interaction with the game is not part of the gameplay. This makes it a metagame.

And metagames have the same success vectors as any other game.

One of the things that makes Boston sports fandom so... passionate and crazy I guess is the best way to put it... is that a Boston fan is being fed with a near perfect reward psychology cycle.

Boston teams win often enough (and often quite excitingly) to attract attention and generate spectacle. This  acquires new fans (or brings back those whose engagement has weakened); and it presses the "happy button" in existing fans, engaging their reward pleasure mechanism.

Importantly though, Boston teams don't win so often that fans get victory fatigue, and need reward escalation to maintain engagement.

When they're NOT winning, Boston teams are rarely just mediocre... they tend to alternate between "oh God so close..." and "total abject failure" (at least psychologically if not objectively). It may seem counterintuitive, but this is actually far more engaging than consistent high performance or even consistent victory.

In terms of gaming theory, this 3 point cycle (victory, near victory, failure) helps create spectacle to attract and acquire participants; and helps create, reinforce, and increase engagement.

Very importantly, it also helps maintain engagement (and thus retention) by reducing victory fatigue, anticipation fatigue, and expectation escalation.

So... getting into that second and third part...

Retention is achieved through continued engagement. When engagement is weakened or broken, you lose participants (gamers, fans).

Engagement is created, reinforced, and increased; with spectacle, novelty, fascination, and competitive reward psychology as described above.

Engagement is weakened or broken and you lose participants (gamers, fans) through frustration, demoralization, boredom, and fatigue.

So, the challenge is to maintain or increase engagement over time.

In general, you deal with boredom and fatigue, through novelty. Change things up, so that a participants experience, expectations, and emotional engagement with the game are maintained, and thus they are retained.

I mentioned victory fatigue above, but didn't define it, I should probably define the three elements of "game fatigue" now.

Victory fatigue is what happens when a player receives too many rewards, or wins too much too easily. This tends to cause boredom, and frustration; because the rewards no longer feel like rewards. This weakens or breaks engagement.

In an interactive game you can deal with victory fatigue (and to a lesser extent anticipation fatigue) by varying gameplay (introducing new and different ways of earning rewards) increasing challenge (NOT just increasing difficulty, though that is one way of doing so), increasing penalty for failure (though you can't do that too much or you break engagement through frustration and demoralization), varying rewards (making the rewards new, interesting, and different), or by increasing intensity or spectacle (making the rewards bigger or more desirable). These mechanisms keep the players anticipation and pre-reward engagement high, and their reward pleasure mechanisms responding strongly to the rewards.

In most spectator sports however, you don't have those mechanisms available to you (or they are severely limited). The difficulty and rewards do escalate somewhat over the course of a season, but are basically fixed year to year (win a game, win a conference, win a division, win a playoff game, win a championship game). So, frequent and consistent victories, particularly championships, result in expectation escalation.

The three major expectations to this issue of fixed challenge and fixed rewards by the way, are motor racing, premiership style football (soccer), and NCAA football and basketball. Not surprisingly, the first two are the two most popular spectator sports in the world; and the third creates a degree of unreasoning passion far greater than any other sports in America.

Anticipation fatigue is a more interesting issue. When you get that "so close" feeling too much, it actually tends to discourage and disappoint you, which increases frustration and breaks engagement i.e. "they get our hopes up every time then disappoint us every time... what's the point".

Expectation escalation, is what happens when performance or rewards consistently exceed expectations (or consistently exceed the mean performance of a peer group).This causes people to "reset" their emotional expectation of what poor, acceptable, and excellent are, such that their median level of performance, even if it is objectively far better than average, is simply "expected".

So, a team that wins 80% of the time, year after year, will eventually be expected to do so. If that team starts to win consistently less than 80%, even if they are still better than most teams and win 60% of the time; the emotional reaction of their fans will be the same as if they had objectively poor performance, rather than simply "less good".

Lesser success can feel like failure, when you're used to greater success.

Cycling between "not quite great", and "really bad" (even if "really bad" is actually mediocre statistically, the victories and near victories redefine emotional expectations such that mediocre FEELS like abject failure), actually creates and reinforces engagement, and passion; far more, and far more intensely, than consistently high performance.

This by the way, is the exact same reinforcement cycle that creates and reinforces addiction. Reward (the high), anticipation (the process up to the high), and penalty (the come down and the jones).

So... for Boston fans, it's like vegas slot machine designers were controlling things for optimum fan acquisition, engagement, and retention.

It's an almost perfect metagame... arising without design... which is kinda neat.

Monday, October 28, 2013

Somebody has been reading too much buzzfeed

Execution Failure

Many libertarians... and these days many conservatives... have a problem with public education.

I'm one of them.

But unlike a lot of libertarians, I don't have a problem with the CONCEPT of public education... or at least publicly FUNDED education.

Public education, truly is (or should be) a public good, and is of significant benefit to all in society. A free society, that respects the rights and liberties of all, requires an intelligent, educated, and productive populace. History has shown that a poorly educated populace is the largest step down the road to tyranny.

So, I have no problem with the concept of publicly funded education.

I have a HUGE problem with the execution.

Or rather, I have a number of huge problems with the execution.

Like the fact that at least 3/4 of the dollars spent on "education" don't end up in the classroom.

Like the fact that DC public schools spend $29,409 per student (for the 2009-2010 school year. 2012 info in the link below) yet are among the worst in the country.

More money won't fix the problems.

Every additional dollar we spend on education, gets soaked up by the large mass of the public education industrial complex rather than actually being spent on educating students. Just like with federal financial aid for college... the more dollars the feds pump in, the more the schools raise their tuitions.

Oh and lest my liberal friends think the source for that almost $30k number is biased, here's Huffpo talking about how D.C. lied and said they only spent $18k per student in 2012 when in fact they spent $28k... HUFFPO...

From that same article, you can see class size is NOT the problem. Staffing is NOT the problem.

In D.C. public schools (which have among the highest teacher and administrator salaries in the country), there are 11 students per teacher...

 11? What about the "if republicans had their way there would be 40 students per teacher" crap?   about "Our classes are so big students can't hear the teacher?" Absolute garbage. Pure fraud.

The 50 state average for public school spending today is $12,500. Many states spend double that. DC is spending almost $30,000 a year.

Most private schools don't cost that... Sidwell friends, the most exclusive high end private school in D.C. costs about that much actually...

Public education in this country is not UNDERfunded... it's MISfunded. The money isn't going where it needs to go. It's subsidizing failure, and punishing success.

Try signing up in Icelandic?

Removed the embed due to un-disablable autoplay, but the link is worth watching: SNL Parody of Kathleen Sebelius discussing Obamacare website

OSX Mavericks Quick Review

The one line summary:

You probably won't notice anything that's worse, and you probably WILL notice a few things are a little better... and a couple things that are a LOT better.

Those couple things are the finder (tabbed finder FTW), the activity monitor (WAY more useful now), much better power management (a little to a LOT longer battery life, and less fan time at lower speeds; though they still don't allow granular control over sleep and hibernation. I use SleepLess for that), the slightly less irritating notifications; and the slightly to considerably lower cpu, memory, and power usage, in high overhead situations (when you've got lots of stuff open and idle in the background).

It also seems to boot a tiny bit quicker, and the virtual memory access patterns seem a bit better (my disk I/O indicators aren't flashing as much in low-medium workload situations). Scratch that... after a few hours, my virtual memory utilization is WAY better, as is my overall memory utilization and management.

Oh and multi-monitor setups are WAY better and more useful now, particularly if you're using an HDTV as a monitor.

I'm an android user, so I sync my contacts with google not iCloud (and my notes with evernote, and my files with g-drive and dropbox, and my passwords and bookmarks with another multi-platform independent service etc... etc...); and I exclusively use webmail and web calendars, so the changes there don't really impact me at all, and I can't give my opinion thereof.

I use Chrome and Firefox not Safari... if Google would make a native OSX 64 bit Chrome that would support appnap and sleeping idle plugins and tabs... that would be really nice. Until then... meh.

Also, iWork for free with new Macs.. cool I guess... I don't use it, again, can't give my opinion

Oh one bad thing... 

On my wife's mid 2010 13" MBP the install was so slow, with so little feedback (it stayed at "about 7 minutes remaining" for over 20 minutes), that I thought it had frozen, and I restarted it. It turns out it was just in the middle of uncompressing a REALLY large file, and it took like 15 minutes to do so.

I figured it out because my early 2011 top end 15" MBP uncompressed the file in like 5 minutes and then moved on. So I started the install over on the 13" and watched the install log. I let it keep running and eventually it finished the install just fine.

Actually two things...

Downloading it sucked. It took 2 days to download 5 gigs, and I had to restart the download multiple times. Mac Appstore downloads in general are slow and glitchy, especially for apps over a few hundred megs.

Oh and a feature request... 

Apple, next time include a free program... in fact make it an option in the installer... to create an installation USB drive or somesuch?

I have no problem donwloading a third party app (DiskMaker X works great), using a commandline hack, or extracting the install image from the app and imaging it onto a USB drive... but I'm a technology professional. Most people don't know how to do it, and they don't want to have to download 5 gigs for every computer they own, or when their computer dies have to download the OS again from the recovery console (allowing a reload from disk there would be nice too).

Thursday, October 24, 2013

Useful Complexity

...And even then it will be cracked, because GPU based cracking and cracking method optimization, have reduced the time required to crack the entire passwordspace for most passwords down to a matter of minutes, or at worst hours.

According to several recent articles in various industry publications and websites, approximately 85% of all Windows passwords can be recovered in less than 60 minutes, and more than 90% within 24 hours, using only a single multi-core cpu, multi-gpu computer (basically a high end gaming rig).

Using small clusters of multi-cpu many-gpu systems (basically, spend $20,000 on off the shelf hardware) the entirety of the 8 character Windows passwordspace (all possible 0-8 character Windows passwords) can be cracked in a few days, or less.

With the computing power available today, the only useful thing high password complexity does, is make your password harder for a human to guess.

...Unfortunately, the bad part is, that also makes it harder to remember, and harder to enter.

Here's the level of minimum password complexity that is actually useful: 

8 or more characters, not forming any dictionary word or combination of words (including letter substitutions), and including at least one special character.

Anything else is just making your users life more difficult, without actually making them any more secure in the real world.

Ok, so... why is this the "useful level of complexity" ?

Because in the real world, an 8 character password, without any dictionary words or variants on dictionary words, and including at least one special character, requires a cracker to use the entire characterspace to crack your password.

Wait... what? No, that's wrong isn't it? There's 128 ASCII characters, or 255 in the extended character set right? Upper and lower case alphabetic characters, numerals 0-9 and a whole bunch of "special characters"... All of those can be used in passwords right?

Well, yes, theoretically the possible characterspace is 255 characters (or 256 for ISO-8859/UTF-8 encodings).


In reality, it's not. First thing is that most password systems don't allow the entire 8 bit characterspace.

While it is theoretically possible to use the entire 8 bit U.S. character set (extended ASCII or UTF-8) in a password (or even to use a multibyte character set), it requires special keyboard codes, and these characters are difficult to enter. Further, most mobile devices do not allow you to enter characters other than those on the standard keyboard (or make it very difficult to do so).

There are 94 or 95 characters available on a standard US keyboard (depending on whether you count the nonbreaking space i.e. the space bar): 10 numerals, 32/33 special characters, and 52 letters (upper and lower case).

By the by, these are generally referred to as the "printable characters", with the remaining characters referred to as "non-printable".

Even if you wanted to use them, accepting that they are difficult to enter and mobile devices may not be able to enter them at all... most password systems exclude unprintable characters, leaving a maximum of 95 possible characters.

For those password systems which allow the non-printable character set, they generally limit passwords to the 7 bit basic ASCII character set (or sometimes ANSI-1 or UTF-7, which are technically different, but include the same characters), which is 128 characters.

Oh and yes, there are non-us character sets, even multibyte character sets that include many thousands of characters, and it's certainly possible to code a system that accepts all of these characters.

... but no-one does.

Even computing systems that accept large character sets for text input (those supporting the Chinese GB18030 standard for example, or a full implementation of UTF-32, with over 1.1 million possible characters), generally only accept a limited subset of characters (usually UTF-8) for passwords (because you can't guarantee compatibility with large character sets across all hardware and software combinations).

So yes, the theoretically possible characterspace is actually many more than 255 characters, but the 95 keyboard characters comprises the entirety of the passwordspace most people might actually use.

Oh and many password systems exclude some or all of the characters !@&*$?/|\ and almost all password systems exclude the nonbreaking space (the space bar), because they can cause problems with parsing. Some actually exclude all special characters, but this is rare now.

What it comes down to, is that the "normal" characterspace is 94 characters.

That would seem to make it even MORE important to use case shift, and numerals; as they comprise 38% of the available characters.

In theory just using lowercase and special characters takes 36 of those 94 characters out, meaning that crackers only need to use 72% of the characterspace to crack your password.

...In theory, it would be better to make them need to use 100%...

...but in reality it doesn't work that way.

Okay... why doesn't more complexity increase security?

At this point, the computational power of multi-gpu cracking system, is enough so that in any serious cracking run, crackers can include the entire alphanumeric space without undue penalty; so including numbers and case changes can help a bit, but not much.

The first cracking run on a password will be optimized for high speed, and will include an optimized dictionary, and tables of common dictionary variations and substitutions (substituting 3 for E, @ or 0 for O etc...). Combined with a full lowercase alpha only run, that only takes a few minutes, to at most a few hours, for the entire 0-8 character passwordspace.

From there, crackers go to brute force, with or without optimizations. The first thing they're going to do is add in the full alphanumeric space, before they add in special characters; and any run that includes special characters will therefore almost certainly include mixed case and numerals.

That means that in a bruteforce attack, whether you included mixed case and numerals in your password or not, the cracker will still try all of them as if you did, and therefore it will take just as long to crack your password as if you did include them.

So, any password with a special character is likely to be slightly more secure than those including numbers and case changes, and unlikely to be less secure (presuming equal length). To put it another way, using a special character (or preferably more than one), has a higher expected security value, than using mixed case and numeric characters.

Yeah, it MIGHT take longer to bruteforce your password if you've got all 3... but your password is going to be one in a hundred, or a thousand, or a million, the cracker is trying to crack all at once; and they're going to run the entire mixed case alphanumeric space, before they even start adding special characters.... and these days "longer" is a few hours, or at most a couple days, not "more than 30 days".

So, unless your password policy is that users change their passwords every week (and that would be a huge support nightmare, causing more lost productivity than any value doing so might provide)... adding any more complexity doesn't significantly increase the security of a password; but does significantly increase the trouble to your users.

Include more complexity if you want to... but don't make it a requirement.

My personal recommendation for how to create good passwords?

Using the first or last letter of each word (or better, both the first AND last letter) in a phrase, poem stanza, song lyric, or other memorable passage, combined with special characters; is generally a good way of producing a pseudorandom non-dictionary string that is of sufficient length to provide reasonable security, but which you can still actually remember.

Include more than one special character, and don't make the specials ONLY the first, last, or middle/joining characters in the password. Also, don't make the only special characters you use, common letter substitutions like $ for S, ! for I etc...

All of these are common optimizations which crackers use to reduce the time it takes to bruteforce a password by the way. Not doing them forces the cracker to bruteforce the entire passwordspace, not just the MUCH reduced optimized space.

Going to more than 8 characters is actually useful, if the password system doesn't drop or ignore the extra characters (many do).

More than 16 characters generally isn't useful for a pseudorandom password, because 16 characters using the 94 character passwordspace, is essentially uncrackable at this time (it's computationally infeasible within a reasonable time horizon). Really any complex pseudorandom password with 12 characters or more is likely to remain computationally infeasible for at least 10 years.

Telephone company studies to determine the ideal length of phone numbers, figured out that human beings are pretty good at remembering strings of 1, 2, 3, and 4 characters, and combinations of those strings (2+3=5, 3+4=7, 3+3+4=10 etc...); with 3 and 4 character strings being the easiest to remember due to something they called "memory chunking" (the human memory seems to run 4/4 time).

Those same studies showed that humans are generally bad at remembering strings of other lengths, more than 4 strings total, and more than 13 characters total (with optimal recall at 3 or fewer strings, and 10 or fewer total characters).

Given that, I say make your passwords 9-12 characters long, with at least two special characters. You can improve your password strength dramatically with every additional character up to 16, but you trade off on memorability.

The standard recommendation is to use a different password for every account; but given the huge number of accounts people often have these days, it seems unrealistic to expect them to remember that many different passwords.

One solution is to use a password manager, which will create a unique strong password for every account, and store them, requiring you only to enter the strong password you created for the password manager itself.

Another solution is to create unique strong passwords for your high security impact accounts (those with banking, credit, legal, and healthcare impact for example), and then to have several other passwords that you use for other security levels, having just one for each level, but changing them at least every 90 days.

Whatever you do, it's always a tradeoff between length and complexity (increased entropy), and memorability and easy of entry.

Speaking of length and memorability... passphrases?

If the password system in question doesn't drop or ignore characters beyond 8, 12, 16 etc... you can also use longer passphrases instead of pseudorandom passwords.

At first glance, this would seem to be an easy way to have a memorable password that is still very long; which is true, but there are some major issues with passphrases that make each character in added length of much less value than in a pseudorandom password.

Multi word phrases using common dictionary words are less secure for an equivalent length, than pseudorandom passwords with special characters, simply because the possible solutionspace for each is very different.

With an 8 character password, in a 94 character passwordspace, there are 6,095,689,385,410,816 possible combinations of characters. There are only about 30,000 8 letter words.

There are between 250,000 and 400,000 words in the english language (depending on what words you count and whose estimates you believe). The average English speaker however only knows 20-40,000 words, and only uses about 2000-4000 words regularly.

Further, English words exhibit very strong letter frequency patterns, which are well understood in statistical analysis (in fact that understanding is critical for cryptanalysis). For example, the average english word is 5 letters long, and more than 80% of english words contain at least two of 6 most common consonants, and at least one of the vowels e, i, or a.

Reducing the dictionary set to common words of 8 letters or fewer, brings your wordspace down from 400,000 to something like 100,000.

These characteristics dramatically reduce the total entropy of passphrases; and dictionary optimized bruteforcing, based on common words, and english letter frequency, can be many orders of magnitude faster than a straight bruteforce.

Essentially, each word in a passphrase provides less than the entropy of a pair of pseudorandom characters.  

In fact, given the reduction in entropy inherent in using dictionary words; if you are going to use a passphrase without increasing the complexity, I would recommend at least 8 words and at least 32 characters (not including the non-breaking space. Longer words are better).

... which really means you should be increasing the complexity. 

The first and most basic thing, is to use at least one word 8 characters or longer, preferably an uncommon one (say... antidisestablishmentarianism for example). This makes the wordspace required to crack your passphrase DRAMATICALLY larger (the average English word is 4.5 characters long. Going from 4-5 character words to an 8 character word increases the cracking space from around 40,000 to over 150,000 words).

Passphrases should include as much of the full 94 character passwordspace as possible; using mixed case, multiple special characters (punctuation is good for that, but because spacing between words is common, it has a lower expected value than other special characters), and if it is easy to remember, and makes sense, numerals. Also, using a special character substitution in more than one word here provides a dramatic increase in entropy that is very worthwhile, particularly if it's not a common substitution.

I would also recommend replacing (os letter substitution with) one or more dictionary words in the phrase with a pseudorandom string. For example, use the first two and/or last two words of the phrase to create a pseudorandom string with the first and last letters.

Increasing word complexity and adding pseudorandom strings to a passphrase of any length more than 5 words or more, and at least 20 characters should make it functionally impossible to bruteforce.

Common words of 3 characters or fewer are actually easier to bruteforce than single additional pseudorandom characters. So you want to average at least 4 characters per word... preferably 5 or more (more than the average word length).

Oh and as spacing is predictable in standard English phrases, make it unpredictable. This results in combination words that together are harder to brute force than the multiple individual words with spaces would be.

Basically... if your pass phrase includes "the" and "end" you should make sure that you've got two 6 letter words in there and make it something like "Always-beTTer intheenD!" (which would be functionally impossible to bruteforce any time in the forseeable future).

At that point you have the same entropy as a pseudorandom string of the same length... it's just easier to remember.

Friday, October 18, 2013


4x4 heavy mustard fried only, extra toast, pickles on the side, fries light-well, black and white shake.

A not so brief explanation and history of U.S. federal government debt

So, a "debt deal" has been reached... which most on the right are characterizing as "the republicans caving to the democrats"... which is frankly bull.

You can't fairly characterize doing what they always intended to do, and everyone who knew anything about the situation other than slogans knew they were going to do in the first place, "caving".

This whole "shutdown" exercise was nothing more than a PR and fundraising exercise for the 70% or so of each party who have completely safe seats; and an attempt to "challenge from the right", the 15% or so Republican seats that are completely safe for the party, but for whom the voters may be persuaded to choose a different Republican.

This is not to say that U.S. federal debt isn't a serious problem, of COURSE it is... just that no-one in Washington was ever going to try to actually do anything about it.

To do so would require cutting spending; and regardless of what voters claim to be for, very few of them would stomach actual cuts to actual programs they like or approve of "only those unnecessary things the other guy likes".

Congress is not there to govern, they are there to acquire and spend money, in order to get votes. If you don't understand this by now... you probably shouldn't bother reading the rest of this.

They are SUPPOSED to be there to govern yes, but they pretty much gave up on that, some time between February 3rd, and May 31st, 1913 (look it up).

So, to be clear, the alternatives here were not "raising the debt ceiling" or "cutting spending"... because no-one was ever even approaching the idea of making meaningful cuts in spending.

The alternatives were "raising the debt ceiling", and "continuing to manipulate the currency and using accounting tricks to pretend that we aren't raising the debt ceiling".

Oh yeah... the fourth option there, "government default" wasn't ever going to happen either. The likely consequence of the federal government defaulting on debt right now, would be a global financial panic followed by a global depression.

Ok... so, given that what's the basic situation right now?

As of today, the U.S. federal debt stands at $16.75 trillion USD, and increasing about 2.7 billion every day. We've actually technically been over the debt ceiling of $16.7 trillion for about 4 months, but we've been using accounting tricks to avoid "officially" breaking it.

Right now, we're taking in about $14 billion per day, and spending about $17 billion per day.

That's in direct spending of course, and doesn't cover unfunded liabilities (those are MUCH larger).

Given today's "budget" (we haven't had an actual budget in 4 years, just long series of continuing resolutions and special appropriations... but that's another post) they're going to have to increase the debt ceiling 1.6 trillion to get us through another year.

So, this time next year expect the debt to be $18.3 trillion.

How exactly did we get here?

The short version?

Spending more than we took in... in some years only a little more, in a couple years slightly less, but in many years FAR more than we took in.

The long version is very long... but I think very illustrative and useful to know.

The LOOONG version

Ok guys this is a really long post full of mostly numbers. If you want the upshot, it's at the end. But if you would like to know just how much we've been screwed, and just exactly who did the screwing... Well, the devil is in the details.

First, the earth cooled, then Woodrow Wilson came

The U.S. Federal Debt ceiling was originally established in 1917 as a check against war spending for WW1.

WW1 was a war most of us didn't want to be in in the first place. Woodrow Wilson basically defrauded the country to force us into it, because he wanted to have the whip hand in post war negotiations to fulfill his grand design of a "league of nations".

With the debt ceiling legislation, congress were in theory, trying to keep the war spending under control, and to keep it from becoming a much larger war (and specifically to keep us from bailing out the British, French, and Belgians financially, or taking on the majority of the warfighting and war materials procurement).

In practice, they didn't actually do much to control the costs (though we did stay the "minority partner" in the war, as the U.S. electorate, and congress, intended; much to Wilsons disappointment).

In 1917, the debt ceiling was set at $13.5 billion dollars ($247 billion in 2013 dollars). It was quickly raised through the course of the war to $43.5 billion in 1919 ($590 billion).

That's more than doubling the debt in 2 years.

From 1919 through 1941, it grew to 50 billion ($800 billion).  That even including the government spending of the "New Deal" years, which far exceeded our depression diminished revenues every year.

Between the wars debt grew considerably, but not at a ridiculous rate... about 30% in real terms, in 22 years. That's not all that bad considering the government growth after the 16th and 17th amendments, and particularly the post WW1 expansion of the executive branch (thanks again Wilson), and then the mother of all government wealth transfer programs, "the new deal".

Then World War II happened

From 1941 to 1945, US federal debt rapidly increased, from $50 billion ($800 billion), to $300 billion (4 trillion).

So, in 28 years, and two world wars, U.S. federal debt multiplied by a factor of 16; and in 4 of those years, it multiplied by a factor of 5.

That's a lot of debt... but hey, wars are expensive right?

The 50's...

Well, for the next 18 years, the debt stayed almost completely flat... actually it was reduced several times. It stayed around the 300 billion mark from 1945 all the way to 1963.

1963 was both the last time it was reduced, and the last time it was at 300 billion. However, because of inflation, that wasn't actually flat debt; it was in fact a significant reduction from 1945s $4 trillion 2013 dollars, down over 40% to $2.3 trillion 2013 dollars.

That's still 4 times the debt at the end of World War 1 of course.
Oh and by the by... throughout this explanation I'm going to use the debt limit, and the actual federal debt, as if they were the same thing... because for all but 8 years during this time period, they WERE the same thing. We have generally run either just under, or actually in most years jsut OVER the debt limit, but used accounting tricks to seem like we were under it. 
The '60s

Wars are expensive yes, but through the Viet Nam years, federal debt grew far slower as a percentage than it had during WW1 or WW2.

It took 'til 1967 to get to $350bb ($2.45tt), and 1970 to get to 400bb (2.41tt); really a very small increase from 1963... and again, still a significant reduction from the end of WW2.

Overall, there was a less than 5% constant dollar growth in the debt through the entire 1960s; and that is coming off of a large drop in the debt from the end of WW2.

The '70s

Now, inflation started rising rapidly from 1968, particularly from 1970-1984, so you can't really compare the raw year to year numbers  from here. You need to compare the inflation adjusted numbers from here on out.

We hit $450bb in '72 ($2.51tt), $500bb in '74 ($2.371tt), $600bb in '75 ($2.608tt)... You can see again, slow growth or even shrinkage in constant dollar terms.

We hit $700bb in '76 ($2.9tt), $800bb in '78($2.9tt), 900bb in '79 ($2.9tt), and 935bb in '80 ($2.7tt), a big jump from '75 to '76, but then flat to a even a small reduction for 2 years.

Note, that was during the Carter administration, with a Dem controlled house and senate.

During the entire 1970s, we increased the debt by about 25% in constant dollar terms.

Then we hit

...the Reagan years... 

Except the Reagan years really weren't the Reagan years; they were really the Tip O'neil years (speaker of the house from 1977 to 1987).

Everyone remembers Reagan as a big spender on the military... because he was, but also because that was the media portrayal of him, but they forget that for every dollar Reagan got added to the military, O'neill and the democrats got $2-3 added to other government spending; and Howard Baker and Bob Dole (the Republican Senate majority leaders from 1981 through 1986) were happy to go along with it.

In 1981, the U.S. federal debt hit $1 trillion for the first time, ending the year at $1.1 trillion ($2.83 trillion)... but in constant dollar terms, that was actually less than 1979, and only slightly more than 1980.

Really it's not all that much more than 1963... only about 22%, in a period where population had doubled, military spending had quintupled, social security had jumped from insignificant to 15% of the budget.

It took 'til 1983 to get to $1.5tt ($3.5tt), and then to '85 for $2tt ($4.35tt), just two years to double the debt in absolute terms. Though that was "only a 22%" increase in constant dollars; that's almost exactly the same amount the debt increased from 1963 to 1983 in constant dollars.

20 years debt in 2 years.  Way to go guys.

Post 1984 inflation slowed dramatically, so the constant dollar differentials year to year are considerably less.

Oh and for you history buffs, 1985 was the year we equalled, then exceeded, our national debt levels at the end of World War 2 (in constant dollars).

So... we come to 1986 and at $2.3tt, we manage to double the 1966 debt, at $4.9 trillion in 2013 dollars (though only a 12% year over year increase from '85).

In '87 we went to 2.8tt (5.8tt) a 20% increase from '86, but managed to not increase the debt ceiling in '88... Unfortunately we made up for it in '89 moving up to $3.1tt ($6tt).

So, across the Reagan years, we went from $1 trillion ($2.83tt) to $3.2 trillion ($6.1tt)... More than tripling the debt in absolute dollars, and more than doubling it in constant dollars (117% to be exact)

Hmm... 40% debt shrinkage in the '50s, 5% debt growth in the 60s, 25% in the '70s... 120% in the '80s.

.. well, at least we slowed inflation down...

The '90s

And hey, we're getting into the '90s right? I mean yeah there was the '89 to '91 recession... and the first gulf war... but then there was the peace dividend and the .dot com boom, and the "surplus"... the '90s were prosperous years... we had surpluses... we shouldn't have increased the debt too much in the '0s right?

Sadly, wrong.

In 1990 alone, we increased our federal debt from $3.1tt ($6tt) to $4.2tt ($7.5tt) a 30% absolute increase, and a 25% constant dollar increase.

In constant dollars, that's also more than triple the 1960 debt

...and more than triple the 1970 debt...

...and more than triple the 1980 debt...

Once again... way to go guys...

But, by some miracle, during an incredibly expensive but incredibly short war, we managed not to increase our debt for 3 years... The entire rest of the first Bush presidency in fact.

On that one, seriously, way to go GHW Bush.


But we started spending again the second Clinton was elected, and by the end of 1993 had increased the debt to $4.9 trillion dollars ($7.9 trillion in 2013 dollars).

Then we had our "surplus years"... all two of them (actually we never really had a surplus, but they made it look like we did by ignoring a bunch of stuff and some accounting tricks... it WAS however the closest we had actually come to a balanced budget since 1958, when we started using the social security taxes as part of the general revenues). We didn't increase the debt ceiling during 1994 and '95..

... but again we made up for it in 1996 by increasing it to $5.5tt ($8.2tt) and '97 to $6tt ($8.8tt).

Then the Republicans got serious... for a second... and we managed not to raise the debt ceiling again until 2002.. when the NEXT war started.

So, we went from a 120% debt increase in the '80s, down to a... 50% increase in the 1990s...

Those were the "surplus" years? The "peace dividend"? Really?

Yeah... not so good...

Bush the younger

Now, all you Bush haters... all you folks claiming Bushes spending is what ruined us etc... etc...

In the first 18 months of George W. Bush's presidency we increased the U.S. federal debt exactly...


Yup, nada.

We didn't increase the debt ceiling at all from '97 'til late 2002

Then, as I said, the NEXT middle eastern war happened. And, as we know from earlier, wars are expensive.
Now, if you like, you can blame Bush for the wars... Though really, no matter who was president, we were probably going to have a war in Afghanistan, and a war in Iraq during the 2000s was a near inevitability as well. They certainly could have been run better and been over quicker however. 
In 2002 we increased the debt to $6.7tt ($8.71tt), a $700 billion, or 12% absolute increase from 1997...

Only in constant dollar terms, guess what...

It was actually a $100 billion dollar decrease.

In 2003 we went up to $7.4 trillion (9.4tt), another $700 billion increase... both in absolute and constant dollars since inflation was pretty much flat.

In 2004 it was an $800 billion increase to $8.2tt ($10.1tt) and again inflation was pretty flat.

These numbers by the way, are also the approximate annual costs of the war.

We didn't increase the debt during 2005... which was actually our highest spend year of the war in absolute dollars by the way; and in 2006 increase it another $800 billion to $9 trillion ($10.4tt), but inflation had increased so it was only a $300 billion constant dollar increase.

... though I should point out, the INCREASE in debt from 2004 to 2006, was 20% more than the ENTIRE debt in 1917 (in constant dollars)

... and the increase in previous years was more than the entire debt in 1941, at the outbreak of World War 2.

In 2007 we went up another... hey it's our friend $800 billion ($900bb) again... to $9.8 trillion ($11.1tt), another $700bb in constant dollars.

And in 2008... yup, another $800 billion to $10.6 trillion ($11.9tt), which was also an $800bb constant dollar increase.

Again, these numbers correspond roughly to the annual cost of the wars in Afghanistan and Iraq.

So for Bush BEFORE the 2008 financial crash we went from $8.8 trillion to $11.9 trillion increase, while fighting two wars, about 35%.

Not spectacular... but a hell of a lot better than it could have been

Except that in the last few months of the Bush presidency, we increased the debt another $700 billion, for the "bailout", bringing us to $11.3 trillion or  $12.3trillion in constant dollars.

$8.8 trillion to $12.3 trillion... about 40%.

And finally we reach...

Barack Obama

Yaknow... I'm not even going to bother going year by year for this guy.

Barack Obama was sworn into office on January 20th 2009.

It's October 18th 2013.

That's 4 years, and 10 months. Let's round up to be generous call it 5 years.

In less than 5 years, under this president and this congress, we have increased our debt from $11.9 trillion to $16.8 trillion. Actually a bit more without the accounting tricks.

So... 5 trillion more or less.

Bush pushed the debt up 3.9 trillion in 8 years, or just about $500 billion a year.

Obama pushed the debt up 5 trillion in 5 years... about $1 TRILLION a year.

In constant dollar terms, that's more debt INCREASE every year than we had TOTAL before  December 7th 1941.

In constant dollars it's 60% MORE THAN THE ENTIRETY OF WORLD WAR 2

Minute by minute...

As I am writing this, Obama has been president for 4 years, 271 days, 17 hours, 48 minutes, and 10 seconds.

That's 1733 days, 17 hours, 48 minutes, and 10 seconds.

That's 41,609 hours, 48 minutes, and 10 seconds.

That's 2,496,588 minutes, and 10 seconds.

That's 149,795,290 seconds.

That's a $33,378.89 increase in the U.S. federal debt EVERY SINGLE SECOND HE HAS BEEN PRESIDENT.

That's $2,002,733.40 every minute.

That's $120,164,004.00 every hour.

This guy has 3 years, 93 days, 7 hours, 11 minutes, and 50 seconds left to be president.

At this rate, that's another $3.5 trillion dollars.

Oh and in the decade of the 2000s, we went from $8.8 trillion to $13.5 trillion constant dollars; about 55% growth.

From 2010 to today, 3.8 years, we've increased the debt from from $13.5 trillion to 16.8 trillion or $3.3 trillion. Basically $900 billion a year more or less. So, we're on track to finish the 2010s at $22.2 trillion in constant dollars.

... To put it another way, in 2017, 100 years after the debt ceiling was first written into law; we will have increased our debt to approximately 100 times what it was in 1917.

The 2010a are on track to go from $13.5 trillion to $22.2 trillion is an $8.7 trillion increase, or about 65%

The final breakdown, decade by decade

40% debt shrinkage in the '50s
5% debt growth in the 60s
25% debt growth in the '70s
120% debt growth in the '80s
50% debt growth in the 90s
55% debt growth in the 2000s
65% debt growth in the 2010s (projected)

Ok... so what does that mean personally?

Well... to simplify, and to keep things more comparable, let's keep it post WW2. As it happens, debt stayed relatively constant in absolute dollars from 1945 to 1963 (though dropped in constant dollars), and 1963 was the last time the debt ceiling actually decreased... plus it's 50 years, a good round number... Let's take it from 1963.

Total debt growth in the last 50 years, in constant dollar terms?

Debt went from $2.3 trillion, to 16.8 trillion, a multiple of 7.3.

In 1963, the debt to GDP ratio was about 40%.

In 2013, the debt to GDP ratio is 101%

In 1963 the U.S. population estimate was 189,241,798
In 1963 the per capita U.S. federal debt was $12,153.76
In 1963, the mean salary for a full time worker was appx. $44,000
In 1963 the federal debt to personal income ratio about 28%
--all constant dollars. Note these are means not medians.

In 2013 the U.S. population estimate is 316,882,000
In 2013 the per capita U.S. Federal debt is  $53,016.58
In 2013 the mean salary for a full time worker is appx. $47,000
In 2013 the federal debt to personal income ratio is about 115%

Note: If you take it by median individual income it's about twice as bad (the mean full time is only full time workers between 18 and 65. The median individual income includes the unemployed, retired, part time workers, under 18 and over 65 etc...). 

So, our debt has increased by a factor of more than seven; and our per capita debt has more than quadrupled, as has our debt to income ratio.

... and yet... some people say "we aren't doing enough, spending enough..."

I have had this conversation

Your adrenals are under-functioning, your thyroid is flat-lining, your pituitary is DOA, your testosterone levels are below those of a twelve year-old girl, and your body is sadly lacking in almost all the critical enzymes that make life possible. 
Give it to me straight, Doc. Don't pussyfoot.
My doc said pretty much the same thing to me.

So, to Chuck Lorre I say... enjoy the music:

Wednesday, October 16, 2013

Because they think funding is "doing something"

One of my favorite television shows of all time is "the west wing". I like it for the writing and the characterization, not so much for the political viewpoints... But those viewpoints are illuminating.

One thing which libertarians and conservatives scoff at, is the language that liberals use when it comes to government programs...  "this is a victory in fighting poverty" etc...

Fighting poverty, racism, AIDS... these things are all INCREDIBLY hard. They're so hard that there's very little any one of us can do about it. 

Conservatives and libertarians solution to that, is the power of voluntary charity, volunteer work... and accepting that we won't fix these problems. That eventually, over the long term, they will get better... but we aren't going to win them ourselves. 

Poverty for example... Poverty isn't a disease, it's not an inevitable mass condition, it's not like Old age. The solution to poverty is more, higher paying jobs. You get more jobs, with more businesses, and more activity in each business. You get more jobs with lower regulation, and a freer market and freer competition, and by making it cost less to employ people.

But that's diffuse... it's not DIRECTLY acting to "solve" the problem. There's no "program" for it, that they can provide funding for. 

The problem is that liberals morals and ethics won't allow them to accept this. They believe that they... and all of us, as a nation in fact... have a moral obligation to "do something". Even if it doesn't work, we have to at least "do something". 

Unfortunately, reality is what it is...

So, their substitute for actually doing something about the problem is "government funding to solve the problem".

This let's them fulfill what they feel to be their moral obligation, because they are able to deceive themselves into believing that funding the government really is "doing something". 

They get to blame the failure on not trying hard enough, or not getting enough funding... though that's pretty much ridiculous. 

They then believe that everyone who opposes their "moral" imperatives is either stupid or evil. 

Some of us are just... practical, logical... 

We spend enough in the "war on poverty" every year (to almost no effect) to actually give everyone below the poverty line, enough money to be well above it. We could literally just send them a check every month.

I'd rather do THAT, than what we do now... It wouldn't be any more expensive, and it would have the added benefit of ACTUALLY WORKING.

Tuesday, October 15, 2013

Tulips and Tuition

Why college costs will soon plunge
"Aid to higher ed already has exploded: In 1964, federal student aid was only $264 million, or $1.7 billion in current dollars. Today, the feds shell out $105 billion a year just in student loans. Total federal aid has soared from $64 billion (in 2000) to $169 billion (in 2010). 
Flooded with such largess, colleges have sent prices skyward (tuition is up more than 500 percent over the past three decades) and indulged in luxuries that would have made Marie Antoinette blush, from gourmet dining halls (sushi at Bowdoin, vegan at JMU) to rock-climbing walls. 
Last month, Virginia Commonwealth University announced the construction of two new dorms that will add 426 beds. Their $41 million cost comes to more than $96,000 per bed. Thank goodness Virginia is, comparatively, fiscally conservative: Princeton recently built a dormitory at a jaw-dropping cost of almost $300,000 per bed. 
Trend lines like these cannot go on — and they won’t. But not because of politicians’ efforts to rein in college costs. College costs will drop because of market forces politicians will be powerless to stop." 
Yes... true... but I think there's really a simpler way of putting it.

College tuition will soon collapse, because eventually people realize that tulips aren't worth very much.

I really did love Aaron Clareys book "Worthless" . I've ended up giving it to several students and parents I know. If you are one of those things, spend $5, to help save $100,000... you won't regret it.

I have a double degree in Aerospace Engineering and Computer Science, from a prestigious private engineering school... and I've never used either degree, except in pursuit my own personal interests (hobbies basically). Worse from a pure economics standpoint; excepting those organizations that require a college degree to consider a candidate, neither have particularly helped in my career.

My college costs were somewhat less expensive 20 years ago (in constant dollar terms, about 20% less), and the Air Force was helping me to pay, but I still had to have side jobs to pay for it... (though I graduated without debt, by choice).

My alma mater is a top 20 school in all its major disciplines by any major ranking, and in the top ten for several disciplines. It was recently ranked as one of the "500 best college values" near the bottom of the top 100. Boiled down... it's a very good school and for very good schools it's only moderately expensive.

To replicate my college education today would cost $184,000... and that's just in tuition, never mind books (books in engineering school can run several thousand a year), food, housing, transportation etc...

The majority of AFROTC scholarships have a tuition cap of $18,000 per year, and the student has to pay the difference. That's not even half the current tuition at my alma mater; which by the way graduates a higher percentage of AF officers than any school other than the Air Force Academy, and more in absolute numbers than any school other than the Academy and Texas A&M.

Much as I loved my school, and as great as the education I got was... not even CLOSE to worth it.

In fact... I don't think ANY college education, except that which is absolutely required for the technical elements in your chosen field (particularly STEM fields of course), is "worth it" from a career or economic standpoint, based on current pricing.

College is great for growth as a person... expanding ones horizons, developing habits and skills of learning and research, finding ones self and ones interests... But $185k is a hell of a lot to pay for that.

... and that's only a MID priced school these days.

Mid priced private schools are running in the $40-$50k a year range for tuition, fees, and board. Top priced schools are in the $60k a year range.

In-state tuition and fees (not including books, room, and board) average $13,000 a year for state schools. Out of state tuition and fees (again, not including books, room, and board) at state schools averages $32,000 a year.

The most expensive of the public universities, charge nearly the same for out of state tuition, as top tier private schools. Michigan charges as much as $40,000 a year tuition only. That's more than some Ivy league schools (the Ivies also may offer generous endowment grants to reduce tuition further)

All that debt, when in the job market... a degree isn't the advantage it once was.

As a hiring manager, my experience is that it doesn't matter how good a school you came out of, if you have no experience in the real world, and a BS or MS... you're worse than useless, you're actively harmful.

I have to spend two years to get you to unlearn the idiotic crap they taught you in college, and another two years getting you to actually be a useful employee.

This is true for everyone but the extremely capable, self motivated, and self educating... and for them, it doesn't matter whether they have a degree from a prestigious university, or a community college, or they are entirely self taught.

I'd rather have someone self taught, with 4 or 6 years of useful experience, with a quick and twisty mind, and a habit of learning; than ANY college graduate.

The degree... it's really nothing more than a piece of paper certifying that you're probably not a felon or a drug addict; and that you can show up, and follow basic instructions, for four-ish years.

Monday, October 14, 2013

Let's celebrate the NordoCeltic peoples ruling the world

We do yaknow... Or at least we did...

As it happens, this is the 947th anniversary of the battle of Hastings, wherein the Normans under William the Conqueror, earned him his name, by rather embarrassingly whipping the force of the Anglo-Saxons under Harold Godwinson (Harold II), the last Anglo-Saxon king of England.

 Now, the historically ignorant amongst us have often made fun of poor Harold for having been conquered by "the French".

... but in reality, the "Norman french" weren't "French" as we think of it.

...well... and to be honest, what we think of as the French were actually Germans and mostly Spaniards originally (the Franks from which France gets it's name, and the peoples historically from the spanish and french borderlands).

 The other people in France were mostly celts, or nords.

 The Norman French were in fact mostly a mixture of the Gauls (celts), and the Norse.

 That's right... William the Conqueror was a Viking crossed with an Irishman. ... which of course is why his grandchildren then invaded Ireland and became "more Irish than the Irish".

And as history has shown, there is no shame in being conquered by the greatest western conquerors and assimilators of culture, the Celts.

 Hell... we loved taking over England so much, that we did it again 539 years later, in the person of James Stewart (himself a further cross between Vikings and Celts).

 So, let's hear it for the NordoCeltic peoples!

Say what you will about Miley Cyrus... the girl CAN sing.

Something that has kinda buggged me about the recent crapstrom surrounding Miley Cyrus...

Many people, most of whom have probably never listened to anything she's done; are saying that she's another no-talent pop "starlet" who can't sing.

 That's not actually true.

 Whatever else you can say about her... and yeah, her recent behavior isn't exactly wonderful, but it's doing exactly what she wants it to... Make her a HELL of a lot of money... She CAN sing.

 Here she is doing her hit single "Wrecking Ball" live on SNL, arranged as an 80s style power ballad... and frankly, it kicks ass:

I've never cared for her music, but I recognize when a performer is good and I just don't care for them, vs. when a performer isn't good.

 I'm not saying she's a spectacular singer... but she has decent vocal range (not used much in her singles, but it's there), expression, timbre, and intonation. She is a little too fond of melisma, but that's very common with female pop vocalists; and she often chooses a somewhat nasal expression, but again that's a choice (one I don't care for, but she CAN sing without it... it's just part of her general style, as is "talk singing". Her speaking voice is somewhat tight and nasal, with the tongue against the soft palate, and that transfers over).

 Listen to this A Capella version of her other current major hit single, with "the Roots" (who by the way are still great, and always have been) on "Late Night with Jimmy Fallon":


Most importantly, she can sing her songs, in tune, without autotune, which is unfortunately rather rare among pop acts these days.

 Now... if you want an example of a pop singer who IS an amazing singer (but whose music is mostly not good), Christina Aguilera.

Or Pink (who is recently billing herself by her full name Alecia Moore), who can sing (though not as well as Christina), and whose pop music I think is actually GOOD. ...Or for that matter, Lady Gaga, who can both sing and play piano quite well, and much of her music is quite good (the stuff that isn't explicitly meant to be performance art).

 For an example of a big young female pop "singer" who can't sing whatsoever... Oh, how about Kesha.

 ...Smart girl though, literally... it's reported her IQ is something like 160, and she turned down early graduation and college scholarships to drop out of school and take a GED, so she could become a full time song writer; which she did very successfully, before ever recording anything. As of now, she's making a great deal of money on the image she has decided to portray.

 Oh and while she can't sing... she's actually a pretty decent musician (she can play piano, saxaphone, guitar, and drums).

 Oh and the thread between all these women?

Their really awful but very commercially successful music? Much of it is produced, and often written by, the same producer and production team, "Dr. Luke".

 The music industry hasn't really changed much from the 1950's and 60's "hit factory" days really... It's still oriented at selling hit singles to teenage girls, and it's still really the producers who are in charge. It's just that the producers own their own production companies now, instead of the record companies controlling everything.

Wednesday, October 09, 2013

Bye Bye 4th and 5th amendment; Obamacare info may be used for Law Enforcement and Audit activities

Well... we knew that the 4th and 5th amendment meant nothing to them... never mind HIPAA... but really?

  Obamacare Marketplace: Personal Data Can Be Used For ‘Law Enforcement and Audit Activities'
Maryland's Health Connection, the state's Obamacare marketplace, has been plagued by delays in the first days of open enrollment. If users are able to endure long page-loading delays, they are presented with the website's privacy policy, a ubiquitous fine-print feature on websites that often go unread. Nevertheless, users are asked to check off a box that they agree to the terms. The policy contains many standard statements about information automatically collected regarding Internet browsers and IP addresses, temporary "cookies" used by the site, and website accessibility. However, at least two conditions may give some users pause before proceeding. The first is regarding personal information submitted with an application for those users who follow through on the sign up process all the way to the end. The policy states that all information to help in applying for coverage and even for making a payment will be kept strictly confidential and only be used to carry out the function of the marketplace. There is, however, an exception: "[W]e may share information provided in your application with the appropriate authorities for law enforcement and audit activities." Here is the entire paragraph from the policy the includes the exception [emphasis added]:
Should you decide to apply for health coverage through Maryland Health Connection, the information you supply in your application will be used to determine whether you are eligible for health and dental coverage offered through Maryland Health Connection and for insurance affordability programs. It also may be used to assist you in making a payment for the insurance plan you select, and for related automated reminders or other activities permitted by law. We will preserve the privacy of personal records and protect confidential or privileged information in full accordance with federal and State law. We will not sell your information to others. Any information that you provide to us in your application will be used only to carry out the functions of Maryland Health Connection. The only exception to this policy is that we may share information provided in your application with the appropriate authorities for law enforcement and audit activities.
The site does not specify if "appropriate authorities" refers only to state authorities or if it could include the federal government, as well. Neither is there any detail on what type of law enforcement and/or audit activities would justify the release of the personal information, or who exactly is authorized to make such a determination. An email to the Maryland Health Connection's media contact seeking clarification has not yet been answered The second privacy term that may prompt caution by users relates to email communications. The policy reads:
If you send us an e-mail, we use the information you send us to respond to your inquiry. E-mail correspondence may become a public record. As a public record, your correspondence could be disclosed to other parties upon their request in accordance with Maryland’s Public Information Act.
Since emails to the marketplace could conceivably involve private matters regarding finances, health history, and other sensitive issues, the fact that such information could be made part of the "public record" could prevent users from being as free with their information than they might otherwise be. However, as noted, any requests for such emails would still be subject to Maryland's Public Information Act which contains certain exceptions to the disclosure rules.
Read the fine print eh?

 These are such clear 4th and 5th amendment violations I can't believe anyone didn't immediately say "uh guys... we cant actually do this"...

... but as I said, we know that our elected and selected "lords and masters" don't give a damn about the 4th or 5th amendments (or really any of the others ones any time they become inconvenient).

So while I'm sure they were told they couldn't do it, I'm sure they said "ahh well the disclaimer and release is enough, we'll be fine".

 Yeah no.

 And as far as HIPAA goes... In reality these terms of use are not anywhere near an adequate HIPAA disclosure release, so using any of this data in any manner other than for healthcare purposes would be a federal offense.

Tuesday, October 08, 2013

The sharpest rebuttal to the "leviticans"

By the by... Martin Sheen is a truly devout catholic, and he helped write this dialogue. This is what I'm talking about when I tell people that Christians are NOT commanded to follow Leviticus and the other Jewish laws (except those specifically related or commanded by Christ). This is the foundation of the new covenant in Christ.

Monday, October 07, 2013

Hmm... yeah... there's your problem right there...

So, my MacBook Pro has been acting a bit wonky lately... and I'm pretty sure I know why.

A few weeks ago, I started getting disk corruption, that couldn't be handled with the normal disk utilities; and required me to get a clean backup on an external drive, wipe and reformat the internal drive, and restore...

Well, I kept getting the corruption problems after a few hours or a few days... and they kept getting worse.

Finally, I ended up rebuilding the thing 5 times in two day; and 3 times in one night (this was the night before my big compliance webinar. I didn't sleep at all the day before or that night, and ended up working all night and all morning before the webinar to try to get sorted).

This was basically two weeks of escalating pain, but until the last night the issues were intermittent with variable recurrence, so I couldn't get enough diagnostic info to nail it down.

With the 3 in on night episode, I was finally able to see the problem occuring...

And it's something I have NEVER seen... never even heard of...

What was happening on the HDD was lots of tiny single bit/single block/single write i/o errors. Ok, that happens... but why? It was a less than 90 day old relatively high end SSD (my last SSD went bad this past summer).

So I looked deeper at the errors, and noted that not all of them were from the hard drive...

Some of them were from the DVD drive...

Which had a scratched up DVD-R in it...

I pulled out the bad DVD-R and... holy crap, no more I/O errors.

What was happening, was that the particular damage on the DVD drive, was causing the I/O controller to constantly attempt to re-read the drive, and fail... hundreds of times a second. Instead of just limiting out though, it was causing enough latency in the SSD, that it was getting I/O errors as well...

Thing is... I didn't notice, because the DVD drive wasn't constantly spinning up... just a couple times an hour maybe? Which could have been explained by finder doing crap.

I've never seen that before... never even heard of that before, in a desktop or laptop (it's something that can happen with large high volume high transaction count servers, if they don't have sufficient spindles or cache, and their i/o controllers don't handle the exceptions properly).

Anyway... I got that resolved, and got my MBP functional...

But, ever since the last rebuild (after I figured out the problems), it's been a bit wonky. The finder doing some weird things etc...

I've run all the normal diagnostics, and at this point I'm pretty sure that to get sorted, I'm going to need to do another clean beackup, but instead of just restoring, I need to do a clean install, then migrate my apps and data.

It's a PITA, so I'm putting it off until I can't put it off anymore...

Meantime, I'm living with assorted wonkiness.

One of the items of said assortment; I hadn't really noticed it until a couple days ago, but I couldn't empty my trash.

This happens on OSX sometimes, it's not really a big deal. Usually it's a file that is locked somewhere and it can't be forced to let go because of a zombie process, or a bad pointer somewhere etc...

It's generally easy to fix. You just go into the trash directory from the command line, and force delete everything.

So, I went in, as root, and did a listing of my .Trash.

And it took a while... a LOOONG while... many many many screens of data flashing by my screen...

24 MILLION ITEMS... for a total of 243.8 gigabytes.

Well... there's yer problem right there...

It seems that the detritus of the multiple rebuilds... including several complete copies of my hard drive... ended up getting stuck in the trash for some reason; and couldn't empty out.

So, I started the force delete and went on to other things in other windows... after about 20 minutes I came back... and my command prompt hadn't come back...

I figured it had frozen up, or otherwise wasn't working; so I cancelled the job. Ran the listing again...

Nope... it had been working... It had deleted 9 million of the items, there were still 15 million left.

So I started the job back up again and went away for 20 more minutes... went back... still working...

As I was about to switch windows away it finally finished.

It took 40 minutes to delete the crap from the command line, no wonder I couldn't empty or open my trash in finder...

To the Baby, We ARE the NSA

Yes... this is EXACTLY right, totally true and correct

Yes... this is EXACTLY right....

The source for the current "it's OK I speak Irish" meme.

Dont you wish YOUR job had raises like this?

This whole government "shutdown" thing has brought out a lot of talk about federal pay.

A liberal of my acquaintance posted something on facebook a couple days ago:

"A Republican I know said, 'If you got furloughed because of the shut down, maybe you should get a real job.' 
Yeah... about that..."

'pon which he linked to a story about the cops, border patrol agents, etc... who were not being paid while protecting congress, and our country.

It's a good point. There are plenty of people doing real, important jobs, who are not being paid... Some of them have gone home, but a LOT of them... actually about 2/3 of the federal non-military workforce, hasn't. They're still doing their jobs.

The problem I have is... there's too many of them... And they are doing too many things, that they don't need to be, or shouldn't be doing.

So, I said something fairly well known in libertarian circles:
"A good friend of mine is a border guard with ICE... yeah, he's got a real job. 
That said, there IS a point when the most liberal liberal in America has to think 'why in the hell do we have 50% more federal government payroll than 1998... we're not getting more than we got then... at least not more good useful stuff....' That's just non-military federal staff payroll by the by, not any other spending..."
His commenters didn't believe me, or just said inflation or homeland security etc...

So I clarified, no, federal non-military payroll; meaning the total compensation (wages, salaries, and benefits) of full time permanent non-military federal workers, has increased, by at least 50%, in constant dollar terms, from 1998 to today.

And homeland security is only a fairly small portion of that increase (Only 9% of the federal workforce, though it is the single largest federal agency - excluding the civilian employees of the military and veterans affairs - in terms of manpower).

To which he said, quite reasonably "would you care to source that?".

Gladly sir....

Congressional Reporting Service report on trends in the federal workforce:

Congressional Reporting Service report on average wages etc... in the federal workforce:

Several other primary sources in the footnotes of this article, notably from the Bureau of Economic Analysis:

So... let's break it down shall we?

The CRS reports there was a 17%... actually 16.7% increase in the federal workforce between 2000 and 2010.

I don't have the numbers from 1998, 1999, 2011, 2012, or 2013, but other sources indicate that it's probably not much, because there were hiring freezes and reductions that make it pretty much a wash. 17% is probably good for 1998 to 2013.

So, a 17% increase in non-military federal staff from 1.8 million to 2.1 million (excluding the civilian employees of the Army, Navy, Air Force, and Veterans Affairs; currently about 900,000).

Oh and it's important to note that these numbers do not include contractors. Contractors compensation does not count against federal payroll, and they are not counted as federal workers... which is one of the major reasons there are so many of them...

How many?

In 1998 there were approximately 1.8 million federal workers, and only 6.5 million contractors.

Well, as of 2013, there are appx 2.1 or million federal non-military workers... and appx 17 million contractors.

Contractor compensation DWARFS the federal payroll. It's well over 20 times federal payroll in fact... though we really have no exact idea how much, because it's buried in hundreds... or possibly thousands... of different budgets, and literally millions of line items (many of which are gray, or black).

So, let's talk money...

First, let's talk about total compensation.

Total compensation includes both wages and other cash compensation, and non-cash compensation such as benefits.

Bureau of Economic Analysis reported average total compensation for federal employees went from appx. $67k in 2000 to $115k in 2012.

In constant dollar terms that is a 29% raise.

Oh but that's just from 2000-2012 I don't have the exact numbers here from BEA for '98,'99, and 2013...

Purely from a trendline analysis, you see a 2.15% annual average real dollar compensation increase. Extend the trendline from 1998 to 2013, and instead of 29% it's about 38%.

A 17% staff increase and a 38% raise, is a 60% increase in total payroll...

Now... even if you just take cash compensation, BEA reports an increase from $56k to $82k; a constant dollar increase of 16%.

That's much lower than the increase in total compensation, but still quite respectable... And remember, this is in constant dollar terms; so that's over and above inflation and cost of living increases.

Again, thats 2000-2012. Extending the trendline from 1998 to 2013 and you get 21%.

21% raise times a 17% staff increase, is a 41% total increase in real dollar terms; for just cash compensation.

Now... those are BEA numbers, what about CRS numbers?

Hmm...  I don't have the exact numbers on total comp increases from those years... But I do have their percentages... in fact I have every percentage increase, and the inflation percentage, for every year since 1969...

Federal Average salary and wage increases year over year, 1999-2013 (1998 would reflect increases from 1997):

1999: 3.4% over inflation
2000: 2% over inflation
2001: 0.3% under inflation
2002: 0.4% under inflation
2003: 0.2% over inflation
2004: 2.0% over inflation
2005: 0.2% over inflation
2006: 1.4% over inflation
2007: 1.6% over inflation
2008: 1.8% under inflation
2009: 1.6% over inflation
2010: 1.9% over inflation
2011: 1.8% over inflation
2012: 1.8% over inflation
2013: 1.8% over inflation

Official numbers have not been released for 2011, 2012, and 2013; the 1.8% is from news reports and other websites stating that though federal salaries have been in a base rate freeze, the average salary has increased 1.8% over inflation in each of the last 3 years. This is consistent with previous increases. 

So, from purely federal internal sources, we have an average wage/salary only, increase of 24.8%. Times a workforce increase of 16.7% (also from the CRS), we have a 45.6% increase.

So... there's the CRS's own estimate, of average wage and salary alone.

Unfortunately, the CRS doesn't estimate total compensation, but if we assume the BEA numbers are reliable, non-cash compensation has increased from appx 22% of cash compensation in 1998 to approximately 40% of cash compensation in 2012.

This estimate is not out of line with other trends and percentages well known in HR (noncash compensation, particularly benefit costs, have doubled or more in the last 15 years)... so I think it's a good and reasonable approximation. 

Oh... might be useful to summarize. I've got two different sets of numbers, which are different enough to be noticeable, but not enough to completely contradict each other.
Note: The difference between the BEA and CRS may include slight differences in the way they calculate compensation; and they definitely include differences in the way inflation is calculated. The BEA numbers used BLS inflation adjustment. CRS uses CPI based inflation adjustment (CPI is a component of the BLS inflation adjustment, but there are other elements included as well).
Workforce increase 16.7%

BEA: cash compensation increase 21% total comp increase 38%
CRS: cash compensation increase 24.8% total comp increase 42.8%

Total payroll increase cash/comp

BEA: 41%/61%
CRS: 46%/66%

So... no matter which numbers you believe, total comp increase is WELL over 50% in 15 years, and according to the CRS cash comp is up nearly 50%; and the lowest estimate is 41%...

Over and above inflation...

Yeah... don't you wish your job had raises like that?

Oh and one more thing...

From the late 1960s, through the 80s and into the early 90s, federal workers as a whole were actually paid quite poorly, as compared to comparable private sector jobs. Their wage scales were originally set at bottom of market to begin with (generally though of as a tradeoff for their better job security and benefits), and the unusually high inflation from 1968 to 1984 had private sector wages rapidly increasing, while federal cost of living adjustments were significantly under the rate of inflation.

This left a population of workers who were dramatically underpaid in comparison to the private sector, all the way through the early 1990s.

Many still are. Those in the bottom 2/3 of the federal pay scale are generally still significantly UNDERPAID, not overpaid as compared to private sector; sometimes dramatically so (permanent non-contractor federal IT staff make less than half industry comparable salary for example).

Those in the top 1/3 though make quite a lot more than comparable private sector jobs.

...Well, that is, until you get to the "senior executive" level, where, once again, they make 1/2 or less what they would in the private sector ($190k a year is the top out. Private sector workers at those levels of education, experience, responsibility etc... typically make anywhere from $200k to over a million, with $400k+ not uncommon).

It is only from the mid 90s that the federal payroll, and specifically average pay (skewed by the top 1/3), began to dramatically outpace private sector pay.

The bottom 2/3 of the federal workforce didn't get very much of that increase.

The top 1/3 of the federal workforce got much larger increases.

Also, there are far more workers in the top 1/3 of the pay scale than there were in 1998. Far more making more than $100k a year, and far more making more than $150k a year.

The middle 1/3 shrank significantly.

So there's more low end, more high end, and less middle...

Not exactly shocking...

Saturday, October 05, 2013

The OTHER 17 times the government shut down without the world ending...

Did you know that this is actually the 18th time the fedgov has "shut down" since 1976?


These included 1 shutdown for 10 days under Ford; 5 shutdowns in 18 months, for a total of 57 days under Carter (with dem controlled house and senate no less); 8 shutdowns for a total of 14 days under the 8 years of Reagan (none longer than 3 days); 1 for 3 days under Bush; and 2 for 26 days under Clinton.

Mostly these things happen because of the Antideficiency act, which, if an executive branch administrator has been found in violation, can cause them to lose their jobs, and possibly face civil and criminal penalties:

My favorite shutdown?

4 of the 5 Carter shutdowns were because of debate between northern and southern Democrats about whether medicare could fund abortions only in the case of danger to the life of the mother; or whether there should be an exception for rape and incest. The rape and incest northern Democrats eventually won.

Note... there was no "Republican obstructionism" involved... it was southern Democrats vs. northern Democrats on this one...

Friday, October 04, 2013

Why I don't bother writing certain posts

A conversation overheard in my household:

Mel: You know what post you haven't written... 
Chris: Hmm? 
Mel: It's one of those "gonna piss people off posts"... about {insert "controversial" topic here} 
Chris: Ahh... yeah... that... 
Mel: Oh? That? 
Chris: ... Frankly... I don't want to deal with those fucktards. 
Mel: Hah! 
Chris: Seriously... there is only so much fucktardery I can deal in any given time period. Those fucktards will exceed my fucktardery limit in about 12 microseconds... I just can't handle that much concentrated fucktardery... 
Mel: Hah! Coward! 
Chris: No... not cowardice... a reasonable stance to protect my sanity... and my blood pressure... and my faith in humanity...

Recipes for REAL Men, Volume 29 - It's "Just" Rice

It's been five years since I published an "official" numbered "Recipes for Real Men" recipe... Which is way too long. I've written a few posts on food and cooking and even published a few recipes in between, but I haven't done a "REAL" Recipe for REAL men so to speak, since Christmas 2008.

Time to fix that...

Rice is the most important staple food in the world. Some 75% of the worlds population depends on rice and rice based dishes for the majority of its caloric content.

Plain white rice, steamed or boiled, sticky or firm; is the basic starch for most dishes, in most cuisines, the world over.


It's plain.


It's just starch. It's plain, it's white, it's starch.

Without adulteration, or special preparation, plain white rice has very little flavor on its own. Though it offers a broad range of textures (depending on variety and preparation), on it's own, it offers no complexity or contrast (of flavor OR texture).

Or... it can be boring... but it doesn't have to be.

In a thread on The Guncounter "Things I didn't know about food" a few months back, I proclaim the virtues of even simple white rice, as a tasty meal:
CByrneIV: "For that matter, properly selected, properly made (and if appropriate seasoned) rice, is a revelation. 
Properly cooked medium or long grain rice, even if just cooked in salted water (or salted acidulated water); when cooked to the right degree of moistness and tenderness, is great with just a bit of butter and pepper, and maybe a bit of acid. 
I like to add a squeeze of citrus, a dash of vinegar, a dash of soy, a dash of hot sauce, a dash of prepared hot mustard etc... Maybe a tiny bit of fresh parsely, cilantro, or mint. Maybe a bit of Parmigiana or Romano, and some toasted pine nuts. 
I can eat a bowl of it, by itself, as a meal, no problem. 
For a bit more substance, just toss in come black or red beans (preferably simmered in flavorful liquid), maybe a bit of crushed up crispy bacon or crisped chopped ham, or some browned loose beef, pork,chicken or sausage. Or some combination thereof. 
It's not red beans and rice, but it's close enough; and it takes maybe ten minutes of prep, and 20-30 minutes of total cook time. 
Crush some fried noodles, fried wontons, fried tortillas/tortilla chips, or some ciccarones (pork rinds, beef rinds, crispy chicken or turkey skin) over it for some textural contrast."
So... white rice isn't necessarily "just white rice"...

But why limit yourself, to just plain old steamed or boiled white rice? There's a huge variety of seasonings, techniques, and additions to plain old white rice, that completely transform it.

White rice, because of its "plain" nature, can act as a canvas for an infinite variety of techniques, textures, and flavors.

A basic flavorful rice preparation is probably the most useful and versatile foundation, for either a side OR a main dish, that there is.

So... how do you get started with rice?

What are the basics?

How do you make something other than just "white rice"?

First, the rice...

There's a full range of rice varieties, and preparations, that can produce just about any texture and mouthfeel that you'd like (and pair with whatever flavors and textures you'd like).

Long Grain

For plain white rice (not to be used in a flavored rice dish, sushi, etc... ) or for a rice dish that uses loose, relatively dry rice with "stuff" mixed in; you're generally going to use a washed long grain rice. Basmati rice, or something similar is generally preferred, as it cooks clean, firm, tender, and loose, without being dry or sticky. It's also particularly useful as a foundation for dishes where a thick or creamy sauce will be a highlight, as it will not take on odd or unpleasant textures when topped with such sauces.

Medium Grain

Many flavored rice dishes will use a medium grain or one of the comparatively starchier softer varieties of long grain rice (which are generally grouped culinarily with medium grain rice), because they take and hold flavors better, and work better when actually served IN a sauce (as opposed to having say, meat in a sauce poured over the top of it).

Medium grain rices can also be good in twice cooked preparations like fried rice, crispy rice, or deep fried rice balls. Calrose rice (or rather one of its many derivative varietals) would be typical, as it's cheap, commonly available, and can cook up with very different textures depending on preparation.

In general, the shorter the rice, the lower the hard starch (amylose) , and higher the soft starch (amylopectin) content of it. The higher the soft starch content, the softer and more glutinous (not gluten as in the protein, glutin as in "gluey in texture") the rice will be (the lower amount of amylose dissolves into the water making the grain soft, the amylopectin gelatinizes with the heat and moisture, making it gluey).

Rice grain and Starch?

Rice's starch content can (and usually is) also be modified before final preparation through polishing, washing, parboiling, "converting", and other means.

The vast majority of rice sold in American supermarkets is polished (thus making it "white rice"); and some is polished and washed (reducing its starch content further). "Instant rice" or "minute rice" is "converted" (meaning it's washed, polished, parboiled, and dried under heat).

Of course, most any process acting on the rice is going to reduce its starch content, and modify the texture of the final cooked rice. You can even wash your own rice before cooking, to reduce the starches it will express into your dish while cooking.

Basically, you use polished and washed long grain rice for a drier or looser dish. Use unwashed long grain rice for a slightly moister or tighter dish.

Use a medium grain rice, with a bit of natural starch to it (a moderate starch variety, not washed), for a moister, tighter, and creamier dish; or if you want to make a crispy rice, crusted rice, or non-sticky rice balls. Wash your medium grain rice if you want a softer grain, but don't want a creamy or tight texture.

What About Short Grain?

Short grain rices are generally the starchiest rices... or rather those that give up their starch the most freely. They are really a more specialized rice, great for the specific preparations and dishes that need them, not very good for anything else.

Don't use a short grain, pearl rice, or "risotto rice" (there are several varieties of rice used in risotto, but the most common is arboreo); unless you want to make a true risotto, sticky rice, sticky rice balls, sushi rice, creamed rice, or rice pudding. Those are an entirely different topic (or really a couple of topics) that I'll cover in another post (or posts), at another time.

Right here and now, we're talking about full grain rice dishes, not modified grain dishes (that's what risottos and rice puddings are. The individual grains of rice express so much starch and soften so much, that they are completely different in texture from other rice dishes).

Ok, how about Brown Rice?

Brown rice is just what we call the unmilled, unpolished whole seed (with just the husk removed) of the various varieties of white rice; including the endosperm, whole bran, and germ (white rice is just the endosperm).

In general, you won't want to use brown rice in flavored rice dishes (though there are some that do).

Also, in general, being "brown" means that these rices are going to give up less starch, and do so slower, than white rices. This makes them tend toward the firmer, harder, and drier side when prepared.

Brown rice can be somewhat more flavorful on its own than a white rice, with a nuttier, toastier, "oatier" flavor. It is higher in fiber, and can be somewhat better nutritionally (having the bran and the germ attached). However, brown rice has an entirely different texture than white rice; it burns quite easily when cooked in fat, it doesn't absorb flavors in cooking as well, and it takes a fair bit longer to cook (because you have to soften the bran).

That said, in dishes where you are substituting rice for barley, cous cous, qinoa, lentils etc... Brown rice can be a better choice (BECAUSE of the protein and fat difference, the firmer texture, and textural contrast between the bran and the endosperm).

Brown rice is sometimes be a better foundation for a clean, fresh preparation of ingredients served on top of it than just a plain white rice. Brown rice can also make a better companion to wild rice in a dish.

Hmm...  Wild Rice then?

Wild rice isn't actually rice, in the conventional sense. It's actually the starchy seed of a river grass, closer to a wheat, oat, or barley than a true rice. It takes forever to cook, doesn't release starches, and cooks to a completely different flavor or texture than true rices.

That doesn't mean it can't be tasty, and doesn't make a good complement to other rice dishes.

For cooking, you can treat wild rice in many ways like a lentil, barley, or qinoa. It can be nice to add some parboiled wild rice (as can lentil, barley, or quinoa) to a flavored rice dish (especially a vegetarian dish, as they add both texture and protein), but it's not really suitable as the primary element of a "rice dish".

Generally, if added to another dish, wild rice varieties have to be parboiled first; because they can take hours to cook, vs. the 20-30 minutes of simmer time most medium or long grain rice dishes take.

Also, parboiled wild rice can make a good flavor and texture addition to a ricelike pasta dish, like fideo, orzo, risi, risoni, mittolini puntine etc... (or ricelike pasta can be added to a rice dish, for a textural and flavor variation).... but again, that's another topic for another day.

Okay, I know about rice(s)... Now... what technique and gear?

With plain rice, you'll usually cook it in a high sided pot, saucepan, or steamer (or a dedicated rice cooker, which amounts to the same thing).

When you're cooking in a high sided pot, the lower surface area; and higher thermal mass, retained moisture, and retained heat; help the rice to finish soft but firm, and moist but not wet (when the rice is mostly finished but not quite, you stir it up thoroughly; then take it off the heat, cover it, and let it finish using residual heat and moisture).

Most flavored rice dishes are a bit different.

You'll want to start with a relatively low sided, non stick, sautee pan, frying pan, saucier, or skillet (preferably with a lid); not a pot or straight sided saucepan.

The large surface area and low sides of these cooking vessels help you evenly and quickly cook the rice and seasonings in the fat at the beginning of the process; and to cook off the liquid to the desired texture at the end of the process.

The "secret" to a flavored rice dish, is that you "cook it twice"; first cooking seasonings and the rice grains out in a fat (to add flavor, improve texture, release and convert starches, and reduce simmer time), then simmering the dish out to your desired texture (NOT boiling or steaming)

Okay, now how do you actually cook it?

First, select and gather your rice, pan, fat(s), seasonings, flavorful liquid(s), and accompaniments (more on those below).

With a flavored rice dish, you want to have everything in place at the beginning if possible. You may also want to pre-cook some elements; for example rending flavorful fat out of bacon or sausage.

You're going to toast the rice in a flavorful fat, but you'll want to prep your seasonings first, because you'll be cooking them out into that fat either while you're toasting your ride, of before you add it.

Oh wait... we need to talk about the fat...

So... "flavorful fat"... what exactly do I mean by that?

Well, your choice of fat is crucial to a flavored rice dish, especially a creamier, saucier rice dish.

You won't be draining the fat (or at least not all of it) out of the pan, it's going to end up in every spoonful of your dish. Given that, you're going to want to start your dish with a fat that you're going to enjoy eating at the end. A fat that cooks well, and ends up with good flavor and texture in the dish.

As always, every element of a dish should "do something"; even the pan, and the oil, you are cooking it in. If you aren't improving texture, or flavor, or mouth feel etc... with every element...

...well, why not, when you could be?

Most of the time, for my flavored rice dishes, I like to use "a bit too much" butter (about 2tblsp per cup of dry rice) that's been cooked out to nutbrown; because I like the flavor, and I like how well it toasts the rice, and the aromatics.

Butter has some disadvantages though. It has a high water content, so you have to cook it out before can toast or saute effectively. Also, it has a low smoke point; and because it has a lot of dairy solids (which add flavor and texture to foods cooked in butter, and enhance browning), it can be very easy to overheat and induce off flavors, or simply burn it.

I still like using regular butter though, because I prefer the flavor.

If it's handy, you can use clarified butter, which still retains some butter flavor, aroma, and mouthfeel; while offering a higher smoke point (and no dairy solids to accidentally burn). Many Indian flavored rice dishes begin with clarified butter for example (including the classic rice pilaf).

You can also use a rendered flavorful animal fat, like bacon fat, sausage fat, beef fat, or schmaltz. They have a higher smoke point, and lower solid content than butter, but still give great flavor (just watch the salt content). And you can always mix them with butter to get both flavor profiles.

Just as an example, natural homemade, or local deli made, schmaltz; is almost ideal for making a creamy rice dish (commercial "mass produced" schmaltz is garbage, and it goes rancid or picks up off flavors quickly. That's why it's not found in most markets).

Schmaltz produces a spectacular flavor and mouth feel that can't easily be replicated with any other fat (that's why commercial matzah balls never have anything approaching home made flavor or texture by the way... no schmaltz).

In fact... just sauteing rice in schmaltz, then using chicken stock or broth (maybe with a bit of acid added, like white wine, vinegar, or lemon juice) as the cooking liquid, makes a really spectacular rice side dish. It has a deep chicken flavor that you don't quite get in any dish made without schmaltz.

If you're already making some bacon, sausage, ham, sauteed chicken etc... for the dish, you might as well use that highly flavorful fat to toast the rice as well (if you are using fresh uncured whole pieces of meat, undercook it just a bit so you can finish it hot in the pan at the end).

...Doubly so if you're making a rice and beans dish; so you can sautee the beans in the fat as well (either before, or with, the rice; depending on the respective cooking times and preparations of each).

Oh and  if you don't have enough volume of flavorful fat after you've cooked out your meats, you can always just add some butter.

For non-lacto vegetarians (or if you don't have, or don't like butter; or need to use a shelf stable fat), you can use any light oil that has a reasonable smoke point; but I strongly recommend using a flavorful oil.

A light olive oil (don't bother with a fine EV, sauteing with it will burn off the delicate flavor anyway), peanut oil, sesame oil, or a prepared flavored oil like a chili or herb oils, can add a big dose of flavor.

Actually, if you're cooking large chunks of meats or veggies, you're often better off using a light vegetable or light nut oil over an animal fat; simply because of their clean cooking characteristics.

Vegetable and lighter nut oils generally cook at a higher temperature than animal fats without burning, smoking, or developing off flavors (excepting ultrarefined pure lard, which is the only common animal fat that has very high smokepoint; but has very little flavor compared to other animal fats). This drives moisture off the surface of the meat or veggies faster, producing better browning and crust.

Unfortunately, many flavorful oils (and fats in general) have low smoke points, are not shelf stable, or are just damned expensive; but you can preserve their flavor in the cooking process by mixing a stronger flavored oil, with a lighter more neutral oil; or by mixing an animal fat with a vegetable fat.

You can also preserve flavor of a lower smoke point fat by cooking longer at a lower temperature (so long as you aren't sauteeing big chunks of meat or vegetables, which will tend to express too much liquid -and in the case of veggies, develop mushy texture; or for some meats can get tough- when cooked at lower temperature).

Toasting rice in oil (particularly with aromatic seasonings) is an application that is very well suited to cooking slower at a lower temperature. This makes it easier to get the rice (and the aromatics, garlic, rosemary etc...) very well toasted, without accidentally overbrowning.

I often like using a mixture of a bit of light neutral or flavorful nut or vegetable oil (olive oil, peanut oil, or sesame oil) and a bit of butter (cooked out in the oil to nut brown); particularly if I'm cooking some meat or vegetables in the fat. This gives me some of the advantages of each type of fat, and enhances browning (the dairy solids in the butter coat the food being browned).

You can also use a mix of a highly refined but relatively low flavor animal fat like lard, with butter; to retain some of the flavor characteristics.

Basically, you get some the harder sear of the vegetable oil or lard, but still retain some of the nutty, savory, rich flavors of cooked butter. You just have to make sure that you brown, but don't burn, the buttersolids.

Really, you can use anything you like, so long as every step and every ingredient is ADDING FLAVOR, or improving texture, or preferably both.

Just don't use "vegetable oil" or "shortening" or god forbid Margarine...

...In fact, never, ever, under any circumstances, use margarine for ANYTHING. 

It's not food... it's lubricant made solid through hydrogenation...

Okay... got the fat thing... now... what do I actually DO with the fat?

You cook with it...

Okay okay, yeah I'm being a wise ass... it IS me after all...

To be serious for a second, your goal here is to build flavor, and improve texture; while also reducing the total cooking time of the rice.

The foundation of a flavored rice dish, is the flavorful fat, followed by the aromatics and seasonings.

Start out by with bit of garlic, cracked peppercorns, maybe some pickled peppercorns, a bit of paprika (smoked paprika if you have it), a bit of ground hot mustard, and whatever other seasonings (or vegetables or accompaniments that may need to start cooking now).

Cook the seasonings out in your flavorful fat before you add the rice, if you want a bit deeper, and more complex flavor; especially if you're using either whole or fresh cracked spices, or a prepared spice blend (to cook out the graininess).

I should explain a bit about the mustard before we go on

Mustard is a flavor kicker, that wakes up the sense of taste and smell; as well as helping to emulsify the fat and flavorful liquid. It's important, so don't leave it out. If you don't like the taste of yellow mustard, don't worry about it. Ground hot mustard tastes nothing like yellow mustard, and when used as described, it doesn't really add any kind of "mustardy" flavor.

Remember, this is all about building flavor

The bulk of the dish is going to be rice... probably white rice... which means every step of the way, you should be building as much flavor as possible. I keep harping on that, because it's really important. Otherwise, you're just having "some rice".

While you're toasting the seasonings out, you can add a dash of either neutral spirit to extract more flavor from the spices; or some flavorful spirit (I like a bit of bourbon, cognac, or triple sec, depending on the flavor profile I'm looking for, or looking to complement) to both extract more, and add some complementary flavors and aromas (you get a great overtone with a sweet, well flavored liquor). Literally, add just a splash, and let it cook off (flame it off if you feel like being showy).

There are certain flavors that are greatly enhanced and better extracted, with a bit of alcohol. This is especially true of hotter and smokier flavored spices (like chili peppers).

If you're doing aromatic vegetables, like a trinity or sofrito, you do that next, in the same pan, with your flavorful spice infused fat.

Once you've added as much flavor to your fat as you're going to, you saute the rice in it... In fact, you're sauteing to the point of toasting it.

Just before the rice is "overdone" in the fat (seriously, you want this stuff smelling like popcorn almost. Just before it starts to get bitter and burned is the maximum point of flavor), deglaze the pan, and then douse the rice with just enough flavorful liquid to keep it simmering for 20-30 minutes without having to stir it more than occasionally (usually about 2 cups of liquid per cup of rice).

For my flavorful liquid (more on that below), I generally prefer chicken broth, or chicken stock, even when served with something other than chicken. I give it a bit of kick with some acid, like lemon juice, vinegar, or white wine.

Pork or Beef broth or stock both work as well, but I think chicken gives a better depth of flavor (even when served with pork or beef). Vegetable broth or stock will serve for vegetarians. Fruit juice mixed with water can work very well depending on what kind of flavors you're going for. Even a little acidulated aromatic water will do.

Always have enough flavorful liquid to reserve some to make textural adjustments. It's better to start off too dry, and have to add more liquid in cooking, than to have to cook some off and ruin the texture of your rice.

Simmer it slow (do not let it come to a full boil. That'll make it hard to get a good final texture) until you can clear the pan with a wooden spoon, and taste it. If the rice is tender but not quite "done", you're good. If not, add a few more ounces of flavorful liquid, and cook that out 'til the texture is right.

Towards the end of cooking, I may add a bit of (or more) butter, for improved mouthfeel and flavor, particular if it's going to be a slightly (or very) creamy dish.

Once you're at that point, you can just cook it the remaining moisture off 'til it's whatever texture you'd like. You can stir it just enough to keep it from sticking or scorching, and you'll get a drier, more separated rice. Or, you can stir it constantly, adding a bit more liquid as you go, to get a softer, moister, creamier rice.

You can add a dash of cream or half and half at this point if you want a mock risotto. You can add cream in twice, once with the initial dousing, once at this point, if you want a creamier, saucier, mock risotto. You can also add some cream cheese in if you want a thicker, creamier sauce with a silkier mouth feel (especially if your rice isn't as starchy as it needs to be).

Finish with some shaved hard aged cheese, a bit of fresh cracked pepper, maybe some toasted pinenuts or almond slices, maybe a bit of fresh parsley or cilantro, maybe squeeze of fresh citrus (lemon, lime, whatever you like), maybe a dash of vinegar or soy...

This is the basis of any flavored rice dish. You've got pretty much infinite options from here:

If you're serving with beef, pork or poultry; add in some cumin, some fennel pods, and a bit of rosemary (with the seasonings). A bit of sage, thyme, or marjoram on top of that go especially well with poultry or pork, but not as well with beef.

If you'd like a bit of indian flavor, add some ground cardamom, and toasted green cardamom pods, a bit of ginger, turmeric, galangal, and fenugreek (or just some garam masala, but make sure you cook it out in the butter, just like the other spices etc..); and finish with some currants and almonds (and some indian or greek yoghurt if you want it creamy).

For thai, start with cumin, ginger, a bit of cilantro, some small dried hot peppers. Add some green curry paste, fine slivered leeks, and coconut milk in with the liquid. Finish with more cilantro, chopped chives, cashews, maybe a bit of chopped lemon grass, and a bit of yogurt, coconut cream, or half and half.

For mexican, start with cumin, some diced green chili, cilantro, a dash of cinnamon; and add some vinegar based hot sauce in with the liquid.

Kicking up the flavor some more... making it "meaty" with or without meat...

One of the key points in building flavor, is to never use water when you can use a flavorful liquid instead.

In most cases where you'd use water, using a broth or stock (so long as you keep an eye on your overall salt and acid) will almost always produce a more flavorful, and better textured, result.

Even when you're just boiling, it makes sense to use acidulated water; adding a bit of broth or bullion, some vinegar or hot sauce, some fresh citrus, salt, aromatics etc... Hell, even for pasta, potatoes, or other starches or vegetables, (unless doing so would modify the starches or proteins of what you're doing in an undesirable way), you'll get more flavor with adulterated water.

The point is to always be adding flavor, not just liquid.

If you don't have a stock or broth handy, you can make a basic aromatic acidulated liquid with about 90 seconds of prep.

Just add salt, cracked black pepper, malt vinegar or vinegar based fermented pepper hot sauce, some soy sauce, some worcestershire sauce, some fresh chopped up up and squeezed citrus (with the rind), and some fresh aromatic herbs (and maybe a trinity, sofrito, or mire poix).

Add some apple juice, orange juice, lemon juice, grape juice, or wine if you have some.

Simmer it all together for about 10 minutes (without letting it boil) for a brighter, fresher flavor; or boil it for 90 seconds to five minutes.

In classical technique terms, that's a basic court-bullion right there.

You can make a vegetable stock or broth pretty simply (and cheaply), in just a few minutes more.

Chop up and sauté in oil (or butter if you're not making it vegetarian. I like a light but flavorful cold pressed but not extra-virgin olive oil), some garlic, onions, peppers, carrots, and celery; with cracked black pepper, hot mustard, cumin, fennel, sage, thyme, and rosemary.

If you can salt and rest the chopped veggies an hour before you make your broth, you'll get a better result.

Add in some mushrooms or dried mushroom if you like. Dried mushrooms especially can add a lot of umami to a vegetable broth. You can add some nice dried tomatoes, as an umami booster as well.

Sauté them all 'til lightly browned, or even properly caramelized. The longer you cook the veggies, the less "fresh" flavor you'll get, but more depth of flavor, and umami you get.

In general, no matter what flavor profile I'm going for, I like to add some umami and flavor kickers to my flavorful liquid.

I like to add some vinegar based hot sauce; or some malt, wine, cider, or balsamic vinegar (anything that has been naturally fermented). This adds umami and depth of flavor. You can also add a bit of soy, and a bit of worcestershire sauce at this stage to build even more depth of flavor and umami.

Be sure to use a natural brewed soy, otherwise you don't get the big glutamate hit. Soy goes into almost anything that has salt (which is almost anything) as a flavor enhancer (just be careful of total salt content). Soy doesn't make food taste"asian" unless you use a lot of it, or add something like hoisin, plum suce, fish sauce etc...

I put a dash of soy, and a dash of a fermented pepper hot sauce in almost everything, to help wake up the pallet and enhance umami.

Remember, you're not going for a final flavor here, you're just adding that extra bit of depth, complexity, and savory feel.

So.... Is there any actual specific recipe stuff here?

Ehh... not really... that's kinda the point of flavorful rice dishes... It's more about technique, and options, and ideas than it is about a recipe.

To make a beautiful vegetarian main dish, fine chop and sweat some onion, pepper, and celery in when you're toasting the rice (maybe some asparagus tops too); then toss in some chopped or shredded carrots in with the liquid, and maybe some baby broccoli florets, snap peas, sprouts etc...

You can add in some lentils cooked in chicken broth, cooked red or black beans, or some cubed, sauteed, firm tofu; if you want to boost the protein.

Some fresh some chopped, seeded, salted, and drained tomatoes (or canned crushed, whole, or chopped tomatoes) can be added at this point if you like (you can peel them if you like, but it's not necessary. Just chop them very fine if they aren't peeled). They'll also add a bit of freshness, some sweetness and acidity, and some depth of flavor. Sautee them out 'til they're almost browned into a paste, and you'll add a lot of umami (at the expense of freshness and sweetness).

If you add fresh or canned (peeled, seeded) tomatoes, a bit of fresh basil and fresh oregano would be highly complementary.

For a very much NOT vegetarian dish, add in some cooked red or black beans and some sliced and crisped spanish chorizo or portuguese linguica, or some marinated and sauteed chicken pieces.

For dirty rice, saute some andouille or linguica (or taso if you can get it), and some chicken livers, and toss that in with some chicken flavored rice; with some red or black beans.

Red beans and rice isn't far off from this.. and the same basic techniques can be used in making the New Orleans classic. Soak the beans first, and saute them in the fat before you add the rice (to equalize cooking time).

Or for a variant on the most basic fundamental singaporan dish there is, make a chicken flavored rice base, and serve with boiled chicken on top, for "chicken rice".

Oh... and most of these dishes can be very easily converted into soups or stews, just by adding more stock or broth at the end of cooking; along with perhaps a bit more acid, some hard cheese, and more beans or meat (and more dairy for creamier soups).

Really, there's just an infinite variety available to you.

And be sure to check out:

Recipes for REAL Men, Volume 28 - The Nog Abides
Recipes for REAL Men, Volume 27 - That's too turducken hard

Recipes for REAL Men, Volume 26 - Hot Smoke
Recipes for REAL Men, Volume 25 - That's a Spicy Polpette
Recipes for REAL Men, Volume 24 - It's Meat, in Loaf Form
Recipes for REAL Women, Volume 23 - Some Like it Hot
Recipes for REAL Women, Volume 22 - Full Fat, Full Dairy, All Killer, No Filler
Recipes for REAL Women, Volume 21 - Forget About the Dough Boy
Recipes for REAL Men, Volume 20 - QDCBS (Quick and Dirty Chili Bean Stew)
Recipes for REAL Men, Volume 19 - Chicken Salmonella
Recipes for REAL Men, Volume 18 - I'll give YOU a good stuffing turkey (1)
Recipes for REAL Men, Volume 17 - REAL Coffee
Recipes for REAL Men, Volume 16 - DTG (Damn That's Good) dip
Recipes for REAL Women, Volume 15 - More Chocolate Than Cookie
Recipes for REAL Women, Volume 14 - Millions of Peaches
Recipes for REAL Women, Volume 13 - Mels 10,000 Calorie Butter Cookies
Recipes for REAL Men, Volume 12 - Lard Ass Wings
Recipes for REAL Men, Volume 11 - Bacon Double Macaroni and Cheese
Recipes for REAL Men, Volume 10 - It's the meat stupid
Recipes for REAL Men, Volume 9 - Labor Day Potatos
Recipes for REAL men, Volume 8 - It's a pork fat thing
Recipes for REAL men, Volume 7 - It may not be Kosher...
Recipes for REAL men, Volume 6 - Andouille Guiness Chili
Recipes for REAL men, Volume 5 - Eazza the Ultimate Pizza
Recipes for REAL men, Volume 4 - Two Pound Meat Sauce
Recipes for REAL men, Volume 3 - Highbrow Hash
Recipes for REAL men, Volume 2 - MuscleCarbonara
Recipes for REAL men, Volume 1 - More Beef than Stew