Wednesday, November 17, 2021

The trouble with Citrix

It  looks for all the world like Citrix... One of the more important technology and services companies of the last 25 years...  is in the middle of imploding...

... As many have expected would happen for quite a long time now...

...Not with a bang, but with several extended half hearted whimpers; a lot of corporate doublespeak, and a lot of very angry and disappointed people.

I  still have a fair few friends working at Citrix... good people all of them, who for some reason had faith that the companies management couldn't possibly stay that bad and that dumb for that long, and somehow they'd reverse course. 

They've been trying for years now, to get management to put REAL effort and support and investment and energy (For more than a couple quarters without reversing course or redirecting or "refocusing" or otherwise making it impossible not to fail) into the remaining useful and long term viable core technologies and solution sets; and into the interesting side developments with potential to become viable core, or substantial secondary or peripheral technologies, products, and solution sets...

...Critical things, that so many good folks have tried to make happen, and tried to make work,, and done their best to patch up around and make things work anyway when they didn't happen... for the last 10-15 years....

And of course, by far most importantly: they have tried their mightiest and best, to get management to invest in, develop, and fully support; all the PEOPLE absolutely necessary to make those technologies, and products, and solution sets; viable, and then successful,, in the marketplace. 

The people to support enterprise sales, and to develop, implement, and support enterprise solutions; to be useful, successful, and EASY to integrate into the operations of every organization; across the breadth of the multi plarform, multi-environment, mixed local and remote, mixed physical.and virtual; mixed on prem, and hybrid cloud, and public cloud, and managed service, and infrastructure and software as a service environments... And every other possible environment and technology, and sector, and solution set... 

Which is the only way that those technologies and products and solution sets... and Citrix as a technology and services company...  can compete and be viable in that marketplace

The people, and the services they provide, that should be Citrixes REAL core business, and biggest source of revenue, and profit.

...But thats not happened so far... and Probably won't in the near future...

In fact... rather the opposite Is happening right now... and has been happening for... way too damn long now. 

Ok...this is going to be a long one, because it needs to be, to illustrate the totality of the... it's not a failure as such, as it is just a total abandonment by management of an entire business, because they didn't know when they needed to change, and how they needed to change when they did... And just couldn't be bothered...

... The problem with Citrix...

A lot of folks have made sage pronouncements about Citrix like "Oh they stopped innovating, and once tech companies stop innovating they're done for"... which...yeah it's generally true, but it doesn't really capture the scope, breath, and depth of the issue...

Citrix didn't just stop innovating... Their business model stopped being viable; because the core technical and business operational functions their products and solution sets  performed for their customers; either became no longer necessary or relevant; or they were eventually wrapped into the base functionality of the platforms and systems they used to provide those functions for. 

Virtualization, remote access, published virtual desktops, and published virtualize applications; are now all included for free or very low cost, in every commercial server operating system or hyperion platform. 

Virtualization, remote access, published virtual desktops, and published virtualized applications; are now all included for free or very low cost, in every commercial and most free  server operating systems, and in every hypervisor platform (again, whether commercial or free); with enterprise class support commonly available at relatively reasonable premiums. 

...Sure, Citrix generally provides a lot of additional functionality... usually in the area of enterprise management and support features, and actual enterprise support... that the stuff the other companies are giving away for free does not (or only does so in a more limited way)... But citrix ain't cheap, and it's awfully hard to compete as "expensive, but somewhat better" against "it's free and it works"... Especially when the product has become less and less "better" over time...

...And frankly... Citrix as an entity, and their core management, have really never dealt with this fact, never mind managing to find or invent a new business model for themselves... They're STILL trying to sell at a hefty premium, what everyone else is giving away for free... 

...and in fact which they ALSO give away for free, in a somewhat decontented, and of course entirely unsupported form... With the message that "you already have the skills and knowledge and configurations ND tools to manage our stuff in your enterprise... why would you risk changing?"... A message that... Let's just say hasn't notably worked out for them so far.

Then, instead of figuring out a new way for a new business model to work, with the technology and people and intelligence to back it; Citrix management just tried to kinda tweak the old model, and announce BIG BOLD NEW things, that really weren't... They were just the old things repackaged and relabled and rejiggeed to sort of do something different from how they did it before, or to sort of replicate some other companies or some other platforms functionality... "Now in that good old citrix color and flavor you love" Never mind trying to actually do something new, or better, or otherwise provide some kind of actual competitive advantage and reason to chose their solution over others..

Then, instead of figuring out a new way for a new business model to work, with the technology and people and intelligence to back it; Citrix management  just tried to kinda tweak the old model, and announce BIG BOLD NEW things, that really weren't... They were just the old things repackaged and relabled and rejiggeed to sort of do something different from how they did it before, or to sort of replicate some other companies or some other platforms functionality... "Now in that good old citrix color and flavor you love" Never mind trying to actually do something new, or better, or otherwise provide some kind of actual competitive advantage and reason to chose their solution over others.. 

And just like they sorta kinda copied other vendors products and solutions... because "hey, the other guy is selling them pretty well,, we should be able to too right?"... they sorta kinda tried to  copy the business models of SEVERAL other vendors...including multiple times trying to execute on multiple contradictory and mutually exclusive models all at once...

...Really...At least from the outside anyway... It seems like management just sort of closed their eyes,, crossed their fingers and pointed their heads down; and hoped that their (once huge but now rapidly and continually shrinking) legacy installed base and relationships... and basically, inertia...would carry them through...

Then, even once it was completely impossible to not know that idea... which could never have worked to begin with...  was inevitably and unalterably failing; they continied to ignore reality, and pretend otherwise. Because exec bonuses maybe? Can't think of any other possible reason, unless those supposedly smart and successful people really CAN be that dumb, and that blind, for that long...

How bad is it, and how do I know it's that bad?.

I have spent almost the entirety of the last 25ish years, working variously as a Citrix customer, partner, or otherwise a contractor or consultant...

...Specifically in roles as presales, post sales, internal, and independent outside:

-- Solutions architect, solutions delivery team leader, and customer success team technical manager.

-- Infrastructure architect and operations manager (traditional on prem, mixed on prem and third party co-location; utility compute and high reliability exascale HPC and OLTP, on prem, distributed, and hybrid; hyperconverged, on prem, distributed, and hybrid; hybrid cloud, full private cloud, and full public cloud... In traditional, DevOps, MSP, and every other kind of operational concept).

-- Service delivery architect and operations manager (including traditional, devops, MSP, and every other kind of perational concept)

-- Information security architect and operations manager (traditional, devsecops, and otherwise); assessor, and auditor.

-- Policy Process and Audit, and Governance and Regulatory Compliance; architect, manager, assessor, and auditor.

-- Disaster prearedness and recovery, business continuity, emergency response, and incident response; architect, operations manager, tactical/emergency/response team, leader/coordinator; investigator and forensic analyst; and mitigation and return to operations coordinator/team leader. 

... And I'm going into way too much detail and way too deep inside baseball, for a specific reason that I will get to in a minute...

...Almost all of these in high risk, highly regulated environemnts, including government, defense, medical and pharmaceutical, and financial; sectors, and legal and regulatory regimes and frameworks

I said all of that, to make it absolutely clear just how badly, and at what scale Citrix have failed in... basically everything they should have been doing for the last decade or more really.

Because, since 2009, in all of the roles I have mentioned above... 

...All areas where prior to about 2005-2009ish Citrix had dominant market positions.

And they had those dominant market positions, in large part, because although they may not have had the best or the ideal technology or implementation out of the box; they had in house (and local regional in every region, including specialists for different markets and industries or sectors, and specialists for unique technical, operational, or legal/regulatory requirements) solutions and subject matter experts, and pre and post sales team support, including implementation and customer validation/UAT support before handing over to a sustaining engineering and support function; and real enterprise customer support from the help desk on up to real support engineers and sustaining engineers; who understood all of the above, and could be counted on to make things work, and solve problems with the customer, or reseller, or end user etc...

You had to pay fairly dearly for it.. But it when you did, at that time, it was worth it. They gave good value, and they made it work, often when others could not. 

Hell.. prior to 2001, all of those markets I mentioned above... all very particular and challenging spaces to compete in... were just THEIRS period. Or at least they were if you had to deal at all with Windows or cross platform, cross environment, remote or published resources, in interactive graphical sessions... which almost every big company and a hell of a lot of small and medium sized companies did... and still do... to some extent or another. 

... But since between 2005 and 2009... almost 17 years at the outside, and almost 13 years at the least... 

...and across all those roles and environments and segments etc... All of which are core mission spaces for Citrix, and critical to the future of any company wanting to operate therein...

I have only seen TWO actual  completely new, de novo blue sky, major implementations of core Citrix solution sets, in critical and core operational functions for the organizations in question.

Two... Yes, really... TWO...

In EVERY OTHER CASE... and we're talking hundreds of clients and customers and partners and employers; across thousands of sites... any growth in Citrix installed base in those organizations was essentially by default, or forced, or something they settled for because of mass and inertia and long term contracts; or was otherwise just something they had to accept, because for whatever reason, they had no other viable option, or any other option was just too difficult or not worth the effort etc...

In almost all cases they were either already in the process of getting rid of Citrix as much as they could (and often already working AROUND their existing Citrix solutions footprint);  investigating the idea and what it would take to move to an alternative; or doing competitive evoluations of alternatives; or at the very least they WANTED to stop depending on Citrix solutions; just that for whatever reason they were stuck for now, and once they could, they'd be replacing or just removing Citrix as soon as it was practical to do so.

And dozens of them did... They either completely ripped Citrix out of their operations entirely, or they reduced them to minimal footprints, of things they absolutely couldn't get rid of or get a viable alternative for... and only until that limitation changed. 

A lot of them TRIED to expand their Citrix solutions footprint...  lot of them actually DID... AND AS I said there were exactly TWO large enterprise customers, with actual brand new, didn't have at least a moderately significant relationship with Citrix before,  major core functionality solutions sets; purchased, architected, implemented, and put into operation...

...And both of them had projects to minimize dependency on Citrix, or rip and replace them completely, within two years of handover on these multi year, mulmillion dollar evaluation, architecture, implementation, and testing projects....

I saw seat counts for existing or replacement products/solutions grow, and expansions to existing products and side products piggyback off them... Almost always in organizations with major enterprise licensing and discount agreements and large longstanding preexisting Citrix footprints... 

And occasionally, though rarely, I'd  see new implementations of small solutions and systems... a few systems, with a few seats, for one or two products... Not major critical or core operations, or just for one or a couple  critical but segmented off things, that were effectively one offs and exceptions, and Citrix was either the enterprise default solution, or for some reason it was either the cheapest solution (again usually an enterprise license agreement), or it was the only solution that supported a specific oddball edge case technology, product, configuration, process etc... 

But that was it... It was as if they were running in place.... Or worse, running backwards...
Suee, they still had a lot of legacy customers... Whether it was because the customers had a huge installed base already and couldn't  afford the license costs, training costs, support costs, down time etc...to move off of Citrix. Or they had specific contractual, legal, regulatory, supportability, or edge case technical  requirements that prevented them. Or for whatever reason there simply wasn't any viable option to not use Citrix...

...In every single case, none of them wanted to be dependent on Citrix anymore, and most didn't want to use any Citrix solutions period...

Why did they not want to use Citrix anymore, after years... sometimes decades... of relying on them?

Mostly, it was because for the last decade or more, the experience of having to rely on Citrix, became more and more unpleasant for them... and less and less reasonable, supportable, viable, or even tolerable. 

Sometimes, the product or solution was literally permanently and unfixably broken and they couldnt use it and had to buy or build and implement workarounds, or entire alternate systems.

 Sometimes it became a bodge job of just barely functional barely supportable chewing gum and baling wire, but they couldn't make anything else work. 

Sometimes  licensing had become so stupid and harmful that they couldn't or wouldn't pay for it anymore.

Sometimes It was because the previously excellent support for weird solutions, edge cases,, unique environmwnts and requirements etc... disappeared, and they were replaced with offshore script readers and box checkers... 

Whatever it was, over the past more than a few years,, there have been myriads of compelling reasons to NOT use Citrix, and not very many good reasons to do so. 

...And then, they just stopped even pretending to try to have a viable plan or technology moving forward, and for some indecipherable reason,, decided to lean into the fact that they were now IBM from the 70s, after cannibalizing their own channel and their own customers, they were just going to try to minimize costs and roll those existing customer seat counts and license revenues up year to year...

...And that's kinda the ballgame in this industry folks...

That doesn't mean there aren't still plenty of good people hanging on...  But that doesn't matter anymore ubfortunately... sadly... 

Because a company with no business model; few relevant products they can actually sell into relevant markets, and operationally support for relevant customers requirements;  and few, or zero, core technological, functional, operational, financial, or otherwise in any way compelling reasons, for any potential customer to select that companies products, solutions, and services, over those of any other vendor... 

...Except perhaps by inertia, edge exception, or ignorance of other options... 

...They aren't really a technology company anymore... They're just a contract servicer, cutting whatever costs they conceivably can, without impeding what is really the only meaningfully functioning part of the business: collecting whatever "dead money" revenues they can from their existing contracts and customer base, for as long as they can; making their quarterly earnings per share appear to be as high as they can plausibly make them appear... 

Thus, "executive management" can squeeze the most possible cash compensation for themselves out of this one time great company; before finally breaking up the rusting hulk, to extract any last pennies of value, from whatever bits and pieces anyone might offer any money for; of the remains of what used to be Citrix.

Tuesday, November 16, 2021

Kafka got nothin' on Taxachussetts

In 1999, I had a Massachusetts drivers license, and officially a Massachusetts residential and mailing address, and I bought and registered a car... a Saturn SC2, the only brand new vehicle I've ever purchased for myself in fact... in the commonwealth of Massachusetts. 

In 2000, I lived in California, with a California car registration for that same car (and another one that I purchased as a present for my first wife... A special edition Jeep Cherokee, in Kentucky wildcat Blue, as she was a UK Wildcat) and California driver's license. In 2001, 2002, and 2003, I lived in Ireland. In 2004 I moved back to the U.S, living in Arizona. 

Subsequent to moving back to the U.S. in 2004, I've lived and had drivers licenses and car registrations in Idaho, Florida, and both Arizona and New Hampshire multiple times... moving back and forth a couple times for work, family, and health reasons. 

In 2000, 2001, 2002, 2003, and 2004, Massachusetts attempted to charge me vehicle excise tax for a vehicle that was not registered in the state, nor was it present in the state... Nor was I resident in the state, nor did I have a driver's license in the state. In two of those years they also, somehow, issued me traffic citations, and cited me for allowing my registration to expire, and not having insurance on said car with said expired registration. 

... Said registration was in fact CANCELLED in 2000 with the license plates turned back into the Massachusetts registry of Motor Vehicles. Not only that, but the vehicle itself was sold back to Saturn in April 2001, just before I moved to Ireland... 

This didn't stop Massachusetts from tacking on fines and fees and interest and penalties... Then issuing a bench warrant for me, when I didn't pay those taxes and fines and fees and penalties I didn't actually owe, and suspending my drivers license which I didn't actually have... and of course not notifying me, as I hadn't actually lived in the state since 1999.

That process began almost 23 years ago, and every few years since then, it has caused me massive issues. Because other states have honored these fraudulently issued warrants, suspensions etc... And have then suspended my driver's licenses canceled my insurance and car registrations etc... Then I would get pulled over after years of no issues, and suddenly I'm driving on a suspended license and have a warrant... again...

Over the years I have paid Massachusetts more than $14,000 in taxes, and the associated fees, and penalties, and court costs, and fines,, and every other thing they could think of to extort money from me,, that I didn't actually owe them, to attempt to resolve this problem.

I have also spent thousands of dollars in fines and fees and reinstatement charges and appeals fees etc...in those other states in which I actually lived. 

And then there's the multiple attorneys I've retained to try to deal with these issues, and THEIR fees.

Eventually I would end up having to go to an appeals process in each of the states I lived in that actually honored the suspension notices and warrants from Massachusetts, to get their motor vehicle and licensing department to ignore Massachusetts invalid and ultimately fraudulent attempts to suspend my drivers license over and over again etc... etc... Or I just happened to live in states like Idaho, Florida, and New Hampshire, that had had enough experience with Massachusetts that they did not honor their requests anymore.

...Massachusetts is well known among the other states for misbehavior like this... They do it over taxes, tickets, child support... anything they can do to try to extort more money from anyone they can. Many states now either simply ignore these notices from MA, or they automatically grant appeals to them, once you start the appeals process and can show they are not valid. 

Unfortunately Arizona does still (or rather again, as for some years they stopped accepting and honoring MA suspensions,, revocations, and bench warrants, but a few years ago started honoring them again for some reason) accept and honor  such requests from Massachusetts. So, when I moved back to Arizona three years ago, and I went to convert my New Hampshire drivers license over, I ended up having to start the whole long, painful, and expensive process all over again. 

...Except once COVID hit, they stopped processing all appeals, and required appointments scheduled months in advance to do... anything basically... as did the MA Registry of Motor Vehicles, and the excise tax authority in Massachusetts, etc... 

Except this time, after paying Massachusetts over $2500 more in fines and fees I didn't actually owe, using their newly instituted electronic expedited payments system, I was able to short circuit the fact that I couldn't go into an office to talk to a live human; and I managed to get them to find in their own records, why they kept charging me over and over again, suspending and warranting me over and over again, etc... 

ONE of the court records, from ONE incident, 20 years ago... actually it was from 1996 but the mis-filing of the disposition was from 1999...  had been misfiled with the wrong disposition and status. This had then been entered into a state computer incorrectly in 2004, and had never been fixed, and in 2006 the original paper record had been put into a box in a court basement....and thankfully said box had not yet been disposed of, as it had been scheduled to be destroyed in 2016.

I managed to explain all of the extended saga of pain and cost and inconvenience and trouble, to a sympathetic court clerk in one of the states court districts. That clerk was able to find the original record, and the proper disposition and status for the case, corrected it in the country courts computer system, and sent a fax of the correction to the states clerk who fixed in I the states computer system, and to the MA state police, and the registry of motor vehicles... and then she called each of those offices for me, and got them to get the correction off their faxes and fix it in their computers... At which point I was able to get the other two courts that had open dispositions which could not be closed until the first courts disposition was corrected, and the Massachusetts registry of motor vehicles suspension, revocation, and reinstatement office, to correct their records... and to credit me for the payments I had just made... though not the thousands in prior payments I had paid but was never credit for and which kept being marked as "not discharged" because of this ONE COMPUTER ERROR.

Oh, and MQ being MA and the RMV being the RMV... they still charged me another $1100 in a "reinstatement fee" from the registry of motor vehicles, to lift the  "revocation of my driving privilege" and "reinstate my driving privilege".

I  paid that $1100 last year. 

I also had to talk to the Massachusetts state police office that dealt with warrants and "fugitive" records because they had officially been periodically reporting me to the FBI, NCIC and NICS as a fugitive. Which, of course, caused my NICS checks to be denied several times and forced me to go through multiple appeals on that level, as well as resulting in my being arrested several times as a fugitive with an active fugitive warrant (each time being let go once they actually got the details from Massachusetts that it was only ever a bench warrant for failure to appear for a traffic citstion and for unpaid civil traffic fines, not even minor misdemeanor criminal warrants, and that I wasn't actually a fugitive). They were able to immediately fix the problem once the court disposition record was fixed.

 I then had to talk to the FBI to get them to fix the NICS records based on the correction from Massachusetts... that required some paperwork and some faxing... But the FBI is also long used to Massachusetts screwing everything up, and they actually had a standard process in place to fix the NICS records once I got the MA state police to fax them the right paperwork. The NICS official Inspoke to actually ranted about MA and their crap and how they've been screwing up NICS for years, and everyone was tired of it etc... 

Then I had to pay Arizona another $500 to reinstate my driving privilege here in AZ after the out of state suspension and revocation. 

All of that was over a year ago now... But I still wasn't able to get a new driver's license because of covid, making it take months to get an appointment to clear it all up on the Arizona end etc... etc... 

Finally, six weeks ago, after YEARS AND YEARS AND THOUSANDS AND THOUSANDS OF DOLLARS... I thought I was finally done... I had checked every couple months to make sure that MA hadn't reentered me as a fugitive or entered another suspension etc... And I went to go get my now expired NH driver's license converted over to an AZ driver's license... 

Except no... They still couldn't issue me a new drivers license, because even though I'd had a driver's license in state for years before this, and they knew I was who I said I was and had enty of documentation with multiple forms of acceptable ID off the list etc... etc... 

...My current documents proving my identity, all listed my name slightly differently... 

Some said Chris, and some Christopher. Some had my middle initial, some my full middle name, some had neither. Some listed my generational suffix the fourth, some did not... and because of RealID they couldn't issue me any new ID or DL without two documents from the acceptable list that both matched perfectly.... which, of course, I did not have. 

Amazingly, absurdly, ridiculously, and of course arbitrarily and capriciously, there was no way for them to show some actual human judgment and discretion, or the most basic degree of sense... It all had to exactly match no matter what.

So, I had to go and apply to the social security administration for a new card, and pay the state registry of vital records in Massachusetts, yet another $100 for expedited processing and shipping, to get certified copies of my birth certificate; and to receive several pieces of mail at my current address from either a governmental agency, a utility, or certain acceptable financial institutions, all of which exactly matched how they listed everything, and had my name all the same way, with current address all the same way etc...

Last week, the final piece I needed arrived.

Yesterday, after over 20 years, and by now I believe well over $20,000 in accumulated fines and fees and penalties and assessments I never actually should have had to pay... and after having been arrested multiple times for warrants that never should have been issued, for offenses I didn't actually commit... and having had firearms purchases denied and had to appeal those.. and had my FFL denied and canceled and having had to appeal THAT... and after paying yet another $97 in licensing fees to the state of Arizona, for the actual license and ID etc...

...FINALLY...

I was issued a new drivers license, without having to go through an appeal, because Massachusetts is no longer trying to screw me for no good reason, after having extorted all the money they could out of me, even though I haven't officially lived there in almost 23 years. 

I now have a new Arizona drivers license, with motorcycle endorsement... and hell, I even passed the eye test... and separately an Arizona state ID that is RealID compliant for federal purposes... and two Arizona disabled placards... Or at least I have the temporary copies thereof and the RealID digital mobile version (which is legal for use as ID in AZ and federally) on my phone; and in anywhere from 4 days to 4 weeks, I should have the actual permanent physical copies thereof.

... Oh and my car registration should be fixed and associated properly with my Arizona Motor Vehicle Department account, and driving record... Somehow either the dealership or the MVD screwed that paperwork up, and it ended up creating a duplicate record with all the same data but NOT associated with my actual identity or account... So I couldn't actually renew that registration without getting it fixed first... since my car was registered a year ago now, and needs renewal this month.

Kafkqesque doesn't BEGIN to describe this entire process... But at least for now, it finally seems to be fixed.

...Until the next time Massachusetts screws something up anyway...