Gartner: 85% of AI Implementations Will Fail By 2022

A series of 2019 predictions by Gartner were reported on by Venture Beat on June 28, 2021. As explained in a prior post, “AI”, or machine learning, relies on datasets and algorithms. If the data is imperfect or incomplete, a computer has a chance of giving bad output. If the algorithm that tells the computer what to do with the data is imperfect, the computer has a chance of giving bad output. It’s easy to point to anecdotal cases where “AI” makes a bad call. There have been reports of discrimination in facial recognition technology, driverless cars killing people, or Amazon’s algorithm deciding to fire drivers that are doing their job. I’ve seen plenty of data on the failings of overhyped technology and commercial ASR. What I hadn’t seen prior to today was somebody willing to put a number on the percentage of AI solutions that succeed. Today, we have that number, and it’s an abysmal 15%.

Perhaps this will not come as a surprise to my readers, considering prior reports that automatic speech recognition (ASR), an example of machine learning, is only 25 to 80 percent accurate depending on who’s speaking. But it will certainly come as a surprise to investors and companies that are dumping money into these technologies. Now there’s a hard number to consider. And that 15% itself is misleading. It’s a snapshot of the total number of implementations, not just ASR. ASR comprises a percentage of the total number of implementations out there. And it’s so bad that some blogs are starting to claim word error rate isn’t really that important.

Judge,
I know I botched 20 percent of the words.
But word error rate really isn’t that important.

That 15% is also misleading in that it’s talking about solutions that are implemented successfully. It is not talking about implementations that provide a positive return on investment (ROI). So imagine having to go to investors and say “our AI product was implemented with 100% success, but there’s still no money in this.”

The Venture Beat article goes on to describe several ways to make AI implementation a success, and I think it’s worth examining them briefly here.

  1. Customizing a solution for each environment. No doubt that modeling a solution for every single business individually is bound to make that solution more successful, but it’s also going to take more staff and money. This would be almost like every court reporting company having their own personal software development staff to build their own CaseCAT or Eclipse. Why don’t they do that? It’s hopelessly expensive.
  2. Using a robust and scalable platform. The word robust doesn’t really mean anything in this context. Scalability is tied to modular design — the ability to swap out parts of the program that don’t work for specific situations. For this, you need somebody bright and forward thinking. They have to have the capability to design something that can be modified to handle situations they may not even be aware exist. With the average software engineer commanding in the ballpark of $90,000 a year and the best of them making over $1 million a year, it’s hopelessly expensive.
  3. Staying on course once in production. This involves reevaluating and sticking with something that may appear to be dysfunctional. This would be almost like the court reporter coming to the job, botching the transcript, and the client going “yes, I think I’ll use that guy again so that I can get a fuller picture of my operational needs.” It’s a customer service nightmare.
  4. Adding new AI use cases over time. Piggybacking on number 3, who is going to want to continue to use AI solutions to patch what the first solution fails to address? This is basically asking businesspeople to trust that it will all work out while they burn money and spend lots of time putting out the fire. It’s a customer service nightmare.

I really respect Venture Beat trying to keep positive about AI in business, even if it’s a hopelessly expensive customer service nightmare.

With some mirth, I have to point out to those in the field that believe the stenographer shortage is an insurmountable problem that we now know machine learning in the business world has a failure rate that’s right up there with stenographic education’s failure rate. Beyond the potential of exploiting digital reporters or stealing investor money, what makes this path preferable to the one that has worked for the last hundred years? As I wrote a week ago, the competition is going to wise up. Stenographic court reporters are the sustainable business model in this field, and to continue to pretend otherwise is nothing short of fraud.

CART v Autocraption, A Strategic Overview For Captioners

With the news that Verbit has bought VITAC, there was some concern on steno social media. For a quick history on Verbit, it’s a company that claimed 99 percent accuracy in its series A funding. In its series B funding it was admitted that their technology would not replace the human. Succinctly, Verbit is a transcription company where its transcribers are assisted by machine learning voice recognition. Of course, this all has the side effect of demoralizing stenographers who sometimes think “wow, the technology really can do my job” because nobody has the time to be a walking encyclopedia.

But this idea that Verbit, a company started in 2016, figured out some super secret knowledge is not realistic. To put voice recognition into perspective, it’s estimated to be a market worth many billions of dollars. Microsoft is seeking to buy Nuance, the maker of Dragon, for about $20 billion. Microsoft has reportedly posted revenue over $40 billion and profit of over $15 billion. Verbit, by comparison, has raised “over $100 million” in investor money. It reports revenue in the millions and positive cash flow. Another company that reports revenue in the millions and positive cash flow? VIQ Solutions, parent of Net Transcripts. As described in a previous post, VIQ Solutions has reported millions in revenue and a positive cash flow since 2016. What’s missing? The income. Since 2016, the company hasn’t been profitable.

I might actually buy some stock, just in case.

Obviously, things can turn around, companies can go long periods of time without making a profit, bounce back, and be profitable. Companies can also go bankrupt and dissolve a la Circuit City or be restructured like JCPenney. The point is not to disparage companies on their financials, but to give stenographic captioners real perspective on the information they’re reading. So, when you see this blurb here, what comes to mind?

Critical Thinking 101

Hint. What’s not being mentioned? Profit. While this is not conclusive, the lack of any mention of profit tells me the cash flow and revenue is fine, but there are no big profits as of yet. Cash flow can come from many things, including investors, asset sales, and borrowing money. Most of us probably make in the ballpark of $50,000 to $100,000. Reading that a company raised $60 million, ostensibly to cut in on your job, can be pretty disheartening. Not so once you see that they’re a tiny fraction of the overall picture and that players far bigger than them have not taken your job despite working on the technology for decades.

Moreover, we have a consumer protection crisis on our hands. At least one study in 2020 showed that automatic speech recognition can be 25 to 80 percent accurate depending on who’s speaking. There are many caption advocates out there, such as Meryl Evans, trying to raise awareness on the importance of caption quality. The messaging is very clear: automatic captions are crap (autocraptions), they are often worse than having no captions, and a single wrong word can cause great confusion for someone relying on the captions. Just go see what people on Twitter are saying about #autocraptions. “#NoMoreCraptions. Thank you content creators that do not rely on them!”

Caring about captioning for people who need it makes your brand look good?
I wonder if a brand that looks good makes more money than one that doesn’t…

This isn’t something I’m making up. Anybody in any kind of captioning or transcription business agrees a human is required. Just check out Cielo24’s captioning guide and accuracy table.

Well, this is a little silly. Nobody advertises 60 percent accuracy. It just happens. Ask my boss.

If someone’s talking about an accuracy level of 95 percent or better, they’re talking about human-verified captions. If you, captioner, were not worried about Rev taking away your job with its alleged 50,000 transcribers, then you should not throw in the towel because of Verbit and its alleged 30,000 transcribers. We do not know how much of that is overlap. We do not know how much of that is “this transcriber transcribed for us once and is therefore part of our ‘team.'” We do not know how well transcription skills will fit into the fix-garbage-AI-transcription model. The low pay and mistreatment that comes with “working for” these types of companies is going to drive people away. Think of all the experiences you’ve had to get you to your skill level today. Would you have gotten there with lower compensation, or would you have simply moved on to something easier?

Verbit’s doing exceptionally well in its presentation. It makes claims that would cost quite a bit of time and/or money to disprove, and the results of any such investigation would be questioned by whoever it did not favor. It’s a very old game of making claims faster than they can be disproven and watching the fact checkers give you more press as they attempt to parse what’s true, partially true, and totally false. This doesn’t happen just in the captioning arena, it happens in legal reporting too.

$0/page. Remember what I said about no profit?
It doesn’t matter if they’re never profitable. It only matters that they can keep attracting investor money.

This seems like a terrifying list of capabilities. But, again, this is an old game. Watch how easy it is.

It took me 15 seconds to say six lies, one partial truth, and one actual truth. Many of you have known me for years. What was what? How long will it take you to figure out what was what? How long would it take you to prove to another person what’s true and what’s false? This is, in part, why it is easier for falsehoods to spread than the truth. This is why in court and in science, the person making a claim has to prove their claim. We have no such luxury in the business world. As an example, many years ago in the gaming industry Peter Molyneux got up on stage and demo’d Milo. He said it was real tech. Here was this dynamically interactive virtual boy who’d be able to understand gamers and their actions. We watched it with our own eyes. It was so cool. It was BS. It was very likely scripted. There was no such technology and there is no such technology today, over eleven years later. Do you think Peter, Microsoft, or anybody got in trouble for that? Nope. In fact, years later, he claimed “it was real, honest.”

Here’s the point: Legal reporters and captioners are going to be facing off with these claims for an indeterminate amount of time. These folks are going to be marketing to your clients hard. And I just showed you via the gaming industry that there are zero consequences for lying and that anything that is lied about can just be brushed up with another lie. There will be, more or less, two choices for every single one of you.

  1. Compete / Advocate. Start companies. Ally with deaf advocates.
  2. Watch it happen.

I have basically dedicated Stenonymous to providing facts, figures, and ways that stenographers can come out of the “sky is falling” mindset. But I’m one guy. I’m an official in New York. Science says there’s a good chance what we expect to happen will happen and that’s why I fight like hell to get all of you to expect us to win. That’s also why these companies repeat year after year that they’re going to automate away the jobs even when there’s zero merit or demand for an idea. You now see that companies can operate without making any profit, companies can lie, much bigger companies haven’t muscled in on your job, and that the giant Microsoft presumably looked at Verbit, looked at Nuance, and chose Nuance.

I’m not a neo-luddite. If the technology is that good, let it be that good. Let my job vanish. Fire me tomorrow. But facts are facts, and the fact is that tech sellers take the excellent work of brilliant programmers and say the tech is ready for prime time way before it is. They never bother to mention the drawbacks. Self-driving cars and trucks are on the way, don’t worry about whether it kills someone. Robots can do all these wonderful things, forget that injuries are up where they’re in heaviest use. Solar Roadways were going to solve the world’s energy problems but couldn’t generate any energy or be driven on. In our field, lives and important stakeholders are in danger. What happens when there’s a hurricane on the way and the AI captioning tells deaf people to drive towards danger?

Again, two choices, and I’m hoping stenographic captioners don’t watch it happen.