How We Discuss Errors and Automatic Speech Recognition

As a stenographic court reporter, I have been amazed by the strides in technology. Around 2016, I, like many of you, saw the first claims that speech recognition was as good as human ears. Automation seemed inevitable, and a few of my most beloved colleagues believed there was not a future for our amazing students. In 2019, the Testifying While Black study was published in the Language Journal, and while the study and its pilot studies showed that court reporters were twice as good at understanding the AAVE dialect as your average person, even though we have no training whatsoever in that dialect, the news media focused on the fact that we certify at 95 percent and yet only had 80 percent accuracy in the study. Some of the people involved with that study, namely Taylor Jones and Christopher Hall, introduced Culture Point, just one provider that could help make that 80 percent so much higher. In 2020, a study from Stanford showed that automatic speech recognition had a word error rate of 19 percent for “white” speakers, 35 percent for “black” speakers, and “worse” for speakers with a high dialect density. How much worse?

The .75 on the left means 75 percent. DDM is the dialect density. Even with fairly low dialect density, we’re looking at over 50 percent word error rate.

75 percent word error rate in a study done three or four years after the first claim that automatic speech recognition had 94 percent accuracy. But in all my research and all that has been written on this topic, I have not seen the following point addressed:

What Is An Error?

NCRA, many years ago, set out guidelines for what constituted an error. Word error guidelines take up about a page. Grammatical error guidelines take up about a page. What this means is that when you sit down for a steno test, you’re not being graded on your word error rate (WER), you’re being graded on your total errors. We have decades of failed certification tests where a period or comma meant a reporter wasn’t ready for the working world yet. Even where speech recognition is amazing on that WER, I’ve almost never seen appreciable grammar, punctuation, Q&A, or anything that we do to make the transcript readable. It’s so bad that advocates for the deaf, like Meryl Evans, refer to automatic speech recognition as “autocraptions.”

Unless the bench, bar, and captioning consumers want word soup to be the standard, the difference in how we describe errors needs to be injected into the discussion. Unless we want to go from a world where one reporter, perhaps paired with a scopist, completes the transcript and is accountable for it, to a world where up to eight transcribers are needed to transcribe a daily, we need to continue to push this as a consumer protection issue. Even where regulations are lacking, this is a serious and systemic issue that could shred access to justice. We have to hit every medium possible and let people know the record — in fact, every record in this country — could be in danger. The data coming out is clear. Anyone selling recording and/or automatic transcription says 90-something percent accuracy. Any time it’s actually studied? Maybe 80 percent accuracy, maybe 25; maybe they hire a real expert transcriber, or maybe they outsource all their transcription to Kenya or Manila. Perception matters; court administrators are making industry-changing decisions based on the lies or ignorance of private sector vendors.

The point is recording equipment sellers are taking a field which has been refined by stenographic court reporters to be a fairly painless process where there are clear guidelines for what happens when something goes wrong, adding lots of extra parts to it, and calling it new. We’ve been comparing our 95 percent total accuracy to their “94 percent” word error rate. In 2016, perhaps there were questions that needed answering. This is April 2021, there’s no contest, and proponents of digital recording and automatic transcription have a moral obligation to look at the facts as they are today and not what they’d like them to be.

If you are a reporter that wants more information or ideas on how to talk about these issues with clients, check out the NCRA Strong Resource Library, and Protect Your Record Project. Even reporters that have never engaged in any kind of public speaking can pick up valuable tips on how to educate the public about why stenographic reporting is necessary. Lawyers, litigants, and everyday people do not have time to go seeking this information; together, we can bring it to them.

Aggressive Marketing — Growth or Flailing?

During our Court Reporting & Captioning Week 2021 there were a couple of press releases and some press releases dressed up as journalism all about digital recording, automatic speech recognition, and its accuracy and viability. There’s actually a lesson to be learned from businesses that continually promise without any regard for reality, so that’s what I’ll focus on today. I’ll start with this statement. We have a big, vibrant field of students and professionals where everyone that is actually involved in it, from the smallest one-woman reporting armies to the corporate giants, says technology will not replace the stenographic court reporter. Then we have the tech players who continuously talk about how their tech is 99 percent accurate, but can’t be bothered to sell it to us, and whose brilliant plan is to record and transcribe the testimony, something stenographers figured out how to do decades ago.

Steno students are out there getting a million views and worldwide audiences…
And Chris Day? He’s posting memes on the internet.

You know the formula. First we’ll compare this to an exaggerated event outside the industry, and then we’ll tie it right into our world. So let’s breeze briefly over Fyre Festival. To put it in very simple terms, Fyre Festival was an event where the CEO overpromised, underdelivered, and played “hide the ball” until the bitter end. Customers were lied to. Investors were lied to. Staff and construction members were lied to. It was a corporate fiasco propped up by disinformation, investor money, and cash flow games that ended with the CEO in prison and a whole lot of people owed a whole lot of money that they will, in all likelihood, never get paid. It was the story of a relative newcomer to the industry of music festivals saying they’d do it bigger and better. Sound familiar?

As for relative newcomers in the legal transcription or court reporting business, take your pick. Even ones that have been incorporated for a couple of decades really aren’t that impressive when you start holding up the magnifying glass. Take, for example, VIQ Solutions and its many subsidiaries:

I promise to explain if you promise to keep reading.

VIQ apparently trades OTC so it gives us a rare glimpse of financial information that we don’t get with a lot of private companies. Right off the bat, we can see some interesting stuff. $8 million in revenue with a negative net income and a positive cash flow. Positive cash flow means the money they have on hand is going up. Negative income means the company is losing money. How does a company lose money but continue to have cash on hand grow? Creditors and investors. When you see money coming in while the company is taking losses, it generally means that the company is borrowing the money or getting more cash from investors/shareholders. A company can continue on this way for as long as money keeps coming in. Companies can also use tricks similar to price dumping, and charge one client or project an excessive amount in order to fund losses on other projects. The amazing thing is that most companies won’t light up the same way Fyre did, they’ll just declare bankruptcy and move on. There’s not going to be a big “gotcha” parade or reckoning where anyone admits that stenographic court reporting is by far the superior business model.

This is juxtaposed against a situation where, for the individual stenographic reporter, you’re kind of stuck making whatever you make. If things go badly, bankruptcy is an option, but there’s never really an option to borrow money or receive investor money for decades while you figure it out. Seeing all these ostensible giants enter the field can be a bit intimidating or confusing. But any time you see these staggering tech reveals wrapped up in a paid-for press release, I urge you to remember Fyre, remember VIQ, and remember that no matter what that revenue or cash flow looks like, you may not have access to the information that would tell you how the company is really doing.

This also leads to a very bright future for steno entrepreneurs. As we learn the game, we can pass it along to each other. When Stenovate landed its first big investor, I talked about that. Court reporting and its attached services, in the way we know them and love them, are an extremely stable, winning investment. Think about it. Many of us, when we begin down this road, spend up to $2,000 on a student machine and up to $9,000 on a professional machine and software. That $11,000 sinkhole, coupled with student loan debt, grows into stable, positive income. So what’s stopping any stenographic court reporting firm from getting out there and educating investors on our field? The time and drive to do it. Maybe for some people, they just haven’t had that idea yet. But that’s where we’re headed. I have little doubt that if we compete, we will win. But we have to get people in that mindset. So if you know somebody with that entrepreneurial spirit, maybe pass them this post and get them thinking about whether they’d like to seek investors to grow their firm and reach. Business 101 is that a dollar today is more valuable than a dollar tomorrow. That means our field can be extremely attractive to value investors and be a safe haven from the gambling money being supplied to “tech’s” habitual promisors.

Know a great reporting or captioning firm that needs a spotlight? Feel free to write me or comment about them below. I’ll start us off. Steno Captions, LLC launched off recently without doing the investor dance. That’s the kind of promise this field has. I wish them a lot of luck and success in managing clients and training writers.