How We Discuss Errors and Automatic Speech Recognition

As a stenographic court reporter, I have been amazed by the strides in technology. Around 2016, I, like many of you, saw the first claims that speech recognition was as good as human ears. Automation seemed inevitable, and a few of my most beloved colleagues believed there was not a future for our amazing students. In 2019, the Testifying While Black study was published in the Language Journal, and while the study and its pilot studies showed that court reporters were twice as good at understanding the AAVE dialect as your average person, even though we have no training whatsoever in that dialect, the news media focused on the fact that we certify at 95 percent and yet only had 80 percent accuracy in the study. Some of the people involved with that study, namely Taylor Jones and Christopher Hall, introduced Culture Point, just one provider that could help make that 80 percent so much higher. In 2020, a study from Stanford showed that automatic speech recognition had a word error rate of 19 percent for “white” speakers, 35 percent for “black” speakers, and “worse” for speakers with a high dialect density. How much worse?

The .75 on the left means 75 percent. DDM is the dialect density. Even with fairly low dialect density, we’re looking at over 50 percent word error rate.

75 percent word error rate in a study done three or four years after the first claim that automatic speech recognition had 94 percent accuracy. But in all my research and all that has been written on this topic, I have not seen the following point addressed:

What Is An Error?

NCRA, many years ago, set out guidelines for what constituted an error. Word error guidelines take up about a page. Grammatical error guidelines take up about a page. What this means is that when you sit down for a steno test, you’re not being graded on your word error rate (WER), you’re being graded on your total errors. We have decades of failed certification tests where a period or comma meant a reporter wasn’t ready for the working world yet. Even where speech recognition is amazing on that WER, I’ve almost never seen appreciable grammar, punctuation, Q&A, or anything that we do to make the transcript readable. It’s so bad that advocates for the deaf, like Meryl Evans, refer to automatic speech recognition as “autocraptions.”

Unless the bench, bar, and captioning consumers want word soup to be the standard, the difference in how we describe errors needs to be injected into the discussion. Unless we want to go from a world where one reporter, perhaps paired with a scopist, completes the transcript and is accountable for it, to a world where up to eight transcribers are needed to transcribe a daily, we need to continue to push this as a consumer protection issue. Even where regulations are lacking, this is a serious and systemic issue that could shred access to justice. We have to hit every medium possible and let people know the record — in fact, every record in this country — could be in danger. The data coming out is clear. Anyone selling recording and/or automatic transcription says 90-something percent accuracy. Any time it’s actually studied? Maybe 80 percent accuracy, maybe 25; maybe they hire a real expert transcriber, or maybe they outsource all their transcription to Kenya or Manila. Perception matters; court administrators are making industry-changing decisions based on the lies or ignorance of private sector vendors.

The point is recording equipment sellers are taking a field which has been refined by stenographic court reporters to be a fairly painless process where there are clear guidelines for what happens when something goes wrong, adding lots of extra parts to it, and calling it new. We’ve been comparing our 95 percent total accuracy to their “94 percent” word error rate. In 2016, perhaps there were questions that needed answering. This is April 2021, there’s no contest, and proponents of digital recording and automatic transcription have a moral obligation to look at the facts as they are today and not what they’d like them to be.

If you are a reporter that wants more information or ideas on how to talk about these issues with clients, check out the NCRA Strong Resource Library, and Protect Your Record Project. Even reporters that have never engaged in any kind of public speaking can pick up valuable tips on how to educate the public about why stenographic reporting is necessary. Lawyers, litigants, and everyday people do not have time to go seeking this information; together, we can bring it to them.

Collective Power of Stenographers

One piece of feedback I get back from time to time is “we can’t stand up to XYZ Corporation. They make 100 million in revenue!” I deeply empathize with this reaction because I’ve felt that before. Back in freelance, that feeling was constant. How could I negotiate with a company that was only offering $3.25? They were a big company with lots of work. I was basically a kid just out of college with my extremely shiny AOS. I didn’t even have a squid hat yet.

With this thing on, I became unstoppable.

But about 3 years ago I started to teach myself very basic computer programming. I began to learn a little bit more about numbers and math. I had always hated math, and the whole experience completely changed that perception. I started to like math. One the first programs I ever wrote was a simple counter program similar to this one:

This program loops for as long as steno is awesome, and steno never stops being awesome.

In this code, you start with the number 0 and it adds one forever until the computer malfunctions or the program is shut down. What you see happen very quickly is that when you’re adding one several times a second, one quickly becomes 10, 100, 1,000, 1,000,000.

What the hell does that have to do with stenographers? We are the ones that add up in this program called life. For example, let’s say we have XYZ Corporation and it makes $100 million a year in revenue. Now let’s say there are 23,000 reporters, like vTestify said almost three years ago, and let’s assume that reporters ONLY make a median salary of about $60,000 a year. Those reporters make $1.3 billion in revenue annually. You take two percent of that a year and throw it in advertising pot, and you’re talking a $26 million annual advertising campaign.

5 percent? I said 2 percent. Someone should fix this immediately.

So now to bring this out of theory and into reality, you can see it happening in real life. There’s no group of people that’s going to have a 100 percent contribution rate. But when you look at the numbers, you start to see that overall we put far better funding into our organizations and activities than alternative methods or spinoffs. Take, for example, AAERT, which pulled in about $200,000 in 2018 revenue. For those that don’t know AAERT, they’re primarily engaged with supporting the record-and-transcribe method of capturing the spoken word. As I’ve covered in past blog posts and industry media, it’s an inefficient and undesirable method (page 5), and most digital reporters would do a lot better if they picked up steno.

Published by ProPublica

Then we can look towards the National Verbatim Reporters Association, which seems to focus more on voice writing, but definitely includes and accepts stenographic reporters. We see the 2017 revenue here come in at almost $250,000. Not bad at all.

As far as I’m concerned, every dollar is deserved. I’ve never heard a bad word from an NVRA member.

But then we look to our National Court Reporters Association, which is primarily engaged in promoting stenography and increasing the skill of stenographic court reporters. This is where we see the collective power of reporters start to add up in a big way. In 2018, the NCRA saw more than $5.7 million in revenue. The NCRF brought in an additional $368,000. That’s over $6 million down on steno that year.

I think I can see my membership dues somewhere in there.
When I pay off my massive personal debt, I’m going to become an NCRF Angel / Squid.

What conclusions can be drawn here? As much as the anti-steno crowd wants to say the profession’s dead, dying, or defunct, there’s just no evidence to support that. Here you get to see some fraction of every field contributing to nonprofits dedicated to education, training, and educating the public. We know from publicly-available information that our membership dues are not 30x more than these other organizations, so we know that there are a lot more of us, and we know that there are a lot more of us participating in continuing education and sharpening our skills. We’re the preferred method. We’re the superior method. We’re training harder every day to meet the needs of consumers. There are only a few ways this goes badly for stenography.

  1. We lack the organization or confidence to counter false messaging.
  2. We lose trust in our collective power and institutions, stop supporting them, and stop promoting ourselves. Kind of like the Pygmalion effect.
  3. We spend time tearing each other down instead of boosting each other’s stuff.

See the common theme? There’s really nothing external that’s going to hurt this field. It all comes down to our ability to adapt, organize, and play nice with each other. In the past, I equated it with medieval warfare and fiction. The easiest way to win any adversarial situation is to get the other side to give up and go home. It’s an old idea straight out of Sun Tzu’s Art of War. Applied to business, if you can convince people not to compete against you, you win by default. This might be in the form of a buyout. This might be in the form of convincing people that stenography is not a viable field so that there are not enough stenographers to meet demand. This might be in the form of would-be entrepreneurs believing they cannot compete and never starting a business. This might be in the form of convincing consumers that stenographic reporters are not available. This might be in the form of casting doubt on stenographic associations. This might be in the form of buying a steno training program and ostensibly scrubbing it out of existence. These are all actions to avoid competition, because as the numbers just showed you, we only lose if we do not compete. If you do nothing else for Court Reporting & Captioning Week 2021, please take the time to promote at least one positive thing about steno. If a guy in a squid hat could get you to think differently about just one topic today, what kind of potential do you have to make a difference in this world?

I’ll launch us off with an older quote from Marc Russo. “If you are a self-motivated person with a burning desire to improve your skills, this is the field.” This is our field. This is our skill. All we have left to do is stand up to the people that take advantage of our stellar customer service mentality and the public perception that we’re potted plants.

Can a potted plant do this?

PS. That $3.25 I was having trouble negotiating up from? Some of my friends were making $4.00+ with less experience than me. The limitation was me and the way that I was thinking about it. We have all had to deal with hurdles that seemed insurmountable. Max Curry talked a little bit about it in his NCRA Stenopalooza presentation “Fear…Let It Go!” when he talked about his father and introversion. It was an amazing presentation. But here’s my takeaway for those that missed it last year. If you’re having a problem, try looking at it another way.