How We Discuss Errors and Automatic Speech Recognition

As a stenographic court reporter, I have been amazed by the strides in technology. Around 2016, I, like many of you, saw the first claims that speech recognition was as good as human ears. Automation seemed inevitable, and a few of my most beloved colleagues believed there was not a future for our amazing students. In 2019, the Testifying While Black study was published in the Language Journal, and while the study and its pilot studies showed that court reporters were twice as good at understanding the AAVE dialect as your average person, even though we have no training whatsoever in that dialect, the news media focused on the fact that we certify at 95 percent and yet only had 80 percent accuracy in the study. Some of the people involved with that study, namely Taylor Jones and Christopher Hall, introduced Culture Point, just one provider that could help make that 80 percent so much higher. In 2020, a study from Stanford showed that automatic speech recognition had a word error rate of 19 percent for “white” speakers, 35 percent for “black” speakers, and “worse” for speakers with a high dialect density. How much worse?

The .75 on the left means 75 percent. DDM is the dialect density. Even with fairly low dialect density, we’re looking at over 50 percent word error rate.

75 percent word error rate in a study done three or four years after the first claim that automatic speech recognition had 94 percent accuracy. But in all my research and all that has been written on this topic, I have not seen the following point addressed:

What Is An Error?

NCRA, many years ago, set out guidelines for what constituted an error. Word error guidelines take up about a page. Grammatical error guidelines take up about a page. What this means is that when you sit down for a steno test, you’re not being graded on your word error rate (WER), you’re being graded on your total errors. We have decades of failed certification tests where a period or comma meant a reporter wasn’t ready for the working world yet. Even where speech recognition is amazing on that WER, I’ve almost never seen appreciable grammar, punctuation, Q&A, or anything that we do to make the transcript readable. It’s so bad that advocates for the deaf, like Meryl Evans, refer to automatic speech recognition as “autocraptions.”

Unless the bench, bar, and captioning consumers want word soup to be the standard, the difference in how we describe errors needs to be injected into the discussion. Unless we want to go from a world where one reporter, perhaps paired with a scopist, completes the transcript and is accountable for it, to a world where up to eight transcribers are needed to transcribe a daily, we need to continue to push this as a consumer protection issue. Even where regulations are lacking, this is a serious and systemic issue that could shred access to justice. We have to hit every medium possible and let people know the record — in fact, every record in this country — could be in danger. The data coming out is clear. Anyone selling recording and/or automatic transcription says 90-something percent accuracy. Any time it’s actually studied? Maybe 80 percent accuracy, maybe 25; maybe they hire a real expert transcriber, or maybe they outsource all their transcription to Kenya or Manila. Perception matters; court administrators are making industry-changing decisions based on the lies or ignorance of private sector vendors.

The point is recording equipment sellers are taking a field which has been refined by stenographic court reporters to be a fairly painless process where there are clear guidelines for what happens when something goes wrong, adding lots of extra parts to it, and calling it new. We’ve been comparing our 95 percent total accuracy to their “94 percent” word error rate. In 2016, perhaps there were questions that needed answering. This is April 2021, there’s no contest, and proponents of digital recording and automatic transcription have a moral obligation to look at the facts as they are today and not what they’d like them to be.

If you are a reporter that wants more information or ideas on how to talk about these issues with clients, check out the NCRA Strong Resource Library, and Protect Your Record Project. Even reporters that have never engaged in any kind of public speaking can pick up valuable tips on how to educate the public about why stenographic reporting is necessary. Lawyers, litigants, and everyday people do not have time to go seeking this information; together, we can bring it to them.

A Night In Brooklyn, PYRP 78

This past weekend 78 pop-up events across the country launched for stenographers, almost all of which were at the same time on January 18, 2020. For Sabbath observers and those who couldn’t make the January 18 pop up, Devora Hackner organized and hosted one on the night of January 19 in Brooklyn. It was a fantastic night and a good indicator of what just a little solidarity can achieve. Protect Your Record Project, started in California by Kimberly D’Urso and Kelly Bryce Shainline, has swelled to a national movement where stenographers are saying loud and clear to the consumer that we are the fastest and most efficient method of capturing the spoken word.

The Brooklyn event was a real showing of stenographic society in NY. Every attendee’s presence was important and brought something special to the table. Nancy Silberger, immediate past president of NYSCRA and host of New York’s Saturday PYRP event was there. Howie Gresh and Reid Goldsmith, both longtime working reporters and educators were there. Ellen Sandles, a reporter who has done extensive research into the Federation of Shorthand Reporters was present. Representatives and owners of Little, Lex, and Diamond were also present. NYSCRA’s President, Joshua Edwards, also made an appearance some time after the event’s start. There were over thirty decades of collective reporting experience in the room and nearly two dozen attendees.

Everyone came together to talk about how to advocate for stenography. Ms. Silberger mentioned her ability to host some meetings. Ms. Sandles talked about having potential press contacts. Jane Sackheim of Diamond mentioned that space could be offered by Diamond to teach A to Z courses, something NCRA and Project Steno advocates should definitely ask about. Mr. Gresh reminded everyone about NYSCRA’s involvement in offering free test prep classes. Rivka Teich, a masterful reporter working at the Eastern District Brooklyn Courthouse talked about doing a career night and introducing more people to what we do and different jobs in the field. Mr. Edwards reminded everyone about NYSCRA’s mentoring program and urged people to sign up as mentors or to be mentored. He also brought up that attendees were still being accepted for the NYSCRA Court Reporting & Captioning Week Real-Time events.

Many, many ideas were covered. From high school outreach and following NYS legislation to PYRP’s available resources and files, all the way to potential legislative ideas, like copy protection for reporters’ work. The importance of starting discussions on stenography was noted. We talked about the potential of changing covers, parentheticals, and cert pages to say stenographer instead of court reporter. The importance of communicating with the videographer and injecting oneself into the record when necessary to make a better record was talked about.

There is one theme recurrent in all of this. The power of the individual is undeniable. That’s everyone who was present. That’s you. Reporters are getting together and great things are happening. Maybe there’s a skill you have, or some kind of connection you’ve made that can help educate a consumer or empower another reporter. You don’t have to wait for a giant winged creature to invite you, you just have to be brave enough to jump on the wagon.