On Reddit, I spotted a thread that talked about the H3 podcast and how upon requesting captions other members of the audience insisted auto captions were adequate. The podcast itself apparently stated it was too difficult to do the captioning for hours of content.
My audience is mostly stenographers that support good captions on all content. I’ll be preaching to the choir here. The only thing between us and that dream are the greedy corps and individuals that have no problem being rewarded for their success — they’ll take every dime you’ll give — but they have a lot of problems making simple provisions for the disabled.
Redditor discusses the failure of the H3 podcast to provide adequate captions for the deaf.
It’s my position that we need to be firm about this and keep pushing for the expansion of stenographic writers in captioning. No method is perfect, but auto captions and crowdsourcing are failing some of the most vulnerable people in our society. Do we accept that or do we speak out about it?
We need a significant increase in our numbers to be able to cover all the content out there. Hopefully investors see things like this and realize there’s a genuine need for a lot more stenographers. Pump students up with stenographic and business education and set them loose on the market. It’d give this field the tools it needs to fight for the disabled. I believe so strongly in this that I myself might look into school funding.
If anybody from H3 Podcast stumbles across this, your rep is going to take a hit if you keep letting this situation go unresolved. Don’t let it get to that point and just spend the piddly $100 or $200 it would take you to caption your content per hour. If stenographer prices are really a problem for whatever level of revenue you have, you could at the very least turn to transcribers, who are often taken advantage of and underpaid.
How can we spread this message beyond our social circles?
As a stenographic court reporter, I have been amazed by the strides in technology. Around 2016, I, like many of you, saw the first claims that speech recognition was as good as human ears. Automation seemed inevitable, and a few of my most beloved colleagues believed there was not a future for our amazing students. In 2019, the Testifying While Black study was published in the Language Journal, and while the study and its pilot studies showed that court reporters were twice as good at understanding the AAVE dialect as your average person, even though we have no training whatsoever in that dialect, the news media focused on the fact that we certify at 95 percent and yet only had 80 percent accuracy in the study. Some of the people involved with that study, namely Taylor Jones and Christopher Hall, introduced Culture Point, just one provider that could help make that 80 percent so much higher. In 2020, a study from Stanford showed that automatic speech recognition had a word error rate of 19 percent for “white” speakers, 35 percent for “black” speakers, and “worse” for speakers with a high dialect density. How much worse?
The .75 on the left means 75 percent. DDM is the dialect density. Even with fairly low dialect density, we’re looking at over 50 percent word error rate.
75 percent word error rate in a study done three or four years after the first claim that automatic speech recognition had 94 percent accuracy. But in all my research and all that has been written on this topic, I have not seen the following point addressed:
What Is An Error?
NCRA, many years ago, set out guidelines for what constituted an error. Word error guidelines take up about a page. Grammatical error guidelines take up about a page. What this means is that when you sit down for a steno test, you’re not being graded on your word error rate (WER), you’re being graded on your total errors. We have decades of failed certification tests where a period or comma meant a reporter wasn’t ready for the working world yet. Even where speech recognition is amazing on that WER, I’ve almost never seen appreciable grammar, punctuation, Q&A, or anything that we do to make the transcript readable. It’s so bad that advocates for the deaf, like Meryl Evans, refer to automatic speech recognition as “autocraptions.”
Unless the bench, bar, and captioning consumers want word soup to be the standard, the difference in how we describe errors needs to be injected into the discussion. Unless we want to go from a world where one reporter, perhaps paired with a scopist, completes the transcript and is accountable for it, to a world where up to eight transcribers are needed to transcribe a daily, we need to continue to push this as a consumer protection issue. Even where regulations are lacking, this is a serious and systemic issue that could shred access to justice. We have to hit every medium possible and let people know the record — in fact, every record in this country — could be in danger. The data coming out is clear. Anyone selling recording and/or automatic transcription says 90-something percent accuracy. Any time it’s actually studied? Maybe 80 percent accuracy, maybe 25; maybe they hire a real expert transcriber, or maybe they outsource all their transcription to Kenya or Manila. Perception matters; court administrators are making industry-changing decisions based on the lies or ignorance of private sector vendors.
The point is recording equipment sellers are taking a field which has been refined by stenographic court reporters to be a fairly painless process where there are clear guidelines for what happens when something goes wrong, adding lots of extra parts to it, and calling it new. We’ve been comparing our 95 percent total accuracy to their “94 percent” word error rate. In 2016, perhaps there were questions that needed answering. This is April 2021, there’s no contest, and proponents of digital recording and automatic transcription have a moral obligation to look at the facts as they are today and not what they’d like them to be.
