Tag: accuracy

Scopist: Degradation of Reporter Skill is Helping AI Infiltration

As told to me by a valued Stenonymous reader:

“Hi, Christopher. [REDACTED]. I was a decades-long practicing [REDACTED] court reporter, and [REDACTED]. I edited my own work, so I was well aware of my good days and bad days. I started scoping/editing full time for reporters in around [REDACTED]. I am still scoping/ editing for reporters from all over the US — [REDACTED]. I have to say I am extremely disappointed and dismayed at the lack of qualifications I have experienced from the majority of these reporters. I find they are dropping an extraordinary amount of text, at times 15-20% and I am typing in the missing text for the reporters. This is not a one-off or occasional occurrence. I only bring this up to you because of the threat this industry is facing re AI, digital recorders, big box corporations, etc. I don’t know if this is a subject you (gingerly) would consider addressing in your Stenonymous articles – which I thoroughly enjoy, by the way. You are a driving force in the fight to save this industry. I do not expect a response from you. I am asking that you please keep this writing anonymous and confidential.”

I have to say that I have been a voice for quality in our field since before Stenonymous’s time. We used to have this weird mannerism in the profession where we had to pretend we were all perfect and never made mistakes. That may seem like a foreign concept to students of this day and age, whose mentors seem more willing to say everyone makes mistakes. It was never true that we didn’t. Being in New York City, I saw firsthand the degradation of skill in the stenographic reporter. By the time I came into the field, audio had become ubiquitous and many were reliant on it to the point where they couldn’t pass the civil service test that the state eliminated a few months ago. And worse, people pretended, and encouraged others to pretend, that the audio was not in use.

As best I can tell, this was under pressure from agencies, who were under pressure from high-strung clients. The Introverted Lawyer, Heidi Kristin Brown, once spoke to a crowd of court reporters about the nastiness she’d experienced in the legal field. I’ve met that nastiness too. So I understand the wilting nature of the court reporter and its agencies. Without each other to hold us up, it is easy for outside influences to cause us to feel less than — and it is easy for the people holding the wallet to screw any individual “person” out of a day’s pay. That’s understandably scary.

But I never ran from the digital problem or the misinformation campaign spread by the corps. I can’t run from ours either. This is not the first scopist to mention this to me. Probably won’t be the last.

I’ve been on jobs where using the audio as a crutch would’ve made the transcripts unusable.

I’ve been on jobs where the audio was a must because I was physically prevented from interrupting (think public meetings).

I’ve been on jobs where I used audio as a crutch.

I’ve been on jobs where I’ve been ordered not to use audio even where others factually do.

I hired a scopist once who called my writing “labyrinthian.” Haven’t hired one since.

I understand, more than anyone (hyperbole), the importance of the use of the tool, where appropriate.

But if you’re at the point where you’re dropping a fifth of the testimony in your stenographic note taking, it’s time to identify and correct that issue. And might I suggest that the issue is likely that you never learned to “control a room” because you used audio as a crutch. And that’s okay. It’s what you do with the future that counts. Making mistakes in our line of work is a much smaller problem than taking no action to correct them.

Those “button pushers” we denigrate? Some of them are better. Think about the ones that are honest about who they are and what they do. What do we have on them? Isn’t that what all this is about? Speaking out against corporate dishonesty and corner cutting because the transcript is important? I know some of them have told me they were trained to lie, obfuscate, or even provide fake names (cough, cough, Naegeli rumor, cough, worst court reporting company in the country?) But for the ones that are honest and feed us info, aren’t they “on our side?”

Maybe I’m kidding myself.

Maybe everybody’s afraid of the truth?

After all, you don’t think people in positions of power have heard my allegations? I sent them directly to a judge quoted in an article about us once. You think he wrote back?

People so afraid of looking foolish they’d rather promote a comfortable lie or ignore a terrible wrong than look at a situation with the analytical nuance that they do any day-to-day issue or case. It’s not their job. Why care? And the people whose job it is? They can make up excuses ad infinitum why they can’t care right now without consequence or accountability. This is why my publishing strategy is to get louder and louder until they all look like assholes for ignoring it. I don’t see another way forward. Doesn’t anyone else get it? Being nice to people that deny the truth got us where we are.

Reader, in a world of evil apes and ostriches, be a lion.

Or a sheepdog.

P.S.

Anyone actually good at fundraising/P.R./sales? Maybe we can make a deal and split donations. It’s a gamble. I can go months with minimal funding or make $1,000 in a day. It’s not like the money isn’t there to turn this into a small media operation. It just needs to be redirected a little bit. My real-life friends are too busy to help, so I leave it up to my readership.

Having our marriage ceremony soon. It’ll probably get pretty quiet on here for a while. Thanks for putting up with my double-post days these last few weeks. Have a wonderful weekend.

Ever eager to be a platform for your…

Words & Voices, Stenonymous.com

Addendum:

Toward the end of March 29th, Erin Blair made a comment that I asked to share with my audience. They are a respected member of the field, so I’m happy to share their views with my audience.

Erin Blair’s comments on Stenonymous’s blog post “Scopist: Degradation of Reporter Skill is Helping AI.”

We also had a brief discussion where I tried to explain I don’t mean to victim blame and believe the issue is nuanced.

Language Study and Service Revisited

Let’s just get to the point. There is a study to be published in the linguistic journal Language in June 2019. Stenonymous covered this immediately. Succinctly the study showed that court reporters in the Philadelphia area were pretty inaccurate when dealing with the dialect of African American English. We had some suspicions about potential inaccuracy in the way the news was reporting it, and kept an eye out for information as it developed.

In early March, we came across new articles which identified one of the hard-working linguists on the study, Taylor Jones. Upon review of Mr. Jones’s blog — soon to be Dr. Jones as far as we’re aware — we reached out and he responded to everything we had to ask.

Though we haven’t yet gotten to see the study, between correspondence with Jones, review of his blog, and review of media coverage on the topic, we have some conclusions to present:

  • The court reporters were reporters working in court.
  • It’s true that stenographic court reporters were used.
  • The trials were not testing the reporters’ real-time accuracy, and participants were given as much time as they wanted to transcribe.
  • The accuracy of sentences was only 59.5% correct. When measuring word-for-word accuracy the accuracy was as much as 82.9%. Obviously, our stenographer training measures word-to-word accuracy.
  • Small “errors” were not counted as errors, such as if a speaker said “when you tryna go to the store?” Trying to and tryin’ to would both be counted as correct. An error would be “when he tries to go.” So the errors, as best I can tell, would fall in line with what NCRA says constitutes an error.
  • Misunderstandings come from a number of different sources, including common phonetic misunderstandings and dialect-motivated misunderstandings as discussed in William Labov’s Principles of Linguistic Change trilogy. While Jones himself said bias cannot be ruled out, there are a number of syntactical and accent-related issues that may honestly be a challenge for court reporters and the average judge, juror, or listener.
  • There were over 2,200 observations done in this study. 83 statements multiplied by 27 court reporters.

Now for some interesting highlights from my exchange with Jones:

  1. African American English is not wrong. It is not slang. It has grammar and structure. It’s not slang, Ebonics, or street talk.
  2. The people that conducted the study are not accusing court reporters of doing anything wrong. In fact, in my conversation with Jones, he was supportive of a human stenographer over an AI or automatic transcription because we still carry a far greater accuracy than those alternatives.

So here is where we are: We’ve got a piece of evidence from the linguistic community that there is an area we can improve on. I had briefly been in touch with a Culture Point representative who said they can work with organizations around the country on their transcription suite package, and that the budget for the workshop varies dependent on modality and class size.

We should all do our best to incorporate these ideas into our work and training. If you are a state or national association, don’t shy away from the opportunity to dive in and develop training surrounding different dialects, or even fund studies to seek out these deficiencies. If you are a working reporter, don’t be afraid to ask for a repetition. You are the guardian of an accurate and true record, and our work collectively can impact people’s lives and fortunes.

Short last note, I apologize to my readers and to Mr. Jones. I had promised my readers I’d get this article out and the email exchange out much sooner. I feel this is important and want to be a part of spreading the message that we can always do better. Though the initial response by Mr. Jones was March 8, I was unable to get this draft out until April 2. For that, I am sorry.

May 23, 2019 update: This came up in the news again and another person brought to my attention this draft of the study made available before its publication in the Language journal. It was noted by that person that the reporters were asked to paraphrase what was said, and that we do not interpret. My understanding and memory from my email with Jones is that they were asked to transcribe and interpret, and that at least one participant transcribed incorrectly but interpreted perfectly.

June 6, 2019 update:

Philadelphia judges came together to discuss language access after the study. As of this article, it seems the solution would be more training for court personnel than having interpreters for different English dialects.

September 13, 2019 update:

Another article popped up, ostensibly on this same study. With great respect to those article writers, I believe the headline that white court reporters don’t get black testimony is incorrect. I also believe that the contention that this is slang or Ebonics is incorrect. When I wrote Jones he was very clear that AAE is not slang. It’s a dialect. It has rules. I do hope that people really read the work for what it is and not what they want it to be. People mishear things. Judges and juries mishear things. This study brings to light that even we, the people who care most about every word said, can mishear things, and that makes it very, very important to be situationally aware and ask for clarification when it is appropriate, like many of us do every day.

January 28, 2020 update:

It should be noted that Mr. Jones, presumably now Dr. Jones, is listed as a co-founder of Culture Point on LinkedIn.

Addendum:

After some time I had an interview with VICE about this study because I was identified as being a stenographic reporter with a lot of knowledge on it. I will say while, in my mind, it showed us we must do better, ultimately it confirmed that we are people’s best chance at being understood in the courtroom. The pilot study 1 showed regular people were about 40 percent accurate. The pilot study 2 showed lawyers were 60 percent accurate. We were about 80 percent accurate. Clearly, we all want 100 percent, but when you read that we’re twice as good as your average person at taking down this dialect, it changes the spin. Later on, a Stanford study showed that automatic speech recognition had 20 percent error rate in “white speech,” 40 percent error rate in “black speech,” and worse with African American English dialect. When I graded the AAE example on their site, I saw that if it had been a steno test, it would be 20/100! It’s our skill and dedication that keeps us top quality in making the record and broadcast captioning.