Blog

---

• Captioning

Measuring Captioning Accuracy: Why WER and NER Analyses Differ

3Play Media

•

February 28, 2024

Captioning Best Practices for Media & Entertainment [Free eBook]

When it comes to measuring captioning accuracy, there’s no shortage of errors that need to be considered: punctuation, grammar, speaker identification, capitalization, and word errors, to name a few. But what does it mean when a captioning vendor says their captions are 99% accurate? 

It turns out that “99% accuracy” can mean very different things depending on the model a vendor uses to measure said accuracy. Different vendors use different measurement models, which can contribute to confusion when percentages are marketed to describe the accuracy of closed captions. In this blog, we’ll discuss the NER model and how it differs from two commonly used measurements, Word Error Rate (WER) and Formatted Error Rate (FER).

The NER Model

The NER model, which originated in Europe and is often used in Canada, differs from the accuracy measurement rates commonly used in the United States. In the U.S., all errors—including spelling, punctuation, grammar, speaker identifications, word substitutions, omissions, and more—are considered to obtain a percentage that measures the average accuracy of the closed captions on a piece of media. 

In contrast, NER scoring emphasizes meaning and how accurately ideas are captured in captions, making it an extremely subjective and legally risky measurement. For instance, the FCC closed captioning guidelines state, “In order to be accurate, captions must match the spoken words in the dialogue, in their original language (English or Spanish), to the fullest extent possible and include full lyrics when provided on the audio track.” More specifically, the guidelines require captions to include all words spoken in the order spoken (i.e., no paraphrasing). Considering the legal requirements of live and recorded captioning, the subjectivity of NER scoring makes it an inherently risky method.

How are NER Scores Calculated?

Vendors grade each caption error based on its severity or resulting understandability when using the NER model. In many cases, vendors decide for themselves what constitutes a critical error. This subjectivity means that a caption file could get different NER results depending on who scores the file—contributing to significant liability for customers. 

One of the reasons NER scores get inflated so quickly is that the denominator of the NER equation is all of the words written. However, the numerator, which is the number of correct words, also starts at the total count of words and is only deducted by fractions of certain words, even if a whole sentence is paraphrased or several words are wrong in sequence. In addition, the denominator is the total number of words captioned, not the total number of words that should have been captioned based on verbatim dialogue.

 Discover Captioning Best Practices for the Entertainment Industry  

Types of NER Errors

NER errors are categorized under two main types, each with corresponding deduction values of either 0.0, 0.25, 0.5, or 1.0 (a full point deduction). In this way, the NER model functions more as a score than a percentage. Caption scoring begins at 100 and is graded according to the number of errors and their assigned score deductions. 

Of note, the NER marking of “Correct Edition” indicates that paraphrased captions capture the full meaning of the spoken content. However, a Correct Edition marking might have a starkly decreased WER score with no deduction in the NER score. At 3Play Media, we see many examples of this difference, which is consequential for accessibility and legal compliance with FCC standards and other legislation.

Edition errors represent the loss of an idea unit or piece of information. They include:

• Critical Error (False Information): An editing or paraphrasing error provides false but plausible information (-1.0)
• Major Error (Loss of Main Point): Inaccurate captions lose the main point of an idea (-0.5)
• Minor Error (Loss of Detail): Inaccurate captions keep the main point but lose a detail (-0.25)
• Correct Edition: Paraphrase captures the full meaning (0.0)

Recognition errors represent misrecognition of the spoken content. They include:

• Critical Error (False Information): A wrong word, phrase, or punctuation error provides false but plausible information (-1.0)
• Major Error (Nonsense Error): A wrong word, phrase, or punctuation affects comprehension of an idea. (-0.5)
• Minor Error (Benign Error): A wrong word, phrase, or punctuation affects readability but not comprehension. (-0.25)

Word Error Rate (WER) and Formatted Error Rate (FER)

More commonly, captioning accuracy for recorded content is often made up of two pieces: Word Error Rate (WER) and Formatted Error Rate (FER). WER is the standard measure of transcription accuracy and considers the number of inaccurate words versus the total number of words. In contrast, FER is the percentage of word errors when formatting elements such as punctuation, grammar, speaker identification, non-speech elements, capitalization, and other notations are taken into account. 

For closed captioning, the FCC mandates all of these formatting requirements to achieve at least 99% accuracy for recorded content. For recorded and live content, the FCC quality standards do not permit the same amount of flexibility that the NER model allows. While live captioning does not have firm accuracy standards and instead relies on best practices, the FCC still focuses on accuracy, synchronicity, completeness, and placement—which are more aligned with WER and FER than NER. 

WER and FER vs. NER: Unequal Measures of Accuracy and Quality

Compared to WER and FER, NER is not an equivalent measure of accuracy or quality. While captioning with a high NER score may be useful for viewers who value overall meaning instead of absolute accuracy, higher WER and FER measurements are essential for d/Deaf and hard-of-hearing viewers and legal compliance.

Some vendors do not state which model they’re using in determining accuracy for recorded captions, which is a misleading practice and can put your content at risk for litigation. When evaluating a potential vendor, you should always inquire about the models they use to determine the accuracy of their captions.

Additionally, NER scoring is more beneficial for live content and less applicable to recorded content, so be wary when a vendor uses NER to describe accuracy for recorded captioning. There are inherent challenges in captioning live content, and recorded captions should be measured differently because the captioner has more time and can perfect the verbatim transcription. NER scoring, if used, should always be near perfect for recorded content because a recorded captioner should never need to summarize the spoken content—captions should achieve verbatim accuracy and, by doing so, retain meaning.

Ultimately, when offering closed captioning as an accommodation, the best practice is often to provide an equitable experience by presenting content as spoken, which necessitates using WER and FER to measure accuracy.

About the author

3Play Media

The post Measuring Captioning Accuracy: Why WER and NER Analyses Differ appeared first on 3Play Media.

Blog

---

• Legislation & Compliance

Legal Requirements for Stadium Captioning

3Play Media

•

April 19, 2023

Baking Accessibility into Your Event Strategy [FREE webinar]

Whether it’s a concert, sporting event, or theatrical performance, attending live events is a source of joy and excitement for many people. The energy of the crowd, the spectacle of the performance or game, and the sense of being part of something special all contribute to the magic of live entertainment.

However, for those who are deaf or hard of hearing, this experience is incomplete without access to live captions. In-stadium captioning ensures that everyone has the opportunity to fully experience the event. This blog will cover legal requirements for accessible in-stadium viewing.

The Americans with Disabilities Act

Signed in 1990, the Americans with Disabilities Act (ADA) is the most far-reaching piece of accessibility legislation in the U.S.

The act and its amendments guarantee equal opportunity for disabled people in employment, state and local government services, public accommodations, commercial facilities, and transportation. The ADA affects both public and private entities.

The ADA mandates that it’s the responsibility of public and private organizations to provide equal access through appropriate accommodations. The act includes 5 sections or “Titles;” Titles II and III impact web accessibility and closed captioning.

Stadium Captioning Accessibility Laws Under the ADA

Under Title III of the ADA, stadiums and arenas must provide auxiliary aids and services, including captioning, to ensure effective communication for individuals who are deaf or hard of hearing. Specifically, the ADA’s regulations on “Nondiscrimination on the Basis of Disability in Public Accommodations and Commercial Facilities” (28 CFR Part 36) provide guidance on the requirements for effective communication for D/deaf and hard of hearing individuals. This requirement applies to both new and existing facilities.

The specific requirements for in-stadium captioning under the ADA include the following:

• Captioning for public address announcements: Stadiums must provide captioning for all public address announcements made during events, such as game scores, player names, and other important information.
• Captioning for videos: If stadiums display videos on scoreboards or other screens, they must provide closed captioning for those videos.
• Captioning for emergency announcements: In the event of an emergency, stadiums must provide captioning for any announcements made over the public address system.
• Captioning for other communications: Stadiums must also provide captioning for any other communications that are necessary to ensure effective communication for individuals who are D/deaf or hard of hearing.

There is no minimum seating capacity under the ADA that would exempt a stadium or arena from providing accessibility for disabled individuals. The ADA applies to all public accommodations, regardless of their size or capacity.

While captions are legally required for any type of event, specifications may vary based on factors like venue size or the type of event. For example, a sports event may require captioning that can keep up with fast-paced commentary, whereas a concert may require captioning that can be synced to the music.

It’s also important to consider that live captions are not enough to be fully accessible. American Sign Language (ASL) interpreters should be considered for in-stadium events in addition to live captioning. As professional sports sign language interpreter Brice Christianson explained in an episode of 3Play Media’s Allied podcast, English is a second language for many in the Deaf community:

 Learn how to bake accessibility into your event strategy 

Past Legal Settlements for Stadium Accessibility

Let’s review some settlements between the U.S. Department of Justice (DOJ) and various universities and venues.

Ohio State University
In 2009, a group of deaf students at Ohio State University filed a complaint alleging that Ohio State’s athletic department discriminated against D/deaf and hard of hearing individuals by failing to provide auxiliary aids and services at Ohio Stadium and Value City Arena at the Jerome Schottenstein Center.

An agreement was reached between Ohio State University and the DOJ that requires the university to provide open captioning on the scoreboard and closed captioning through individual devices at all home games.

The settlement also requires the university to provide open captioning for all public announcements and emergency alerts made through its public address system. Captions must be visible from all areas of the stadium and remain on the scoreboard until the corresponding announcement is complete.

Under the agreement, Ohio State University must also provide training to its staff about how to ensure that the captioning is functioning properly and provide assistive listening devices to D/deaf and hard of hearing attendees.

Ohio State is part of the Big Ten Conference of universities, the oldest Division 1 collegiate athletic conference in the United States. The NAD used the Ohio State settlement as a model to other Big Ten universities, sending them a letter outlining the settlement agreement with Ohio State and requesting that these universities adopt similar policies and practices to ensure their stadiums provide equal access to deaf and hard of hearing fans.

The Denver Pepsi Center

In 2018, a deaf individual filed a complaint against the Denver Pepsi Center, alleging that the arena violated the ADA by failing to provide captioning during games.

The owner of the Denver Pepsi Center settled the lawsuit with a consent decree that requires open captions on ribbon boards that can be seen from every seat in the stadium.

The captions cover all public announcements, and an independent monitor was appointed to check the accuracy of the captions.

The University of Maryland

In 2013, the NAD filed a lawsuit against the University of Maryland on behalf of two deaf individuals who regularly attended athletic events at the university. The events were not captioned and violated the ADA.

The agreement between the University of Maryland and the DOJ requires the university to provide accessible captioning services, including closed captioning on screens and assistive listening devices, for all home football and basketball games.

The University of Maryland must provide captions that are “accurate, complete, and synchronized with the spoken words,” and provide training to staff on the use of captioning equipment and services.

In-Stadium Captioning: A Necessity for Accessibility and Legal Compliance

In-stadium captioning is a legal requirement that must be fulfilled by stadiums and event organizers. Failure to comply with in-stadium accessibility requirements can result in legal action and penalties. Therefore, it is essential for stadiums to prioritize fulfilling these legal requirements to avoid legal consequences and to promote equal access for all fans.

About the author

3Play Media

The post Legal Requirements for Stadium Captioning appeared first on 3Play Media.