Report Finds Public-Safety LTE Devices Have Speech Intelligibility Concerns
Wednesday, September 25, 2019 | Comments

A new report from a device testing company found Long Term Evolution (LTE) devices for public safety have speech intelligibility concerns.

The findings in the report show that phrase misunderstandings can occur in mission-critical communications scenarios up to almost 50 percent of the time, even when using devices that are marketed specifically for use by the public-safety community. The tests included two devices marketed specifically to the public-safety sector and two consumer-grade devices.

Spirent conducted testing of a representative sample of public-safety LTE handsets in its Maryland-based performance center, a testing facility that specializes in evaluating the communications performance of voice, video and data devices. The company evaluated speech intelligibility in four commercially available devices. Testing emulated use within a quiet environment, followed by six emulated first-responder noise environments including a nightclub, emergency siren and a fire-rescue situation using a saw.

Speech intelligibility is the ability of a device to transmit all the critical components of speech in the presence of background noises so that words and phrases can be easily understood.

“The testing that was completed was based on our ongoing work with a major North American wireless operator,” said Eric Sinclair, senior manager, business development public safety and mission-critical services for Spirent.

Spirent did not release the names or models of the devices it tested. In the findings, the company said that in some cases, consumer-grade devices outperformed devices purpose-built for public-safety use. The top-performing device outperformed the others tested by at least a six-to-one margin in certain emulated environments. No device proved perfect; intelligibility failure rates ranged from 1% to 49%, depending on the environment. Great performance in one environment did not necessarily correlate with similar performance in other emulations.

“Our testing looked at certain devices through the lens of how well they were able to transmit the essentials of speech while suppressing most background noise because that’s where the largest performance gaps emerge in public-safety applications,” said Spirent Vice President of Product Marketing Saul Einbinder. “The significant performance discrepancies we uncovered indicate that device makers must further re-evaluate designs and components to assure performance on the mission-critical LTE networks being rolled out around the world.”

Spirent applied an algorithm designed by the U.S. National Telecommunications and Information Administration (NTIA) and used by the National Institute of Standards and Technology (NIST) Public Safety Communications Research Division (PSCR). The ABC-MRT16 algorithm models the human auditory system and, in conjunction with a head and torso simulator in a 3D sound reproduction lab with automated testing, was used to assess over 80,000 distinct speech points to determine the quality of speech intelligibility.

Achieving a higher bar for device performance in public-safety environments begins in the development process, Spirent said. Software algorithms determine which parts of the audio are actually speech and yet still allow some background sounds to be transmitted to help the listener better understand situational context. Independently testing elements of the handset voice path, together with fine-tuning microphone circuitry and audio processing software, are some of the ways device designs can better accommodate first responder scenarios. Testing for a range of common public-safety scenarios is critical to establishing that a device will perform every time a first responder relies on it.

Spirent shared the performance results with the manufacturers whose devices were tested to offer guidance on where necessary adjustments can be made on future devices to improve performance. “One device manufacturer has implemented fixes to their handset design after our recommendations,” said Sinclair. “This is a service we offer to any device maker and are often brought into the process ahead of launch to help identify and address performance challenges before they hit the market.”

“While our testing found substantial room for improvement in some cases, the good news is that there is a pathway to better performance,” said Einbinder. “This is absolutely a solvable problem. First responders need to trust that the devices they use for life-saving measures will perform when it matters most, and we believe pinpointing performance challenges that exist early in the development process will help the entire industry take a major step toward that goal.”

Device speech intelligibility tests are nothing new in the public-safety sector. A 2017 NTIA report summarized the results of speech intelligibility tests on five speech codec operating modes that might be chosen to provide mission-critical voice services to public-safety users over LTE networks. In 2015, NTIA described an effort to identify which digital speech and audio technologies are best suited for mission-critical voice communications 4G commercial networks.

PSCR led tests around the audio of Project 25 (P25) digital radios beginning in 2008, paving the way for current public-safety LTE speech intelligibility testing. Spirent plans to conduct additional public-safety LTE device testing in its lab but does not have a confirmed timeframe.

“When we showed this report to other device manufacturers, they all wanted to see how their devices compared to the ones we tested already,” Sinclair said. “In just a short period of time, we’ve already had a new device in our lab for these purposes and more are coming shortly.”

Access the full report here.

Would you like to comment on this story? Find our comments system below.



 
 
Post a comment
Name: *
Email: *
Title: *
Comment: *
 

Comments
On 10/2/19, Leon van der Linde said:
Two-way radio manufacturers had decades to perfect noise-cancelling devices in the radios. LTE devices have a very long way to go before they reach the same levels. LTE devices and radio differ considerably so there will be some decades before they find the same solutions. I feel sorry for the first responders who have to battle through the noise until they find solutions to the problem.
One hurdle is that LTE devices are still commercial devices so they will not easily build in the right solutions.

On 10/2/19, Richard Wolverton said:
The communications industry is pushing too fast to get public safety off of LMR. This has placed restraints on agencies just trying to keep up with the P25 Phase 1 and 2 projects. Public safety is not ready for LTE, and the sooner the industry learns that error, the safer the community will be knowing that its public safety has reliable communications to respond to the needs of its citizens.

On 10/2/19, Greg Glenn said:
This goes back to the same issues we had with the vocoders and digital audio that came along with Project 25 (P25). Also with LTE I would imagine there will be potential latency issues with call setup.

Did my partner just say "shoot," and I missed the "don't" because of call setup?

On 10/2/19, Joe Blaschka said:
This should not be a surprise to anyone who has worked around digital voice including going back to the early days of digital cellular. The vocoding process is based on the assumption that the incoming audio is human voice with certain characteristics. When the human-voice audio is accompanied with other non-human audio signals, the vocoder does not estimate the total signal adequately. This has driven the P25 radio vendors to develop extensive noise cancelling and voice processing algorithms to clean up the voice audio prior to being vocoded. The same or similar approaches will be needed for MCPTT applications as well. This certainly places constraints on the types of radio equipment that work in a noisy public-safety response area or where the first responder is yelling into the microphone. This ground has been covered before, but I guess the lessons have to be relearned.

On 9/27/19, Eric Sinclair said:
Thank you for your comment John. All devices tested are certified devices included in the document provided by NIST entitled "NIST List of Certified Devices DATE 09 05 2019 VERSION 16.00." However the speech intelligibility testing we conducted is not a test currently performed by NIST PSCR FirstNet for certification.

On 9/26/19, John Roberts said:
This is a serious deficiency. The report does not state if any of these devices were FirstNet Certified, a certification of which comes from the same division labs that created the algorithm used in the testing. Myself, and I'm sure many other public-safety communication managers, would like to know if the NIST certification of FirstNet LTE devices requires them to pass this test at a certain level for the user equipment (UE) portion. Fortunately this can be corrected. In many cases, however, a device that fails should never be released to a first responder. Maybe MissionCritical Communications can follow up with FirstNet labs or we can get a response from them as to where this is headed. Great article.


Magazines in Print







Events
March 2020

30 - 4/3
International Wireless Communications Expo (IWCE)
Las Vegas
https://www.iwceexpo.com/

April 2020

6 - 9
APCO Western Regional Conference
Ogden, Utah
http://www.apcowrc.org/

7 - 9
ENTELEC Conference and Expo
Houston
https://www.entelec.org/event/2020-entelec-conference-expo/

May 2020

6 - 7
Comms Connect New Zealand
Wellington, New Zealand
https://www.comms-connect.co.nz/

More Events >

Site Navigation

Close