Challenge 3: LLMs for Spoken Language

LLMs are fundamentally dependent on data. In language use, substantial variations exist among different types of discourse. The majority of linguistic data derive from written resources. Extensive spoken data resources, if accessible for research, are typically owned by various companies or institutions and are rarely available to the research community. However, even when these data are publicly accessible, they differ significantly from the speech used in everyday conversations. Features such as looser sentence structures, interruptions, cross-talk, silences, repairs, repetitions, misarticulations, clarifications, backchannels, conflicts, slurs, and jargon (Yeomans et al., 2023) exemplify surface-level specifics of conversation and indicate that investigating such data has great impact potential in the linguistic research (Love et al. 2014) as well as poses significant challenges for artificial intelligence (Wahlster 2023). In this challenge, we aim to advance speech technologies and research through the newest methodologies in collecting, filtering, automatically transcribing, and pragmatically processing speech data with the help of LLMs.

Task 3.1 Task 3.2 Task 3.3 Task 3.4

T3.1 Efficient spoken language data collection

Some conversational data in Slovene are available (Verdonik et al. 2024), yet they fall significantly short of the current state-of-the-art (Love et al. 2017). Thus, developing a modern and sustainable solution for conversational data collection is the first challenge we address. Collecting speech data remotely is essential for obtaining geographically dispersed data, especially for languages other than English (Parent & Eskénazi, 2011). However, the primary challenge lies in motivating citizens to donate speech. Little attention has been devoted to comprehending citizens’ motivation for contributing speech data or to citizens’ attitudes toward technologies based on LLMs. We aim to investigate citizens’ perspectives on speech-enabled LLM technologies, identify their motivations, and establish an effective online approach for sustainable speech data acquisition.

We will use successful crowdsourcing platforms, Games-With-A-Purpose (GWAPs) and Collect4NLP, designing an online interface for collecting conversational speech data, taking into account economic, ethical, and legal aspects, while simplifying and automating metadata collection. Further, we will investigate user perspectives on speech-enabled LLM technologies and explore strategies to motivate speakers to contribute to speech databases, e.g., the importance of trust and privacy. We will apply self-determination theory to highlight the role of intrinsic motivations like contributing to scientific research, preserving language heritage, and social connection. The collected audio data, manually transcribed data, and the multi-reference transcription of the benchmarking part of the data will be publicly released.

Deliverables 3.1: Online interface for collecting conversational speech data (M4). Manually transcribed conversation data for the spoken learning corpus (5 hours) (M7). Manually multi-reference-transcribed data for the spoken benchmark corpus (1 hour new data + 3 hours of existing ASR data) (M25). Audio speech database of conversation data (M36).

T3.2 Semantic and pragmatic speech processing

Humans interpret meaning in conversation across multiple dimensions. One critical level is the intent or function of an utterance. For example, a simple “Why?” can be interpreted as a request for information or an expression of hesitation, depending on the context. Speech act theory (Austin 1975) elucidated this level of meaning and became one of the most influential theories in pragmatics. However, it proved inadequate for authentic data (Levinson 2017). Consequently, researchers developed alternative interpretations under the term ‘dialogue act’. Describing speech use, particularly conversational speech, with robust, well-balanced, and appropriate annotations and interpretations of dialogue acts and related levels of meaning in context holds promise for a better understanding of how we communicate meaning, as well as for more efficient semantic and pragmatic processing of speech. With the advent of speech-based LLMs (Baevski et al. 2020; Radford et al. 2022) that perform both feature extraction, as well as long-range feature interaction, processing speech while taking into account pragmatic categories has become feasible (Miah et al. 2023).

Annotations of dialogue acts and related semantic levels, particularly the expression of sentiment, will be performed on Slovenian conversational data with a minimum duration of five hours. We will critically analyze the fundamental dialogue act categories and their relationships to other semantic levels, such as sentiment expression. We will automate dialogue act identification and sentiment identification from speech data, using speech-enabled LLMs pre-trained also on Slovenian, e.g., XLS-R, MMS, Whisper, and Seamless.

Deliverables 3.2: Expanded learning spoken corpus with dialogue act and sentiment annotations (min. 5 hours of conversational speech) (M10). Models for dialogue act and sentiment identification in Slovenian speech (M30).

T3.3 Speech data quality control and filtering

Speech data is crucial for developing LLMs and adapting them to recognize spoken language effectively (Prabhavalkar et al., 2023). However, collecting such data, especially for less-resource languages like Slovenian, poses significant challenges. A typical strategy involves sourcing recordings from various publicly accessible media and streaming platforms. These recordings often vary greatly in terms of speaker diversity, domains, dialects, and acoustic environments, and exhibit a broad range of quality levels. Using unfiltered data for large-scale model training or fine-tuning can negatively impact the models’ accuracy and reliability. Moreover, in line with green ICT principles, which prioritize reducing energy consumption and greenhouse gas emissions (Georgescu et al., 2021), indiscriminate use of large datasets is environmentally concerning. Therefore, assessing the suitability of data prior to its deployment in training or fine-tuning processes is imperative to manage large speech datasets effectively. This step is also crucial for maintaining accuracy and minimizing biases in digital humanities research that relies on language technologies. We aim to develop speech data selection methods to enhance efficiency in speech data collection and analysis.

Slovenian speech data will be acquired from publicly available resources, e.g., publicly available videos. These datasets encompass a diverse range of speakers from various demographic groups. The recordings may present challenges such as low quality, background noises, non-speech elements (e.g., silences, singing, music), overlapping speech, and foreign language content. We will protect speakers who are underage or belong to socially vulnerable groups. Initially, we will analyze the characteristics and attributes of the sample data. The identified categories will be utilized to (a) select objective speech/audio metrics for quality control and (b) define attribute-based acoustic classifications of the speech recordings. Both components will be integrated into a pre-selection process for the collected speech recordings. The developed speech database will be evaluated using the ASR system with large general speech models.

Deliverables 3.3: Slovenian audio speech database from publicly available resources (min. 300 hours) (M36)

T3.4 LLMs for domain-specific speech recognition

Highly accurate automatic speech recognition (ASR) remains challenging, especially in real-world domains with background noise, accented speech, code-switching between languages/dialects, and domain-specific vocabularies (Hu et al, 2024). Traditional ASR models often struggle in such conditions. Integrating LLMs pre-trained on vast text data, can potentially improve ASR robustness by providing linguistic knowledge to constrain the transcription search space (Ma et al., 2024; Miao et al., 2022; Min & Wang, 2023). However, effectively combining ASR and LLM components poses research challenges, specifically for less-resource languages that are under-represented in LLM pretraining data (Zhengdong et al, 2024; Hu et al, 2023b). The objective of this research task is to conduct a comprehensive comparative analysis to assess the efficacy of various techniques for integrating ASR and LLMs taking into account the limitations of less-resource languages. We aim to develop an innovative method for domain-specific ASR that will leverage the potential of ASR-LLM integration and will be optimized for less-resource languages.

We will conduct a comprehensive study on integrating ASR and LLMs (covering shallow and deep fusion approaches) to measure the efficacy of the specific integration methods for ASR improvement and robustness. The focus will be on a) the identification of approaches that are best suited to less-resource languages given their limitations in terms of LLMs, and b) the usability of the particular ASR-LLM integration methods for domain/context switching (in real-time) without the need for ASR model adaptation. Based on the results, we will develop a new method for domain-specific speech recognition that will leverage the potential of ASR-LLM integration for less-resourced languages.

Deliverable 3.4: A new ASR-LLM integration method for domain-specific ASR for low-resource languages (M36)

Challenge 3: LLMs for Spoken Language

T3.1 Efficient spoken language data collection

T3.2 Semantic and pragmatic speech processing

T3.3 Speech data quality control and filtering

T3.4 LLMs for domain-specific speech recognition

CONTACT

DURATION

LOCATION

FINANCED BY

Archive