Objectives

Industrial Research and Experimental Development

The goals of the RDI program are divided into two groups: 1. goals related to industrial research (TRL 3-4) and goals related to experimental development (TRL 5-6).

The goals of industrial research are based on state-of-the-art artificial intelligence technologies – large language models that will be enhanced with additional knowledge and specialized training datasets and adapted for increased computational efficiency, speech recognition, industrial use, and the Slovenian language.

The goals of experimental development relate to the use of developed and adapted large language models and speech recognizers in specific industrial applications: in medicine, industry, humanities, and in generating code for computer infrastructure.

Large Language Models

The project addresses large language models that currently impact almost the entire field of artificial intelligence and machine learning, significantly affecting various other fields and society as a whole. The developed new, freely accessible, computationally efficient language models will serve as the basis for advanced applications in medicine, humanities, industrial environments, and software development. The research will yield numerous cutting-edge results, described first for TRL 3-4 levels and then for TRL 5-6 levels.

Infrastructure for Artificial Intelligence

To develop training datasets for large language models, we will create a language technology pipeline that utilizes existing large language models to generate initial versions of the training dataset, which will be subsequently improved by human annotators. Such a pipeline does not currently exist but represents a significant acceleration of capturing human knowledge for use in artificial intelligence. We will demonstrate the pipeline’s use in four important areas.

Built freely accessible large generative language models and their adaptation for following commands, dialog communication, and the Slovenian language constitute the fundamental infrastructure for artificial intelligence applications in Slovenian. Current large language models, built in previous research, have garnered significant interest with thousands of monthly downloads from the HuggingFace repository. They form the basis for numerous language technology applications in multiple languages. We expect that the new models will be even more successful, enabling a new generation of artificial intelligence applications.

Embedding Knowledge into Large Language Models

We will address the potentially unreliable and non-robust operation and hallucinations of large language models with new methods of embedding knowledge and external sources. The proposed approaches will tackle issues with logical reasoning, common-sense reasoning, language and morphological peculiarities of the Slovenian language, and adherence to ethical norms. The results will enhance the reliability, robustness, and transparency of large language models, both for Slovenian and other morphologically rich languages.

Adaptations for Low-Power Devices

Developed adaptations of large language models for computationally low-power devices and industrial applications will tailor new compression and distillation approaches for large language models, as well as adapt quantization and approximate computation for morphologically rich languages. As the high computational demand of large language models is a significant obstacle to their use, the developed new models and adaptation methods will open up new application possibilities in the economy and public sector.

Speech Technologies

Speech technologies and speech interfaces significantly enhance the user experience for many interactive applications. The proposed new method of incorporating large language models into speech recognition, based on their computational efficiency, can substantially improve recognition quality in noisy environments, with multiple speakers, and in recognizing non-standard language (e.g., dialects). Since the underlying improved speech recognition models will be freely accessible, this will enable new applications in the public sector and the economy. The developed models will support multilingual and multimodal applications, adding another dimension of applicative potential.

Industrial Applications

For the field of medicine, developed large language models and language interfaces will improve the user experience, reduce the time required by doctors and healthcare staff, enhance the quality of inputted results, and reduce unwanted errors. The novelty of the developed application will have a significant impact on the field of information solutions in medicine and enable next-generation applications.

For the field of humanities, developed large language models and language interfaces will significantly shorten the preparation time for materials, increase the accessibility of materials, and improve the user experience. The novelty of the application will have a significant impact on the field of information solutions in humanities and enable next-generation applications in museums.

In industry, especially in noisy production environments, improved speech recognition will enhance human-machine communication. The improvements will enable a different ergonomics of voice communication and the integration of speech technologies into IT infrastructure, improving communication in noisy environments and environments with non-standard pronunciation and different languages. This will reduce the onboarding time for new workers, decrease errors, and increase workplace safety.

For the software industry, adapted technologies of computationally efficient large language models and a new annotation pipeline will enable the efficient construction of a new language model for generating descriptions of computer infrastructure in code. This will increase the robustness and usability of products using these technologies and serve as a model for innovative applications in the software development industry.