{"id":248672,"date":"2024-08-05T09:43:39","date_gmt":"2024-08-05T07:43:39","guid":{"rendered":"https:\/\/silicon-saxony.de\/tu-dresden-development-of-multilingual-language-models-opengpt-x-team-publishes-its-european-llm-leaderboard\/"},"modified":"2024-08-05T09:43:39","modified_gmt":"2024-08-05T07:43:39","slug":"tu-dresden-development-of-multilingual-language-models-opengpt-x-team-publishes-its-european-llm-leaderboard","status":"publish","type":"post","link":"https:\/\/silicon-saxony.de\/en\/tu-dresden-development-of-multilingual-language-models-opengpt-x-team-publishes-its-european-llm-leaderboard\/","title":{"rendered":"TU Dresden: Development of multilingual language models – OpenGPT-X team publishes its European LLM Leaderboard"},"content":{"rendered":"

<\/p>\n

Model development using common multilingual benchmarks<\/h3>\n

While most available benchmarks for evaluating language models are predominantly available for the English language, the OpenGPT-X consortium has set itself the goal of comprehensively expanding language accessibility for multilingualism and thus paving the way for fairer and more effective language technology. To reduce language barriers in the digital domain, the scientists carried out extensive multilingual training runs and then tested the developed AI models with regard to tasks such as logical reasoning, commonsense understanding, multitasking learning, truth content and translation skills.<\/p>\n

When developing LLMs, it is important that training and evaluation go hand in hand. To enable comparability across multiple languages, some of the most common benchmarks such as ARC, HellaSwag, TruthfulQA, GSM8K and MMLU were machine translated into 21 of the 24 supported European languages using DeepL. In addition, two further multilingual benchmarks were added to the leaderboard, which were already available for the languages considered in the project.<\/p>\n

The plan is to use the leaderboard to automate the evaluation of models from the AI platform Hugging Face Hub in order to enable the traceability and comparability of the results. TU Dresden will provide the necessary infrastructure for this and carry out the evaluation jobs on the HPC cluster. Following the current release of the European LLM Leaderboard, the OpenGPT-X models will be published this summer and will also be visible there. This is because one of the core goals of OpenGPT-X is to make the benefits of these AI language models available to a wider audience in Europe and beyond and to support a large number of European languages. This progress is particularly important for languages that are traditionally underrepresented in the field of natural language processing.<\/p>\n

TU Dresden with bundled big data, AI and HPC expertise in the project<\/h3>\n

With the expertise of the two competence centers ScaDS.AI (Scalable Data Analytics and Artificial Intelligence) and ZIH (Information Services and High Performance Computing) at TU Dresden, OpenGPT-X has a cooperation partner that pools expertise in training and evaluating large language models on supercomputing clusters. The joint efforts will focus on several critical tasks, including developing scalable evaluation pipelines, integrating various benchmarks, and performing comprehensive evaluations on supercomputing clusters. The team also focuses on improving model performance, scalability and efficiency, continuously monitoring the impact of pre-training and fine-tuning it, and leveraging innovative high-performance computing resources.<\/p>\n

Overview of the benchmarks translated in the project<\/h3>\n

ARC (https:\/\/huggingface.co\/datasets\/ai2_arc<\/a>) and GSM8K (https:\/\/huggingface.co\/datasets\/openai\/gsm8k<\/a>) focus on general education and math.<\/p>\n

HellaSwag (https:\/\/huggingface.co\/datasets\/Rowan\/hellaswag<\/a>) and TruthfulQA (https:\/\/huggingface.co\/datasets\/truthfulqa\/truthful_qa<\/a>) test the ability of models to provide plausible continuations and truthful answers.<\/p>\n

MMLU (https:\/\/huggingface.co\/datasets\/cais\/mmlu<\/a>) provides a wide range of tasks to assess the ability of models to perform in a variety of domains and tasks.<\/p>\n

While FLORES-200 (https:\/\/huggingface.co\/datasets\/facebook\/flores<\/a>) focuses on assessing machine translation skills, Belebele (https:\/\/huggingface.co\/datasets\/facebook\/belebele<\/a>) focuses on understanding and answering questions in multiple languages. <\/p>\n

Funding, project management and contacts<\/h3>\n

The OpenGPT-X project has been funded by the BMWK since 2022 and is largely coordinated by the Fraunhofer Institute for Intelligent Analysis and Information Systems (IAIS).<\/p>\n

Contacts<\/h3>\n

IAIS – Dr. Nico Flores-Herr, Dr. Michael Fromm
TU Dresden, ScaDS.AI: Dr. Ren\u00e9 J\u00e4kel, Klaudia-Doris Thellmann<\/p>\n

Publications<\/h3>\n