HuMuLaDisCo: Human and Machine in Multimodality, Multilingual Communication, Language, and Distributed Cognition

The EU-funded project "NewWorkTech" engages in research-based enhancement of the work-related capacities of people with disabilities and the general workforce, with particular focus on technologically-mediated tasks and interaction. We consider people with disabilities to be experts in using technical systems to improve their work performance, and through an analysis of a variety of use cases, will identify elements of successful use as well as factors that cause problems for interaction or the performance of tasks. The NewWorkTech project has received funding from European Union’s Horizon research and innovation programme under Grant Agreement No 101177176.

The nature of cognition in real-life work settings has attracted the interest of scholars from different academic disciplines in the last decades. However, a professional activity that so far has received limited attention in research, despite its potential to produce considerable insight into the nature of cognition in large-scale organisations, is translation work in institutional settings. Institutional translation, e.g. in the European Union institutions, is characterized not only by institutional norms and regulations which are transferred across actors, artifacts, and time, but also by the ubiquitous use of technology. As such, it appears particularly well-suited for advancing the current knowledge of how cognitive processes, such as knowledge, remembering, and decision making, are distributed across actors in teams in institutional settings, as well as through their interactions with technology. This project aims to explore and model distributed cognitive processes by investigating social interaction and human-technology interaction in institutional translation settings, thus pushing the boundaries of current conceptualizations of distributed cognition in institutions. Ultimately, the study will result in the development of a theoretical model of distributed cognition in institutional translation settings that will inform future research into socio-cognitive aspects of cognition, interaction, and cross-linguistic communication. These research objectives will be pursued through the interdisciplinary integration of theoretical insights and methodologies from cognitive science, linguistics, anthropology, and translation studies. Specifically, it will combine ethnographic fieldwork with micro-ethnographic multi-modal interaction analysis to gather data on team interactions in the EU institutions, primarily by means of video recordings, but also with other types of discursive, naturally occurring data, such as documents produced and used in the institutions. TRAILS is a postdoctoral fellowship awarded to Raphael Sannholm and funded by the Horizon Europe Marie Skłodowska-Curie Actions.

HuMuLaDisCo conducts multi-and interdisciplinary research collaboration with different fields such as (audio) signal processing. For example, in the project "Guided audio captioning for complex acoustic environments" (GUIDE), we have investigated audio captioning as textual description of the most important sounds in the scene, guided towards finding the most important content while ignoring other sounds, similar to humans. We seek to understand to what extent automatically produced audio captioning is suitable for subtitling for deaf and hard-of-hearing audiences, if the manual work required for the descriptions of the non-speech soundtrack can be replaced by automatic methods, similar to the dialogue being translated into subtitles using automatic speech recognition.