publications

2024

1. FTVA

   Towards an End-to-End Personal Fine-Tuning Framework for AI Value Alignment

   2024

   Abs PDF

   This study introduces a novel architecture for value, preference, and boundary alignment in large language models (LLMs) and generative AI systems, accompanied by an experimental implementation. It addresses the limitations in AI model trustworthiness stemming from insufficient comprehension of personal context, preferences, and cultural diversity, which can lead to biases and safety risks. Using an inductive, qualitative research approach, we propose a framework for personalizing AI models to improve model alignment through additional context and boundaries set by users. Our framework incorporates user-friendly tools for identification, annotation, and simulation across diverse contexts, utilizing prompt-driven semantic segmentation and automatic labeling. It aims to streamline scenario generation and personalization processes while providing accessible annotation tools. The study examines various components of this framework, including user interfaces, underlying tools, and system mechanics. We present a pilot study that demonstrates the framework’s ability to reduce the complexity of value elicitation and personalization in LLMs. Our experimental setup involves a prototype implementation of key framework modules, including a value elicitation interface and a fine-tuning mechanism for language models. The primary goal is to create a token-based system that allows users to easily impart their values and preferences to AI systems, enhancing model personalization and alignment. This research contributes to the democratization of AI model fine-tuning and dataset generation, advancing efforts in AI value alignment. By focusing on practical implementation and user interaction, our study bridges the gap between theoretical alignment approaches and real-world applications in AI systems.

2. wtf

   Where to Fuse?

   L Petersson

   2024

   Abs PDF

   This thesis investigates fusion techniques in multimodal transformer models, focusing on enhancing the capabilities of large language models in understanding not just text, but also other modalities like images, audio, and sensor data. The study compares late fusion (concatenating modality tokens after separate encoding) and early fusion (concatenating before encoding) techniques, examining their respective advantages and disadvantages. It examines a mid-fusion approach, aiming to combine the strengths of both methods. The effectiveness of this approach is evaluated in terms of accuracy and computational impact on the Visual Question Answering (VQA) task. Using a pretrained T5 model, the research incorporates image tokens (calculaed by Vision Transformer, ViT) into intermediate activations of the model. The findings indicate that standard early fusion techniques underperform with larger decoders, while late fusion with a smaller decoder yields the best results on the VQA task. This conclusion also extends to pooled modality tokens. Additionally, the thesis includes a comprehensive literature study, identifying benchmark datasets for video understanding in multimodal learning and highlighting datasets that demand a robust understanding of all involved modalities. This research contributes to the field by exploring and validating a novel fusion technique in multimodal learning, offering insights into its practical applications and limitations.

2021

1. mbse

   Leveraging the Eclipse Modeling Framework to work with Electronic Datasheets

   L. Petersson, and Perillo D

   Model Based Space Systems and Software Engineering, 2021

   Abs PDF

   This abstract provides a practical guide to leverage the Eclipse Modeling Framework (EMF) for working with Electronic Datasheets (EDS). Starting from the SOIS EDS definition, available on the SANA website, it will be explained how to setup an EMF working environment, and how to generate a Tree Editor for editing and visualizing EDSs. It will also be explained how to exploit the Acceleo Model2Text (M2T) transformation language to navigate EDS models, and to generate artefacts in an almost automated manner. The problem of validating EDS models will also be discussed. A simple EDS use case will serve as a running example throughout the abstract. All the code mentioned in this abstract will be made available on the ESSR website.