Computational Technology for All

Authors: (1) Athanasios Angelakis, Amsterdam University Medical Center, University of Amsterdam – Data Science Center, Amsterdam Public Health Research Institute, Amsterdam, Netherlands (2) Andrey Rass, Den Haag, Netherlands. Table of Links 2.4 Alternative Architectures Have Variable Effect On Augmentation-Induced Bias Having carried out a data-centric analysis of DA-induced class-specific bias, we dedicated the last series of experiments (see Figure 4) to a more model-centric approach to the phenomenon. Balestriero, Bottou, and LeCun (2022) illustrates that

Authors: (1) Suzanna Sia, Johns Hopkins University; (2) David Mueller; (3) Kevin Duh. Table of Links 3. Where does In-context MT happen? 3.1. Layer-from Context Masking In-context learning differs from task-specific supervised learning in that, during test time, the desired task must be identified from the context first, then executed. At what stage in the feed-forward computation does a GPT-style model transition from an in-context learner to a translation model? To explore this question, we

Authors: (1) Suzanna Sia, Johns Hopkins University; (2) David Mueller; (3) Kevin Duh. Table of Links 7. Conclusion We demonstrate evidence that In-context Causal Decoder models locate the translation task at a specific layers during forward inference. To study this, we introduced causal masking of self-attention over the context from layer ℓ onwards (Section 3). The findings generalise across models of different sizes and in both non instruction-tuned and instruction-tuned models. We further identify certain

Authors: (1) Suzanna Sia, Johns Hopkins University; (2) David Mueller; (3) Kevin Duh. Table of Links 6. Further Analysis In the following sections, we focus on GPTNEO and BLOOM to conduct deeper analysis on the main phenomena presented in the paper. 6.1. Does the Number of Prompts Affect Task Recognition? In Section 3 we study context-masking with a fixed number of prompts. However, it is not clear if the number of prompts affects how fast,