Publications

2026

1. NeurIPS

   Reconsidering Positional Supervision in Masked Diffusion Language Model Training

   Mengyu Ye, Keito Kudo, Ryosuke Takahashi, and Jun Suzuki

   In The Fortieth Annual Conference on Neural Information Processing Systems (to appear), Dec 2026

   Abs Paper Project Page

   Masked diffusion language models (MDLMs) generate text by unmasking tokens in parallel and have recently emerged as alternatives to autoregressive language models. They can be viewed as parallel decoders trained with a position-wise cross-entropy (CE) loss, the same setup as non-autoregressive translation (NAT). In NAT, CE-trained parallel decoders have been argued to be sensitive to small positional shifts, since CE penalizes them harshly. We ask whether CE-trained MDLMs are similarly sensitive to such shifts under iterative decoding. To probe this, we apply a controlled intervention that introduces them during decoding. On LLaDA-8B-Instruct with Arena-Hard, displacing as little as 1% of generated tokens by one position substantially reduces win rates against the unintervened model, showing that MDLMs are sensitive to such small shifts under iterative parallel decoding. Motivated by this, we adapt connectionist temporal classification (CTC), an alignment-flexible objective known to mitigate it there, to MDLM supervised fine-tuning. By relaxing the strict position-wise match that CE imposes, CTC gives the loss room to absorb small positional shifts; concretely, we modify the CTC objective to use a special <slack> token that absorbs positional uncertainty between target tokens and output positions, and an updated collapse map that preserves target surface forms. Across four open-ended generation benchmarks, the resulting model consistently improves over both the original model and a matched cross-entropy-trained baseline, with statistically significant gains on all four. These results identify training-side alignment flexibility as a useful design dimension for MDLM SFT, complementary to the inference-time approaches explored in prior work.

2. EMNLP

   Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages

   Tarek Naous, Anagha Savit, Carlos Rafael Catalan, Geyang Guo, Jaehyeok Lee, Kyungdon Lee, Lheane Marie Dizon, Mengyu Ye, and 12 more authors

   In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (to appear), Oct 2026

   Paper
3. arXiv

   Sumi: Open Uniform Diffusion Language Model from Scratch

   Mengyu Ye^*, Keito Kudo^*, Wataru Ikeda, Ryosuke Matsuda, Keisuke Sakaguchi, and Jun Suzuki

   Jun 2026

   Abs Paper Project Page

   Diffusion models have become a promising alternative to autoregressive models. Among these, uniform diffusion language models (UDLMs) permit any token to be updated at any step, in principle enabling more flexible generation. However, no UDLM has yet been pretrained from scratch at both large parameter scale and large token budget. Both autoregressive modeling and masked diffusion modeling already have capable models at scale that the community can study and build on; uniform diffusion has none. A scratch-pretrained UDLM at scale would provide a clean reference point for studying scaling behavior, generation dynamics, controllability, and trade-offs against established autoregressive and masked diffusion models. To this end, we introduce Sumi ("ink" in Japanese), a fully open 7B uniform diffusion language model pretrained from scratch on 1.5T tokens. Sumi performs competitively with autoregressive models trained at comparable token budgets on knowledge, reasoning, and coding benchmarks, while under-performing on commonsense benchmarks, where our education-heavy data mixture is a likely contributor. We release our model weights, checkpoints, and full training recipe, including a complete specification of the data mixture over publicly available corpora. We hope this release enables the community to study native uniform diffusion at scale and catalyzes work on its as-yet poorly understood aspects.

2025

1. arXiv

   An Open and Reproducible Deep Research Agent for Long-Form Question Answering

   Ikuya Yamada, Wataru Ikeda, Ko Yoshida, Mengyu Ye, Hinata Sugimoto, Masatoshi Suzuki, Hisanori Ozaki, and Jun Suzuki

   Dec 2025

   🏆 Best Static Evaluation Prize (Text-to-Text, Open Source), MMU-RAG Competition @ NeurIPS 2025

   Abs Paper Code

   We present an open deep research system for long-form question answering, selected as a winning system in the text-to-text track of the MMU-RAG competition at NeurIPS 2025. The system combines an open-source large language model (LLM) with an open web search API to perform iterative retrieval, reasoning, and synthesis in real-world open-domain settings. To enhance reasoning quality, we apply preference tuning based on LLM-as-a-judge feedback that evaluates multiple aspects, including clarity, insightfulness, and factuality. Our experimental results show that the proposed method consistently improves answer quality across all three aspects. Our source code is publicly available at https://github.com/efficient-deep-research/efficient-deep-research.

2. NeurIPS

   Transformer Key-Value Memories Are Nearly as Interpretable as Sparse Autoencoders

   Mengyu Ye, Jun Suzuki, Tatsuro Inaba, and Tatsuki Kuribayashi

   In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Dec 2025

   Abs Paper Project Page

   Recent interpretability work on large language models (LLMs) is increasingly dominated by a feature discovery approach with the help of proxy modules, where the quality of features learned by, e.g., sparse auto-encoders (SAEs), has been evaluated. This paradigm naturally raises a critical question — how much better features such proxies have indeed discovered, especially compared to those already represented within the original model parameters — and unfortunately, such a comparison has little been made so far. In this work, we revisit the interpretability of feature vectors stored in feed-forward (FF) layers, given the perspective of FF as key-value memories, with modern interpretability benchmarks. Our extensive evaluation revealed that SAE and FFs exhibits a similar range of interpretability, although SAEs displayed an observable but minimal improvement in some aspects. Furthermore, in certain aspects, surprisingly, even vanilla FFs yielded better interpretability scores than the SAEs, and features discovered in SAEs and FFs diverged. These bring questions about the advantage of SAEs from both perspectives of feature quality and faithfulness, compared to directly interpreting FF feature vectors.

3. ACL Findings

   Can Input Attributions Explain Inductive Reasoning in In-Context Learning?

   Mengyu Ye, Tatsuki Kuribayashi, Goro Kobayashi, and Jun Suzuki

   In Findings of the Association for Computational Linguistics: ACL 2025, Jul 2025

   Abs Paper

   Interpreting the internal process of neural models has long been a challenge. This challenge remains relevant in the era of large language models (LLMs) and in-context learning (ICL); for example, ICL poses a new issue of interpreting which example in the few-shot examples contributed to identifying/solving the task. To this end, in this paper, we design synthetic diagnostic tasks of inductive reasoning, inspired by the generalization tests in linguistics; here, most in-context examples are ambiguous w.r.t. their underlying rule, and one critical example disambiguates the task demonstrated. The question is whether conventional input attribution (IA) methods can track such a reasoning process, i.e., identify the influential example, in ICL. Our experiments provide several practical findings; for example, a certain simple IA method works the best, and the larger the model, the generally harder it is to interpret the ICL with gradient-based IA methods.

2023

1. EMNLP

   Assessing Step-by-Step Reasoning against Lexical Negation: A Case Study on Syllogism

   Mengyu Ye, Tatsuki Kuribayashi, Jun Suzuki, Goro Kobayashi, and Hiroaki Funayama

   In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (oral), Dec 2023

   🏆 Preliminary version received the Student Research Workshop Award at ACL 2023

   Abs Paper

   Large language models (LLMs) take advantage of step-by-step reasoning instructions, e.g., chain-of-thought (CoT) prompting. Building on this, their ability to perform CoT-style reasoning robustly is of interest from a probing perspective. In this study, we inspect the step-by-step reasoning ability of LLMs with a focus on negation, which is a core linguistic phenomenon that is difficult to process. In particular, we introduce several controlled settings (e.g., reasoning in case of fictional entities) to evaluate the logical reasoning abilities of the models. We observed that dozens of modern LLMs were not robust against lexical negation (e.g., plausible\rightarrowimplausible) when performing CoT-style reasoning, and the results highlight unique limitations in each LLM family.

2. SemEval

   TohokuNLP at SemEval-2023 Task 5: Clickbait Spoiling via Simple Seq2Seq Generation and Ensembling

   Hiroto Kurita, Ikumi Ito, Hiroaki Funayama, Shota Sasaki, Shoji Moriya, Mengyu Ye, Kazuma Kokuta, Ryujin Hatakeyama, and 2 more authors

   In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), Jul 2023

   Abs Paper

   This paper describes our system submitted to SemEval-2023 Task 5: Clickbait Spoiling. We work on spoiler generation of the subtask 2 and develop a system which comprises two parts: 1) simple seq2seq spoiler generation and 2) post-hoc model ensembling. Using this simple method, we address the challenge of generating multipart spoiler. In the test set, our submitted system outperformed the baseline by a large margin (approximately 10 points above on the BLEU score) for mixed types of spoilers. We also found that our system successfully handled the challenge of the multipart spoiler, confirming the effectiveness of our approach.