Reconsidering Positional Supervision in Masked Diffusion Language Model Training

Masked diffusion language models (MDLMs) generate text by unmasking tokens in parallel and have recently emerged as alternatives to autoregressive language models. They can be viewed as parallel decoders trained with a position-wise cross-entropy (CE) loss, the same setup as non-autoregressive translation (NAT). In NAT, CE-trained parallel decoders have been argued to be sensitive to small positional shifts, since CE penalizes them harshly. We ask whether CE-trained MDLMs are similarly sensitive to such shifts under iterative decoding. To probe this, we apply a controlled intervention that introduces them during decoding. On LLaDA-8B-Instruct with Arena-Hard, displacing as little as 1% of generated tokens by one position substantially reduces win rates against the unintervened model, showing that MDLMs are sensitive to such small shifts under iterative parallel decoding. Motivated by this, we adapt connectionist temporal classification (CTC), an alignment-flexible objective known to mitigate it there, to MDLM supervised fine-tuning. By relaxing the strict position-wise match that CE imposes, CTC gives the loss room to absorb small positional shifts; concretely, we modify the CTC objective to use a special <slack> token that absorbs positional uncertainty between target tokens and output positions, and an updated collapse map that preserves target surface forms. Across four open-ended generation benchmarks, the resulting model consistently improves over both the original model and a matched cross-entropy-trained baseline, with statistically significant gains on all four. These results identify training-side alignment flexibility as a useful design dimension for MDLM SFT, complementary to the inference-time approaches explored in prior work.

We relax position-wise matching with a CTC-inspired loss, using a <slack> token to absorb small positional shifts. In NAT, this sensitivity has been addressed by replacing position-wise CE with connectionist temporal classification (CTC), which lets the model emit blank symbols between content tokens and treats outputs that differ only in blank placement as equivalent. Inspired by this, we add a special <slack> token that plays the role of the blank and is removed from the output after decoding. Given a masked response \(\tilde r_t\) that contains <slack> tokens, our CTC-S loss sums over every alignment \(a\) that the collapse map \(\beta_{\mathrm{slack}}\) turns into the clean response \(r_0\) by removing <slack>:

In training, we insert <slack> tokens at random positions at a ratio \(s = s_{\max}(1-t)\) with \(s_{\max} = 0.5\), and add CTC-S to the standard CE loss: \(\mathcal{L}^{\mathrm{SFT}} = \mathcal{L}^{\mathrm{CE}} + \lambda\, \mathcal{L}^{\mathrm{CTC\text{-}S}}\) with \(\lambda = 0.1\).

How the model uses <slack>

• Selective. Every training response contains <slack>, yet only 83 of 160 MT-Bench outputs (51.9%) use it at all.
• Calibrated. In outputs that use it, <slack> makes up 27.5% of tokens, close to the 25% average rate in training.
• Between words. Training inserts <slack> anywhere, including inside words, yet only 4.9% of emitted <slack> tokens fall inside a word.

Sensitivity to the slack ratio

We fixed \(s_{\max} = 0.5\) in advance rather than tuning it. With \(s_{\max} = 0.2\) or \(0.8\), CE + CTC-S still improves significantly over CE-Only on Creative Writing Bench v3 and MT-Bench. \(s_{\max} = 0.5\) gives the best overall balance, and only \(s_{\max} = 0.2\) is significantly worse than CE-Only on any benchmark (Arena-Hard).