GitHub - kuleshov-group mdlm: [NeurIPS 2024] Simple and Effective . . . We introduce MDLM, a M asked discrete D iffusion L anguage M odel that features a novel (SUBS)titution based parameterization which simplifies the absorbing state diffusion loss to a mixture of classical masked language modeling losses
Simple and Effective Masked Diffusion Language Models As shown in the table below, MDLM outperforms the previous diffusion models and nearly matches the performance of AR models in text generation on the LM1B dataset
arXiv:2406. 07524v2 [cs. CL] 10 Nov 2024 % improvement on the perplexity bound Finally, MDLM gets within 14% of an AR baseline and c ntinues to improve with more training We see the same trend for models trained on OWT, a larger dataset, shown in Table 2 – MDLM outperforms prior diffusion meth