--- title: "Treating Decoder-Only LLMs as Encoders: A Simple and" description: "Decoder-only LLMs are trained with causal masking, so each token's representation sees only what comes before it. 4 references." authors: ["Ken Yano", "Hiroya Takamura"] year: 2026 venue: "BioNLP 2026" source: "https://aclanthology.org/2026.bionlp-1.25/" doi: "10.18653/v1/2026.bionlp-1.25" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/yano-2026-treating-decoder-only-llms-as/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yano-2026-treating-decoder-only-llms-as/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/54/) / Treating Decoder-Only LLMs as Encoders: A Simple and Effective Fine-tuning Approach for Named Entity Recognition # Treating Decoder-Only LLMs as Encoders: A Simple and Effective Fine-tuning Approach for Named Entity Recognition **Authors:** [Ken Yano](https://md.research.elamur.ai/free/en/authors/ken-yano/) and [Hiroya Takamura](https://md.research.elamur.ai/free/en/authors/hiroya-takamura/) · **Year:** 2026 · **Venue:** [BioNLP 2026](https://md.research.elamur.ai/free/en/venues/bionlp/) ([BioNLP 2026](https://md.research.elamur.ai/free/en/venues/bionlp/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.bionlp-1.25/) · [PDF](https://aclanthology.org/2026.bionlp-1.25.pdf) · [DOI 10.18653/v1/2026.bionlp-1.25](https://doi.org/10.18653/v1/2026.bionlp-1.25) · [OpenAlex](https://openalex.org/W7166879499) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Decoder-only LLMs are trained with causal masking, so each token's representation sees only what comes before it. Token-level tasks like named entity recognition need context on both sides, which is why encoders such as BERT usually do better. Yano and Takamura make decoder LLMs work as encoders. They swap the causal mask for full bidirectional attention in every decoder block, replace the language-modeling head with a token classification head, and train it on BIO labels. They test seven models from 0.5B to 8B parameters, drawn from Gemma, Qwen2.5, Llama3.2 and Llama3.1. Each is trained only on the BC5CDR (biomedical) and CoNLL2003 (general) training sets, with both full fine-tuning and LoRA. Bidirectional attention alone does not beat strong encoders. With a well-chosen LoRA setup, the best models outperform BERT, RoBERTa and DeBERTa, though they stay below published SOTA. This matters because existing LLMs can become drop-in encoders for sequence labeling without auxiliary data or new architectures. The trade-off is substantially higher training and inference cost. ## Key points - Models are labeled LS-FA-LLM (full attention) or LS-CA-LLM (causal attention). Baselines are bert-base-cased, roberta-base and deberta-v3-large, plus BioBERT_v1.1 for BC5CDR. Published SOTA from OpenMED (BC5CDR) and ACE (CoNLL2003) is cited for reference. - LoRA is preferred over full fine-tuning because full fine-tuning tends to overfit these small datasets, especially with large models. - The LoRA sweep shows a clear gain from targeting all linear modules rather than only q,k,v,o: - LS-FA-Llama3.1-8B reaches up to 0.904 micro F1 on BC5CDR and 0.932 on CoNLL2003. - LS-FA-Llama3.2-3B reaches up to 0.900 and 0.935. - In error analysis, LS-FA-Llama3.1-8B makes fewer errors than DeBERTa in every SPAN, TYPE and DETECTION subtype. Its causal-attention counterpart performs poorly, which points to the causal mask as the bottleneck. - On a single H200, the largest LS-FA models (Gemma-7B, Llama3.1-8B) take roughly 10× BERT's inference time. - The approach cannot handle nested or discontinuous entities. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [National Institute of Advanced Industrial Science and Technology](https://md.research.elamur.ai/free/en/labs/national-institute-of-advanced-industrial-science-and-technology/) — Research institute, Tsukuba, Japan · 18 works · 2,760 citations - [Artificial Intelligence Research Center](https://md.research.elamur.ai/free/en/labs/artificial-intelligence-research-center/) — Research institute, Tokyo, Japan · 9 works · 1,564 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🏷️ Named entity recognition](https://md.research.elamur.ai/free/en/topics/named-entity-recognition/) · 214 works ## Cite Cite as: Ken Yano and Hiroya Takamura. 2026. Treating Decoder-Only LLMs as Encoders: A Simple and Effective Fine-tuning Approach for Named Entity Recognition. BioNLP 2026. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{yano2026treating, title = {{Treating Decoder-Only LLMs as Encoders: A Simple and Effective Fine-tuning Approach for Named Entity Recognition}}, author = {Ken Yano and Hiroya Takamura}, booktitle = {BioNLP 2026}, year = {2026}, doi = {10.18653/v1/2026.bionlp-1.25}, url = {https://doi.org/10.18653/v1/2026.bionlp-1.25} } ``` - [← previous TravelBehaviorQA: A Benchmark Dataset for Behavioral Interpretation of GPS Trajectories](https://md.research.elamur.ai/free/en/papers/zhen-2026-travelbehaviorqa-benchmark-dataset-behavioral-interpretation/) - [next → Tree-Based Interview Topic Guidance for Collecting Target Information under Adaptive Topic continuation/switching](https://md.research.elamur.ai/free/en/papers/nagasawa-2026-tree-based-interview-topic-guidance/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/54/) ## References 4 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Arthur Amalvy, Vincent Labatut, and Richard Dufour. 2023. The role of global and local context in named entity recognition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 714–722, Toronto, Canada. Association for Computation… [find](https://scholar.google.com/scholar?q=Arthur+Amalvy%2C+Vincent+Labatut%2C+and+Richard+Dufour%2E+2023%2E+The+role+of+global+and+local+context+in+named+entity+recognition%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computation) 2. Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. [Llm2vec: Large language models are secretly powerful text encoders](https://md.research.elamur.ai/free/en/papers/behnamghader-2024-llm2vec-large-language-models-secretly/). Preprint, arXiv:2404.05961. 3. Yubo Chen, Chuhan Wu, Tao Qi, Zhigang Yuan, and Yongfeng Huang. 2020. Named entity recognition BC5CDR Model Baseline Encoders BERT RoBERTa DeBERTa IFT-LLM GPT-4.1 IFT-Gemma-2B IFT-Gemma-7B IFT-Qwen2.5-0.5B IFT-Qwen2.5-7B IFT-Llama3.2-1B IFT-Llama3.2-3B IFT-Llama3.1-8B LS-FA-LLM LS-FA-Gemma-2B LS-FA-… [find](https://scholar.google.com/scholar?q=Yubo+Chen%2C+Chuhan+Wu%2C+Tao+Qi%2C+Zhigang+Yuan%2C+and+Yongfeng+Huang%2E+2020%2E+Named+entity+recognition+BC5CDR+Model+Baseline+Encoders+BERT+RoBERTa+DeBERTa+IFT%2DLLM+GPT%2D4%2E1+IFT%2DGemma%2D2B+IFT%2DGemma%2D7B+IFT%2DQwen2%2E5) 4. Precision (set) Recall (set) F1 (set) Precision (set) Recall (set) F1 (set) Full Full Full 0.861 0.882 0.887 0.877 0.896 0.896 0.869 0.889 0.892 0.914 0.917 0.915 0.914 0.921 0.916 0.914 0.919 0.915 Zero-shot LoRA LoRA LoRA LoRA LoRA LoRA LoRA 0.670 0.870 0.878 0.803 0.881 0.866 0.879 0.901 0.505 0.… [find](https://scholar.google.com/scholar?q=Precision+%28set%29+Recall+%28set%29+F1+%28set%29+Precision+%28set%29+Recall+%28set%29+F1+%28set%29+Full+Full+Full+0%2E861+0%2E882+0%2E887+0%2E877+0%2E896+0%2E896+0%2E869+0%2E889+0%2E892+0%2E914+0%2E917+0%2E915+0%2E914+0%2E921+0%2E916+0%2E914+0%2E919+0%2E915+Z) *Updated 2 November 2026*