---
title: "A Time Series is Worth 64 Words — paper by Nie et al. (2022)"
description: "Zeng et al. (2022) had shown that a plain linear model (DLinear) could beat Transformer forecasters… 43 references, cited by 79 papers here."
authors: ["Yuqi Nie", "Nam H. Nguyen", "Phanwadee Sinthong", "Jayant Kalagnanam"]
year: 2022
venue: "arXiv:2211.14730"
source: "https://arxiv.org/abs/2211.14730"
doi: "10.48550/arxiv.2211.14730"
arxiv: "2211.14730"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nie-2022-time-series-worth-64-words/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nie-2022-time-series-worth-64-words/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/) / A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

# A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

**Authors:** Yuqi Nie, [Nam H. Nguyen](https://md.research.elamur.ai/free/en/authors/nam-h-nguyen/), Phanwadee Sinthong and Jayant Kalagnanam · **Year:** 2022 · **Venue:** arXiv:2211.14730 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2211.14730) · [PDF](https://arxiv.org/pdf/2211.14730) · [DOI 10.48550/arxiv.2211.14730](https://doi.org/10.48550/arxiv.2211.14730) · [arXiv 2211.14730](https://arxiv.org/abs/2211.14730) · [OpenAlex](https://openalex.org/W4310416691)

**Citations:** 548 citations (OpenAlex, 2026-10-14) · [79](https://md.research.elamur.ai/free/en/papers/nie-2022-time-series-worth-64-words/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Zeng et al. (2022) had shown that a plain linear model (DLinear) could beat Transformer forecasters such as Informer, Autoformer and FEDformer, which raised doubts about attention for time series. The authors respond with PatchTST. It keeps the vanilla Transformer encoder but changes how the input is tokenized. Each univariate series is instance-normalized and cut into overlapping subseries patches, which serve as tokens. A token therefore carries local temporal context instead of a single value, and the token count shrinks roughly by the stride. This makes long look-back windows affordable. Multivariate inputs are also split into separate channels that share one embedding and one set of Transformer weights (channel-independence). The same backbone supports masked pre-training, in which random patches are zeroed and then reconstructed. On benchmarks including Traffic, Electricity, Weather, ILI and the ETT sets, PatchTST outperforms earlier Transformer models. Self-supervised and transferred representations push accuracy further. The work shows that Transformers remain useful for forecasting and offers a simple, efficient recipe for representation learning.

## Key points
- In the Traffic case study (862 series, horizon 96), MSE falls step by step. It is 0.518 with L=96, 0.397 with L=336, 0.367 with patching, and 0.349 with self-supervised pre-training.
- Using patch length P=16 and stride S=8 at L=336 shortens training time about 22× on Traffic, 19× on Electricity and 4× on Weather.
- A longer history helps even when the number of tokens stays fixed. Downsampling a 380-step window to 96 tokens gives MSE 0.447, compared with 0.518 for the latest 96 steps.
- Adding channel-independence to Informer, Autoformer and FEDformer generally improves their results, but they still trail PatchTST/42.
- On Weather, channel-independent training benefits more from additional training data, while the channel-mixing variant overfits quickly.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Princeton University](https://md.research.elamur.ai/free/en/universities/princeton-university/) — University, Princeton, United States · 113 works · 48,612 citations
- [IBM](https://md.research.elamur.ai/free/en/companies/ibm/) — Company, Armonk, United States · 154 works · 72,606 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 249 works · [⚙️ Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/) · 188 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 493 works · [🎯 Management Science and Operations Research](https://md.research.elamur.ai/free/en/shelves/decision-sciences/management-science-and-operations-research/) · 134 works · [⚙️ Building and Construction](https://md.research.elamur.ai/free/en/shelves/engineering/building-and-construction/) · 7 works

## Cite

Cite as: Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong and Jayant Kalagnanam. 2022. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. arXiv:2211.14730.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{nie2022time,
  title = {{A Time Series is Worth 64 Words: Long-term Forecasting with Transformers}},
  author = {Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong and Jayant Kalagnanam},
  year = {2022},
  doi = {10.48550/arxiv.2211.14730},
  eprint = {2211.14730},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2211.14730}
}
```

- [← previous A Survey on Automated Fact-Checking](https://md.research.elamur.ai/free/en/papers/guo-2022-survey-automated-fact-checking/)
- [next → Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://md.research.elamur.ai/free/en/papers/smith-2022-using-deepspeed-megatron-train-megatron/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/)

## References

43 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. [wav2vec 2.0: A framework for self-supervised learning of speech representations](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/). Advances in Neural Information Processing Systems, 33:12449–12460, 2020.
2. Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018. [find](https://scholar.google.com/scholar?q=Shaojie+Bai%2C+J+Zico+Kolter%2C+and+Vladlen+Koltun%2E+An+empirical+evaluation+of+generic+convolutional+and+recurrent+networks+for+sequence+modeling%2E+arXiv+preprint+arXiv%3A1803%2E01271%2C+2018%2E)
3. Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEiT: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=p-BhZSz59o4](https://openreview.net/forum?id=p-BhZSz59o4). [↗](https://openreview.net/forum?id=p-BhZSz59o4)
4. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. [On the opportunities and risks of foundation models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/). arXiv preprint arXiv:2108.07258, 2021.
5. G. E. P. Box and Gwilym M. Jenkins. Time series analysis, forecasting and control. 1970. [find](https://scholar.google.com/scholar?q=G%2E+E%2E+P%2E+Box+and+Gwilym+M%2E+Jenkins%2E+Time+series+analysis%2C+forecasting+and+control%2E+1970%2E)
6. Lim Bryan and Zohren Stefan. Time-series forecasting with deep learning: a survey. Phil. Trans. R. Soc. A, 2021. doi: 10.1098/rsta.2020.0209. — [doi:10.1098/rsta.2020.0209](https://doi.org/10.1098/rsta.2020.0209) [find](https://scholar.google.com/scholar?q=Lim+Bryan+and+Zohren+Stefan%2E+Time%2Dseries+forecasting+with+deep+learning%3A+a+survey%2E+Phil%2E+Trans%2E+R%2E+Soc%2E+A%2C+2021%2E+doi%3A+10%2E1098%2Frsta%2E2020%2E0209%2E+%E2%80%94+%5Bdoi%3A10%2E1098%2Frsta%2E2020%2E0209%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1098%2Frst)
7. Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, et al. Spectral temporal graph neural network for multivariate time-series forecasting. Advances in neural information processing systems, 33:17766–17778, 2020. [find](https://scholar.google.com/scholar?q=Defu+Cao%2C+Yujing+Wang%2C+Juanyong+Duan%2C+Ce+Zhang%2C+Xia+Zhu%2C+Congrui+Huang%2C+Yunhai+Tong%2C+Bixiong+Xu%2C+Jing+Bai%2C+Jie+Tong%2C+et+al%2E+Spectral+temporal+graph+neural+network+for+multivariate+time%2Dseries+forecast)
8. Weiqi Chen, Wenwei Wang, Bingqing Peng, Qingsong Wen, Tian Zhou, and Liang Sun. Learning to rotate: Quaternion transformer for complicated periodical time series forecasting. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 146–156, 2022. [find](https://scholar.google.com/scholar?q=Weiqi+Chen%2C+Wenwei+Wang%2C+Bingqing+Peng%2C+Qingsong+Wen%2C+Tian+Zhou%2C+and+Liang+Sun%2E+Learning+to+rotate%3A+Quaternion+transformer+for+complicated+periodical+time+series+forecasting%2E+In+Proceedings+of+the+28t)
9. Yuzhou Chen, Ignacio Segovia, and Yulia R Gel. Z-gcnets: time zigzags at graph convolutional networks for time series forecasting. In International Conference on Machine Learning, pp. 1684–1694. PMLR, 2021. [find](https://scholar.google.com/scholar?q=Yuzhou+Chen%2C+Ignacio+Segovia%2C+and+Yulia+R+Gel%2E+Z%2Dgcnets%3A+time+zigzags+at+graph+convolutional+networks+for+time+series+forecasting%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+1684%E2%80%931694%2E+PMLR%2C)
10. Razvan-Gabriel Cirstea, Chenjuan Guo, Bin Yang, Tung Kieu, Xuanyi Dong, and Shirui Pan. Triformer: Triangular, variable-specific attentions for long sequence multivariate time series forecasting. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22,… — [doi:10.24963/ijcai.2022/277](https://doi.org/10.24963/ijcai.2022/277) [find](https://scholar.google.com/scholar?q=Razvan%2DGabriel+Cirstea%2C+Chenjuan+Guo%2C+Bin+Yang%2C+Tung+Kieu%2C+Xuanyi+Dong%2C+and+Shirui+Pan%2E+Triformer%3A+Triangular%2C+variable%2Dspecific+attentions+for+long+sequence+multivariate+time+series+forecasting%2E+In+P)
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). CoRR, abs/1810.04805, 2018. URL http://arxiv.org/abs/1810.04805. [↗](http://arxiv.org/abs/1810.04805)
12. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). In Internationa…
13. Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 2352–2359, 2021. [find](https://scholar.google.com/scholar?q=Emadeldeen+Eldele%2C+Mohamed+Ragab%2C+Zhenghua+Chen%2C+Min+Wu%2C+Chee+Keong+Kwoh%2C+Xiaoli+Li%2C+and+Cuntai+Guan%2E+Time%2Dseries+representation+learning+via+temporal+and+contextual+contrasting%2E+In+Proceedings+of+the)
14. Eugene F Fama. Efficient capital markets: A review of theory and empirical work. The journal of Finance, 25(2):383–417, 1970. [find](https://scholar.google.com/scholar?q=Eugene+F+Fama%2E+Efficient+capital+markets%3A+A+review+of+theory+and+empirical+work%2E+The+journal+of+Finance%2C+25%282%29%3A383%E2%80%93417%2C+1970%2E)
15. Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In Advances in Neural Information Processing Systems, volume 32, 2019. URL [https://proceedings.neurips.cc/paper/2019/file/53c6de78244e9f528eb3e1cda69699bb-Paper.pdf…](https://proceedings.neurips.cc/paper/2019/file/53c6de78244e9f528eb3e1cda69699bb-Paper.pdf…) [↗](https://proceedings.neurips.cc/paper/2019/file/53c6de78244e9f528eb3e1cda69699bb-Paper.pdf…)
16. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. [Masked autoencoders are scalable vision learners](https://md.research.elamur.ai/free/en/papers/he-2021-masked-autoencoders-scalable-vision-learners/). CoRR, abs/2111.06377, 2021. URL https://arxiv.org/abs/2111.06377. [↗](https://arxiv.org/abs/2111.06377)
17. Dan Hendrycks and Kevin Gimpel. arXiv:1606.08415, 2016. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks+and+Kevin+Gimpel%2E+arXiv%3A1606%2E08415%2C+2016%2E)
18. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Comput., 1997. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter+and+J%2E+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+Comput%2E%2C+1997%2E)
19. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. [Hubert: Self-supervised speech representation learning by masked prediction of hidden units](https://md.research.elamur.ai/free/en/papers/hsu-2021-hubert-self-supervised-speech-representation/). IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021.
20. Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, and Sivanesan Sangeetha. Ammus: A survey of transformer-based pretrained models in natural language processing. arXiv preprint arXiv:2108.05542, 2021. — [arXiv:2108.05542](https://arxiv.org/abs/2108.05542) [find](https://scholar.google.com/scholar?q=Katikapalli+Subramanyam+Kalyan%2C+Ajit+Rajasekharan%2C+and+Sivanesan+Sangeetha%2E+Ammus%3A+A+survey+of+transformer%2Dbased+pretrained+models+in+natural+language+processing%2E+arXiv+preprint+arXiv%3A2108%2E05542%2C+2021)
21. Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto, Xiaofei Wang, et al. A comparative study on transformer vs rnn in speech applications. In IEEE Automatic Speech Recognition and Understanding Worksho… [find](https://scholar.google.com/scholar?q=Shigeki+Karita%2C+Nanxin+Chen%2C+Tomoki+Hayashi%2C+Takaaki+Hori%2C+Hirofumi+Inaguma%2C+Ziyan+Jiang%2C+Masao+Someki%2C+Nelson+Enrique+Yalta+Soplin%2C+Ryuichi+Yamamoto%2C+Xiaofei+Wang%2C+et+al%2E+A+comparative+study+on+trans)
22. Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM Computing Surveys (CSUR), 2021. [find](https://scholar.google.com/scholar?q=Salman+Khan%2C+Muzammal+Naseer%2C+Munawar+Hayat%2C+Syed+Waqas+Zamir%2C+Fahad+Shahbaz+Khan%2C+and+Mubarak+Shah%2E+Transformers+in+vision%3A+A+survey%2E+ACM+Computing+Surveys+%28CSUR%29%2C+2021%2E)
23. Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=cGDAkQo1C0p](https://openreview.net/forum?id=cGDAkQo1C0p). [↗](https://openreview.net/forum?id=cGDAkQo1C0p)
24. Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=UYneFzXSJWh](https://openreview.net/forum?id=UYneFzXSJWh). [↗](https://openreview.net/forum?id=UYneFzXSJWh)
25. Pedro Lara-Benı́tez, Manuel Carranza-Garcı́a, and José C Riquelme. An experimental review on deep learning architectures for time series forecasting. International Journal of Neural Systems, 31(03):2130001, 2021. [find](https://scholar.google.com/scholar?q=Pedro+Lara%2DBen%C4%B1%CC%81tez%2C+Manuel+Carranza%2DGarc%C4%B1%CC%81a%2C+and+Jose%CC%81+C+Riquelme%2E+An+experimental+review+on+deep+learning+architectures+for+time+series+forecasting%2E+International+Journal+of+Neural+Systems%2C+31%2803%29%3A2)
26. Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In Advances in Neural Information Processing Systems, volume 32, 2019. URL [https://proceedings.neurips.cc/pa…](https://proceedings.neurips.cc/pa…) [↗](https://proceedings.neurips.cc/pa…)
27. Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=Shizhan+Liu%2C+Hang+Yu%2C+Cong+Liao%2C+Jianguo+Li%2C+Weiyao+Lin%2C+Alex+X+Liu%2C+and+Schahram+Dustdar%2E+Pyraformer%3A+Low%2Dcomplexity+pyramidal+attention+for+long%2Drange+time+series+modeling+and+forecasting%2E+In+Intern)
28. Barbara Rossi. Exchange rate predictability. Journal of Economic Literature, 51(4):1063–1119, December 2013. doi: 10.1257/jel.51.4.1063. URL [https://www.aeaweb.org/articles?id=10.1257/jel.51.4.1063](https://www.aeaweb.org/articles?id=10.1257/jel.51.4.1063). — [doi:10.1257/jel.51.4.1063](https://doi.org/10.1257/jel.51.4.1063) [↗](https://www.aeaweb.org/articles?id=10.1257/jel.51.4.1063)
29. David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1181– 1191, 2020. [find](https://scholar.google.com/scholar?q=David+Salinas%2C+Valentin+Flunkert%2C+Jan+Gasthaus%2C+and+Tim+Januschowski%2E+Deepar%3A+Probabilistic+forecasting+with+autoregressive+recurrent+networks%2E+International+Journal+of+Forecasting%2C+36%283%29%3A1181%E2%80%93+1191%2C+)
30. Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5149–5152, 2012. doi: 10.1109/ICASSP.2012.6289079. — [doi:10.1109/icassp.2012.6289079](https://doi.org/10.1109/icassp.2012.6289079) [find](https://scholar.google.com/scholar?q=Mike+Schuster+and+Kaisuke+Nakajima%2E+Japanese+and+korean+voice+search%2E+In+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+pp%2E+5149%E2%80%935152%2C+2012%2E+doi%3A+10%2E1109%2FICASSP%2E201)
31. Gaussian error linear units (gelus). arXiv preprint Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. In International Conference on Learning Representations, 2021. URL [https://openreview.net/forum?id=8qDwejCuCN…](https://openreview.net/forum?id=8qDwejCuCN…) [↗](https://openreview.net/forum?id=8qDwejCuCN…)
32. José F Torres, Dalil Hadjout, Abderrazak Sebaa, Francisco Martı́nez-Álvarez, and Alicia Troncoso. Deep learning for time series forecasting: a survey. Big Data, 9(1):3–21, 2021. [find](https://scholar.google.com/scholar?q=Jose%CC%81+F+Torres%2C+Dalil+Hadjout%2C+Abderrazak+Sebaa%2C+Francisco+Mart%C4%B1%CC%81nez%2DA%CC%81lvarez%2C+and+Alicia+Troncoso%2E+Deep+learning+for+time+series+forecasting%3A+a+survey%2E+Big+Data%2C+9%281%29%3A3%E2%80%9321%2C+2021%2E)
33. Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016. — [arXiv:1607.08022](https://arxiv.org/abs/1607.08022) [find](https://scholar.google.com/scholar?q=Dmitry+Ulyanov%2C+Andrea+Vedaldi%2C+and+Victor+Lempitsky%2E+Instance+normalization%3A+The+missing+ingredient+for+fast+stylization%2E+arXiv+preprint+arXiv%3A1607%2E08022%2C+2016%2E+%E2%80%94+%5BarXiv%3A1607%2E08022%5D%28https%3A%2F%2Farxiv%2Eorg)
34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30, 2017. URL [https://proceedings.neurips](https://proceedings.neurips). cc/paper/2017/file/3f5ee243547dee91fbd053c1c… [↗](https://proceedings.neurips)
35. Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey. arXiv preprint arXiv:2202.07125, 2022. — [arXiv:2202.07125](https://arxiv.org/abs/2202.07125) [find](https://scholar.google.com/scholar?q=Qingsong+Wen%2C+Tian+Zhou%2C+Chaoli+Zhang%2C+Weiqi+Chen%2C+Ziqing+Ma%2C+Junchi+Yan%2C+and+Liang+Sun%2E+Transformers+in+time+series%3A+A+survey%2E+arXiv+preprint+arXiv%3A2202%2E07125%2C+2022%2E+%E2%80%94+%5BarXiv%3A2202%2E07125%5D%28https%3A%2F%2Farxi)
36. Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting. In Advances in Neural Information Processing Systems, 2021. [find](https://scholar.google.com/scholar?q=Haixu+Wu%2C+Jiehui+Xu%2C+Jianmin+Wang%2C+and+Mingsheng+Long%2E+Autoformer%3A+Decomposition+transformers+with+Auto%2DCorrelation+for+long%2Dterm+series+forecasting%2E+In+Advances+in+Neural+Information+Processing+Syste)
37. Ling Yang and Shenda Hong. Unsupervised time-series representation learning with iterative bilinear temporal-spectral fusion. In ICML, 2022. [find](https://scholar.google.com/scholar?q=Ling+Yang+and+Shenda+Hong%2E+Unsupervised+time%2Dseries+representation+learning+with+iterative+bilinear+temporal%2Dspectral+fusion%2E+In+ICML%2C+2022%2E)
38. Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8980–8987, 2022. [find](https://scholar.google.com/scholar?q=Zhihan+Yue%2C+Yujing+Wang%2C+Juanyong+Duan%2C+Tianmeng+Yang%2C+Congrui+Huang%2C+Yunhai+Tong%2C+and+Bixiong+Xu%2E+Ts2vec%3A+Towards+universal+representation+of+time+series%2E+In+Proceedings+of+the+AAAI+Conference+on+Art)
39. Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205.13504, 2022. [find](https://scholar.google.com/scholar?q=Ailing+Zeng%2C+Muxi+Chen%2C+Lei+Zhang%2C+and+Qiang+Xu%2E+Are+transformers+effective+for+time+series+forecasting%3F+arXiv+preprint+arXiv%3A2205%2E13504%2C+2022%2E)
40. George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 2114–2124, 2021. [find](https://scholar.google.com/scholar?q=George+Zerveas%2C+Srideepika+Jayaraman%2C+Dhaval+Patel%2C+Anuradha+Bhamidipaty%2C+and+Carsten+Eickhoff%2E+A+transformer%2Dbased+framework+for+multivariate+time+series+representation+learning%2E+In+Proceedings+of+th)
41. Yi Zheng, Qi Liu, Enhong Chen, Yong Ge, and J Leon Zhao. Time series classification using multi-channels deep convolutional neural networks. In International conference on web-age information management, pp. 298–310. Springer, 2014. [find](https://scholar.google.com/scholar?q=Yi+Zheng%2C+Qi+Liu%2C+Enhong+Chen%2C+Yong+Ge%2C+and+J+Leon+Zhao%2E+Time+series+classification+using+multi%2Dchannels+deep+convolutional+neural+networks%2E+In+International+conference+on+web%2Dage+information+manageme)
42. Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, volume 35, pp. 11106–11115, 2021. [find](https://scholar.google.com/scholar?q=Haoyi+Zhou%2C+Shanghang+Zhang%2C+Jieqi+Peng%2C+Shuai+Zhang%2C+Jianxin+Li%2C+Hui+Xiong%2C+and+Wancai+Zhang%2E+Informer%3A+Beyond+efficient+transformer+for+long+sequence+time%2Dseries+forecasting%2E+In+The+Thirty%2DFifth+AAA)
43. Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning, 2022. [find](https://scholar.google.com/scholar?q=Tian+Zhou%2C+Ziqing+Ma%2C+Qingsong+Wen%2C+Xue+Wang%2C+Liang+Sun%2C+and+Rong+Jin%2E+FEDformer%3A+Frequency+enhanced+decomposed+transformer+for+long%2Dterm+series+forecasting%2E+In+Proc%2E+39th+International+Conference+on+)

*Updated 2 November 2026*