File size: 3,470 Bytes
dd2cc16
05a82ef
 
9d8ea8d
aca2b5c
 
7ddaa6a
aca2b5c
 
 
 
cfe0f8d
aca2b5c
 
cfe0f8d
 
 
 
dd2cc16
 
6307ba7
aca2b5c
dd2cc16
79dc62b
dd2cc16
bba110f
5251b05
e9982ea
99bde61
bba110f
5d83163
bba110f
35a991e
6307ba7
6a31c14
79303e2
6a31c14
99bde61
 
 
 
 
 
a792e0e
99bde61
 
386b658
456cf7a
 
 
99bde61
6176caa
5251b05
99bde61
 
 
5d83163
d091651
99bde61
 
 
5592854
99bde61
630c277
99bde61
 
dd2cc16
023bcf2
 
 
dd2cc16
 
 
d2a36be
 
 
dd2cc16
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
97a938f
 
dbe7a49
 
 
 
 
 
 
97a938f
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
---
license: apache-2.0
library_name: timer
pipeline_tag: time-series-forecasting
datasets:
- thuml/UTSD
- Salesforce/lotsa_data
metrics:
- mae
- mse
tags:
- time series
- time-series
- forecasting
- foundation models
- pretrained models
- generative models
- time series foundation models
---


# Time-Series Transformer (Timer)

Large time-series model introduced in this [paper](https://arxiv.org/abs/2402.02368) and enhanced with our [further work](https://arxiv.org/abs/2410.04803).

This version is univariate pre-trained on **260B** time points with **84M** parameters, a lightweight generative Transformer for zero-shot point forecasting.

We evaluate the model on the following benchmark: [TSLib Dataset](https://cdn-uploads.huggingface.co/production/uploads/64fbe24a2d20ced4e91de38a/AXhZLVGR8Cnuxe8CVK4Fu.png).

For more information, please see the [GitHub Repo](https://github.com/thuml/Large-Time-Series-Model).

There's indeed room for improvement on this model. We are actively working around it and are glad to see constructive suggestions and noteworthy cases :)

## Quickstart
```
pip install transformers==4.40.1 # Use this version and Python 3.10 for stable compatibility
```

```
import torch
from transformers import AutoModelForCausalLM

# load pretrain model
model = AutoModelForCausalLM.from_pretrained('thuml/timer-base-84m', trust_remote_code=True)

# prepare input
batch_size, lookback_length = 1, 2880
seqs = torch.randn(batch_size, lookback_length)

# generate forecast
prediction_length = 96
output = model.generate(seqs, max_new_tokens=prediction_length)

print(output.shape)
```

A notebook example is also provided [here](https://github.com/thuml/Large-Time-Series-Model/blob/main/examples/quickstart_zero_shot.ipynb). Try it out!

## Specification

* Architecture: Causal Transformer (Decoder-only)
* Pre-training Scale: 260B time points
* Context Length: up to 2880
* Parameter Count: 84M
* Patch Length: 96
* Number of Layers: 8

## Adaptation

For developers interest in fine-tune this model, we provide model checkpoint and code implementation in this [repo](https://github.com/thuml/OpenLTM).

## Acknowledgments

This work was supported by the National Natural Science Foundation of China (62022050 and U2342217), the BNRist Innovation Fund (BNR2024RC01010), and the National Engineering Research Center for Big Data Software. 

The model is mostly built from the Internet public time series dataset, which comes from different research teams and providers. We sincerely thank all individuals and organizations who have contributed the data. Without their generous sharing, this model would not have existed.


## Citation

```
@inproceedings{liutimer,
  title={Timer: Generative Pre-trained Transformers Are Large Time Series Models},
  author={Liu, Yong and Zhang, Haoran and Li, Chenyu and Huang, Xiangdong and Wang, Jianmin and Long, Mingsheng},
  booktitle={Forty-first International Conference on Machine Learning}
}

@article{liu2024timer,
  title={Timer-XL: Long-Context Transformers for Unified Time Series Forecasting},
  author={Liu, Yong and Qin, Guo and Huang, Xiangdong and Wang, Jianmin and Long, Mingsheng},
  journal={arXiv preprint arXiv:2410.04803},
  year={2024}
}
```

## Contact

If you have any questions or want to use the code, feel free to contact:

* Yong Liu ([email protected])
* Guo Qin ([email protected])
 
## License

This model is licensed under the Apache-2.0 License.