Timer (Large Time Series Model)

This repo provides official code, datasets and checkpoints for Timer: Generative Pre-trained Transformers Are Large Time Series Models. [Poster], [Slides].

Updates

🚩 News (2024.6) Pre-training dataset (UTSD) is available in HuggingFace. Dataloader is also contained.

🚩 News (2024.5) Accepted by ICML 2024, a camera-ready version of 31 pages.

🚩 News (2024.4) The pre-training scale has been extended, enabling zero-shot forecasting.

🚩 News (2024.2) Releasing model checkpoints and code for adaptation.

Introduction

Time Series Transformer (Timer) is a Generative Pre-trained Transformer for general time series analysis. You can visit our Homepage for a more detailed introduction.

Datasets

We curate Unified Time Series Datasets (UTSD) comprised of 1B time points and 4 volumes to facilitate the research on large time series models and pre-training.

Our dataset is released in HuggingFace to facilitate the research of large models and pre-training in the field of time series.

Usage

You can access and load UTSD in the style of TSLib based on the following:

# huggingface-cli login
# export HF_ENDPOINT=https://hf-mirror.com 

python ./scripts/UTSD/download_dataset.py

# dataloader
python ./scripts/UTSD/utsdataset.py

Tasks

Forecasting: We provide all scripts as well as datasets for few-shot forecasting in this repo.

Imputation: We propose segment-level imputation, which is more challenging than point-level imputation.

Anomaly Detection: We provide new benchmarks of predictive anomaly detection on UCR Anomaly Archive.

We provide detailed README files illustrating each task under the folder ./scripts/.

Code for Fine-tuning

1. Install Pytorch and necessary dependencies.

pip install -r requirements.txt

2. Put downstream datasets from Google Drive and Tsinghua Cloud under the folder ./dataset/.

3. Put the checkpoint from Google Drive and Tsinghua Cloud under the folder ./checkpoints/.

4. Train and evaluate the model. We provide the above tasks under the folder ./scripts/.

# forecasting
bash ./scripts/forecast/ECL.sh

# segement-level imputation
bash ./scripts/imputation/ECL.sh

# anomaly detection
bash ./scripts/anomaly_detection/UCR.sh

Train on Custom Dataset

To fine-tune on your time series dataset, you can try out the following steps:

1. The essense is to reload the customized dataloader and load the pre-trained checkpoint (See ./scripts/ folder).
2. CIDatasetBenchmark/CIAutoRegressionDatasetBenchmark in the data_provider folder can train and evaluate models in direct / iterative multi-step mode.

Approach

Pre-training and Adaptation

To pre-train on heterogeneous time series, we propose single-series sequence (S3), reserving series variations with the unified context length. Further, we convert forecasting, imputation, and anomaly detection into a unified generative task.

Model Architecture

Given the limited exploration of the backbone for large time series models, we extensively evaluate candidate backbones and adopt the decoder-only Transformer with autoregressive generation towards LTSMs.

Performance

Timer achieves state-of-the-art performance in each task and we present the pre-training benefit on few-shot scenarios.

Scalability

By increasing the parameters and pre-training scale, Timer achieves notable performance improvement: 0.231 $\to$ 0.138 (−40.3%), surpassing the previous state-of-the-art deep forecasters.

Flexible Sequence Length

The decoder-only architecture provides the flexibility to accommodate time series of different lookback and forecast lengths.

Benchmark

Given the significant value to researchers and practitioners, we provide a summary of several concurrent Large Time Series Models:

• MOMENT is trained on large scale by masking modeling. It can be applied to zero-shot forecasting by concatenating lookback series with a mask with the length to be predicted.
• Chronos is a probabilistic point-level forecaster developed by Amazon. Chronos-S1 samples one prediction trajectory and Chronos-S20 uses the mean of sampled 20 trajectories.
• TimesFM from Google is trained on 100B time points. We use the official checkpoint from HuggingFace. It supports dynamic input and output prediction lengths.
• Moiria is developed by Saleforce, exceling at multivariate time series. It has three different checkpoints, labeled as Moiria-S, Moiria-M, and Moiria-L.
• Timer: We evaluate three versions: Timer-1B is pre-trained on UTSD; Timer-16B is pre-trained on UTSD and Buildings900K; and Timer-28B is pre-trained on UTSD and LOTSA.

We also establish the first zero-shot benchmark to measure LTSMs as a general-purpose forecaster.

It should be noticed that the zero-shot performance of concurrent Large Time Series Models is still lagged behind large models based on few-shot fine-tuning or end-to-end training (similar to the challenges GPT-3 faced in 2020).

Citation

If you find this repo helpful, please cite our paper.

@inproceedings{liutimer,
  title={Timer: Generative Pre-trained Transformers Are Large Time Series Models},
  author={Liu, Yong and Zhang, Haoran and Li, Chenyu and Huang, Xiangdong and Wang, Jianmin and Long, Mingsheng},
  booktitle={Forty-first International Conference on Machine Learning}
}

Acknowledgement

We appreciate the following GitHub repos a lot for their valuable code and efforts.

• Time-Series-Library (https://github.com/thuml/Time-Series-Library)
• LOTSA (https://huggingface.co/datasets/Salesforce/lotsa_data)
• UCR Anomaly Archive (https://arxiv.org/abs/2009.13807)

Contributors

If you have any questions or want to use the code, feel free to contact:

• Yong Liu (liuyong21@mails.tsinghua.edu.cn)
• Haoran Zhang (zhang-hr24@mails.tsinghua.edu.cn)
• Chenyu Li (lichenyu20@mails.tsinghua.edu.cn)
• Guo Qin (qinguo24@mails.tsinghua.edu.cn)