-
Notifications
You must be signed in to change notification settings - Fork 7
/
A-outlier-detection.qmd
288 lines (187 loc) · 10.6 KB
/
A-outlier-detection.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
# Outlier detection and external regressors {.unnumbered #a-out-reg-ext}
## In this chapter
The following sections describe
- how to generate useful external regressors for improving seasonal adjustment or reg-arima modelling
- JDemetra+ solutions for outlier detection and in a time series.
These routines can be used stand alone or as part of a seasonal adjustment process. They can be accessed via the [Graphical User Interface (GUI)](#t-gui-overview) or [R packages]((#t-r-packs).
How to use the generated regressors, or any user-defined variable, in a seasonal adjustment or reg-arima modelling process is discussed in the [pre-treatment](#sa-pre-t) chapter for classic SA and [SA of High-frequency](#a-sa-hf) chapter for infra-monthly data. There you will also find out how to fix the corresponding coefficients and how to allocate the effects to the selected component.
The external regressors described exclude calendar correction which is detailed [here](#a-cal)
<!-- ## Tools Map -->
<!-- ### External regressors using R packages vs GUI -->
<!-- up coming content -->
<!-- ### Outlier detection using R packages vs GUI -->
<!-- up coming content -->
## Generating external regressors
### Outliers
#### Types
The following outliers are available for automatic detection
ADD:
- specifics for HF data (wo outiler)
#### Pre-specifying outliers
Outliers are well-defined types of auxiliary variables, therefore when they are used (reg-arima or tramo modelling) they don't need to be explicitly generated beforehand. Pre-specifying outliers is detailed in chapters on [pre-treatment in SA](#sa-pre-t) and [SA of High-frequency data](#a-sa-hf).
#### Generating regressors for outliers
Nevertheless, explicit regressors corresponding to outliers can be generated with `rjd3toolkit` functions for independent use. Further details rjd3toolkit help pages.
```{r, echo = TRUE, eval = FALSE}
# Outliers in February 2002, for monthly data
library("rjd3toolkit")
ao <- ao_variable(frequency = 12, c(2000, 1), length = 12 * 4, date = "2002-02-01")
ls <- ls_variable(12, c(2000, 1), length = 12 * 4, date = "2002-02-01")
tc <- tc_variable(12, c(2000, 1), length = 12 * 4, date = "2002-02-01")
so <- so_variable(12, c(2000, 1), length = 12 * 4, date = "2002-02-01")
```
### Ramps
A ramp effect means a linear increase or decrease in the level of the series over a specified time interval $t_{0}$ to $\ t_{1}$. Ramps can overlap other ramps, additive outliers and level shifts. In seasonal adjustment their effected will be allocated to the trend.
Adding ramps to a seasonal adjustment (or reg-arima/tramo) specification happens in one step in GUI as well as in R, where ramp regressors can nevertheless be independently generated.
#### Adding ramps in GUI
In the specification window
The effect of the ramps is stored in reg_t pre-adjustment series.
#### Adding ramps in R
Use the function `add_ramp`
```{r, echo = TRUE, eval = FALSE}
# create a specification from a default specification
init_spec <- rjd3x13::spec_x13("RSA5c")
# add ramp on year 2012
new_spec <- rjd3toolkit::add_ramp(init_spec, start = "2012-01-01", end = "2012-12-01")
```
#### Generating ramp regressors in R
Use `ramp_variable` function in rjd3toolkit:
```{r, echo = TRUE, eval = FALSE}
?ramp_variable
# Ramp variable from January 2001 to September 2001 for a monthly series
rp <- ramp_variable(frequency = 12, c(2000, 1), length = 12 * 4, range = c(13, 21))
# Or equivalently
rp <- ramp_variable(12, c(2000, 1), length = 12 * 4, range = c("2001-01-01", "2001-09-02"))
plot.ts(rp)
```
More details rjd3toolkit pages.
### Intervention variables
Intervention variables are modelled as any possible sequence of ones and zeros, on which differencing (regular and seasonal) can be applied.
Adding intervention variables to a seasonal adjustment (or reg-arima/tramo) specification happens in one step when using the GUI, whereas two steps are required in R: generating the regressors and the adding them as an user-defined variable.
#### Adding intervention variables in GUI
step 1:
[](All_images/SA_REGA_calendar_user_def_inter_var_1.png)
Step 2:
[](All_images/SA_REGA_calendar_user_def_inter_var_2.png)
#### Generating intervention variables in R
Using `intervention_variable` function in rjd3toolkit
```{r, echo = TRUE, eval = FALSE}
library("rjd3toolkit")
? intervention_variable
iv <- intervention_variable(
frequency = 12, start = c(2000, 1), length = 60,
starts = "2001-01-01", ends = "2001-12-01"
)
iv
plot(iv)
iv <- intervention_variable(12, c(2000, 1), 60,
starts = "2001-01-01", ends = "2001-12-01", delta = 1
)
iv
plot(iv)
iv <- intervention_variable(12, c(2000, 1), 60,
starts = "2001-01-01", ends = "2001-12-01",
delta = 0, seasonaldelta = 1
)
iv
plot(iv)
```
More details rjd3toolkit help pages.
#### Adding intervention variables in R
Intervention variables can be added to a specification like any other external regressor using the `add_usrdefvar`. They also need to be declared in a "context" using the `modelling_context` function.
```{r, echo = TRUE, eval = FALSE}
# creating one or several external regressors (TS objects),
# which will be gathered in one or several groups
iv1 <- intervention_variable(12, c(2000, 1), 60,
starts = "2001-01-01", ends = "2001-12-01"
)
iv2 <- intervention_variable(12, c(2000, 1), 60,
starts = "2001-01-01", ends = "2001-12-01", delta = 1
)
# regressors as a list of two groups (lists) reg1 and reg2
vars <- list(reg1 = list(iv1 = iv1), reg2 = list(iv2 = iv2))
# to use those regressors, input : name=reg1.iv1 and name=reg2.iv2 in add_usrdefvar function
# creating the modelling context
my_context <- modelling_context(variables = vars)
# customize a default specification
init_spec <- rjd3x13::spec_x13("RSA5c")
# regressors have to be added one by one
new_spec <- add_usrdefvar(init_spec, name = "reg1.iv1", regeffect = "Trend")
new_spec <- add_usrdefvar(new_spec, name = "reg2.iv2", regeffect = "Trend", coef = 0.7)
# modelling context is needed for the estimation phase
# raw series
y <- rjd3toolkit::ABS$X0.2.09.10.M
sa_x13 <- rjd3x13::x13(y, new_spec, context = my_context)
```
### Periodic dummies and contrasts
#### Generating regressors in R
dummies :as many time series as type of periods in a year (4,12)
```{r, echo = TRUE, eval = FALSE}
## periodic dummies : add explanations and examples
p <- periodic.dummies(4, c(2000, 1), 60)
head(p)
class(p)
q <- periodic.contrasts(4, c(2000, 1), 60)
q[1:9, ]
```
### Trigonometric variables
Correction for stable seasonality.
#### Generating in R
### User-defined variables {#usr-def-var}
User defined variables are simply time series used as explanatory regressors in the RegARIMA and the TRAMO models. Although JDemetra+ allows the user to indicate any time series as a variable to avoid misleading or
erroneous results, the following rules should be kept:
* User-defined regression variables are used for measuring abnormalities and therefore they should not contain a seasonal pattern.
* JDemetra+ assumes that user-defined regressors are already in an appropriately centred form.
Therefore the mean of each user-defined regressor needs to be subtracted from the regressor or means for each calendar period (month or quarter) need to be subtracted from each of the user-defined regressors.
JDemetra+ considers two kinds of user-defined regression variables:
* **Static variables**, usually imported directly from external software (by drag/drop or copy/paste). The observations for static variables cannot be changed. The only way to update static series is to remove them from the list and to re-import them with the same names.
* **Dynamic variables** that are imported into the *Variables* panel by dragging and dropping series from a browser of the application, available in the *Providers* window. Dynamic variables are automatically updated each time the application is re-opened. Therefore, it is a convenient solution for creating user-defined variables.
#### In GUI
To create a dynamic variable first right-click on the *Variables* node in the *Workspace* window and chose the option **New**.
Next, double click on the newly created *Vars-1* item to display it in the *Results* panel. By default, JDemetra+ uses the conventions Vars_#number to name
the tabs under the *Variables* node.
Then, go to *Providers* window and open your file that contains external variables following the instructions provided [here](../quick-start/datasources.html).
Drag and drop your external regressors from the *Providers* window to the *Vars-1* window.
The original name of the series is
recorded in the *Description* column of the *Variables* window.
In order to rename the series in the *Variables* window, right click on the series and chose **Rename**.
#### In R
## Outlier Detection
### With Reg Arima models
#### Within an SA processing
In a seasonal adjustment estimation or reg-arima modelling outliers are detected by default. This process can be customized by selecting the type of outliers to be taken into account and the critical values to be used for selection. See the relevant chapters on [SA](#sa-pre-t) and [SA of High-frequency data](#a-sa-hf)
#### Stand alone
In version 3;x, R packages rjd3x13 and rjd3tramoseats provide functions for detecting outliers with reg-arima (tramo) algorithms.
Example using `regarima_outliers`in rjd3x13:
```{r, echo = TRUE, eval = FALSE}
library(rjd3x13)
?regarima_outliers
regarima_outliers(rjd3toolkit::ABS$X0.2.09.10.M,
order = c(1, 1, 1), seasonal = c(0, 1, 1),
mean = FALSE,
X = NULL, X.td = NULL,
ao = TRUE, ls = FALSE, tc = TRUE, so = TRUE, cv = 4
)
```
Example wit rjd3tramoseats::tramo_outliers
```{r, echo = TRUE, eval = FALSE}
library(rjd3tramoseats)
?tramo_outliers
tramo_outliers(rjd3toolkit::ABS$X0.2.09.10.M,
order = c(1, 1, 1), seasonal = c(0, 1, 1),
mean = FALSE,
X = NULL, X.td = NULL,
ao = TRUE, ls = FALSE, tc = TRUE, so = TRUE, cv = 4
)
```
#### Specific TERROR tool
Up coming content
### With structural models (BSM)
Up coming content