A-sa-hf-struct-prov.qmd

# SA: X11 Decomposition {#a-sa-X11 .unnumbered}

left to be modified

-   complete code

-   small tweaks

-   references

to be checked: can be done later

## In this chapter

This chapter focuses on practical implementation of an X11 decomposition using the graphical user interface [GUI](#tools-GUI) and R using [R packages](#t-r-packs) in version 2.x and 3.x. More explanations on X11 algorithm can be found [here](#m-X11).

In recent years X11 has been tailored in JDemetra+ to handle high-frequency (infra-monthly) data, which is described [here](#sa-hf) with more methodological details [here](#m-sa-hf).

The sections below will describe

-   [specifications](#a-sa-X11-default-specs) needed to run X11 

-   generated output

-   [series](#a-sa-X11-out-series)

-   [diagnostics](#a-sa-X11-diags)

-   [final parameters](#a-sa-X11-final-p)

-   [user-defined parameters](#a-sa-X11-user-def)

## Context of use

X11 algorithm is generally the second (decomposition) step in a seasonal adjustment processing with [X-13-Arima](#sa-overview), once a [pre-treatment phase](#sa-pre-t) has been performed. In this case X11 will decompose the linearized series using iteratively different moving averages. The effects of pre-treatment will be [reallocated](#sa-pre-t-realloc) at the end the the relevant components. X11 can also be used [without pre-treatment](#a-pre-t-def-spec-x13), choosing and will decompose the raw series.

## Tools for X11 decomposition {#a-sa-X11-tools-X11}

| Algorithm              | Access in GUI (v2 and v3) | Access in R (v2) | Access in R (v3) |
|----------------------|--------------|-----------------|-----------------|
| X-13 Arima             | ✔️                        | RJDemetra        | rjd3x13          |
| X11 decomposition only | ✔️                        | RJDemetra        | rjd3x13          |


Available frequencies in version 2 and version 3

| Version | GUI and R          |
|---------|--------------------|
| v 2.x   | $p=12, 4, 2$       |
| v 3.x   | $p=12, 6, 4, 3, 2$ |

## Default specifications {#a-sa-X11-default-specs}

The default specifications for X11 must be chosen at the start of the SA processing, one of the options available there is to run a X11 decomposition without pre-treatment.

They are detailed in the [chapter on pre-treatment](#a-pre-t-def-spec-x13).

## Quick Launch

### From GUI

With a workspace open, an SAProcessing created and an open data provider: (link to GUI general process)

-   choose a default specification

-   drop your data and press green arrow

### In R

#### In version 2 using [RJDemetra](https://jdemetra.github.io/rjdemetra/)

```{r, echo = TRUE, eval = FALSE}
library("RJDemetra")
# the input series has to be a Time Series (TS) object
# specification RSA5c including pre-treatment
model_sa_v2 <- x13(raw_series, spec = "RSA5c")
# specification X11 without pre-treatment
model_sa_v2 <- x13(raw_series, spec = "X11")
```

Full documentation of 'RJDemetra::x13' function can be found [here](https://jdemetra.github.io/rjdemetra/reference/x13.html)

The model_sa_v2 R object (list of lists) contains all parameters and results. Its structure is detailed [here](#t-r-packs-rjdv2-structure). It can be printed giving access to selected parameters, series and diagnostics.

```{r, echo = TRUE, eval = FALSE}
print(model_sa_v2)
```

#### In version 3 using [rjd3x13](https://rjdverse.github.io/rjd3x13/)

```{r, echo = TRUE, eval = FALSE}
library("rjd3toolkit")
library("rjd3x13")
# the input series has to be a Time Series (TS) object
model_sa_v3 <- rjd3x13::x13(y_raw, spec = "RSA5")
```

Full documentation of 'rjd3x13::x13' function can be found [here](https://rjdverse.github.io/rjd3x13/reference/x13.html) and of 'rjd3x13::X11' [here](https://rjdverse.github.io/rjd3x13/reference/X11.html).

The model_sa_v3 R object (list of lists) contains all parameters and results. Its structure is detailed [here](#t-r-packs-rjd3x13-structure).

It can be printed giving access to selected parameters, series and diagnostics.

```{r, echo = TRUE, eval = FALSE}
print(model_sa_v3)
```

## Retrieve series {#a-sa-X11-out-series}

### Display in GUI

Final components from the SA Processing are displayed in [Main results](#gui-sa). They contain the re-allocated [pre-adjustment effects](#sa-pre-t-realloc) of outliers (link) or external regressors (link). The final seasonal components contains the calendar effects, if any.

![**Final components in GUI**](All_images/SA_MainResults_Series.png)

(forecasts are added at the end of the series, values in *italic*)

[Detailed results](#m-X11-stages-output) from decomposition are displayed in Decomposition (X11) node.

![**Detailed results**](All_images/RDimage41.png)

The final D Tables contain the re-allocated [pre-adjustment effects](#sa-pre-t-realloc).

(to be checked: v2 vs v3 on pre-treatment effects in D tables)

Output series can be exported out of GUI by two means:

-   generating [output files directly with interactive menus](#t-gui-output)

-   running the cruncher to generate those files as described [here](#t-prod)

### Retrieve in R

In version 2

```{r, echo = TRUE, eval = FALSE}
model_sa <- x13(raw_series, spec = "RSA3") # user's spec choice
# final components
model_sa$final$series
# final forecasts y_f sa_f s_f t_f i_f
model_sa$final$forecasts
```

Detailed X11 tables have to be pre-specified from the [user-defined output list](https://jdemetra.github.io/rjdemetra/reference/user_defined_variables.html).

```{r, echo = TRUE, eval = FALSE}
# display the list of available objects (series, diagnostics, parameters)
user_defined_variables("X13-ARIMA")
# add selected object to estimation
sa_x13_v2 <- RJDemetra::x13(myseries, myspec,
    userdefined = c("decomposition.c20", "decomposition.d1")
)
# retrieve in the user-defined sub-list
sa_x13_v2$user_defined
```

to be modified In version 3

```{r, echo = TRUE, eval = FALSE}
# final components
model_sa$final$series
# final forecasts y_f sa_f s_f t_f i_f
model_sa$final$forecasts
# from user defined output
```

## Diagnostics {#a-sa-X11-diags}

X11 produces the following type diagnostics or quality measures

### SI-ratios

#### Display in GUI

NODE Main Results \> SI-Ratios

![**S-I Ratio**](All_images/SA_MainResults_SI_ratios.png)

For each period (month, quarter) the final value of the seasonal factors (without calendar factors, Table D10) is plotted (blue line). The dots represent $S+I$ or $S*I$ in the multiplicative case, where I= Table D8. The red straight line is the average of the factors over the decomposition (estimation) [span](#sa-pre-t-spans).

In GUI all values cannot be retrieved.

#### Retrieve in R

All the values and the same plot as described above can be generated in R, the span can be customized.

(to be checked the average of the seasonal factors is not recalculated if span modified )

In version 2

```{r, echo = TRUE, eval = FALSE}
# data frame with values
model_sa_v2$decomposition$si_ratio

# customizable plot
plot(mysa2$decomposition)

plot(model_sa, type = "cal-seas-irr", first_date = c(2015, 1))
```

In version 3

```{r, echo = TRUE, eval = FALSE}
# data frame with values
model_sa_v2$decomposition$si_ratio

# customizable plot
plot(mysa2$decomposition)

plot(model_sa, type = "cal-seas-irr", first_date = c(2015, 1))
```

### M-statistics {#a-sa-X11-m-stats}

X11 algorithm provides quality measures of the decomposition called "M statistics" (detailed [here](M-X11-decomposition-M)

-   11 statistics (M1 to M11)

-   2 summary indicators (Q et Q-M2)

-   by design $0<M_x<3$ and acceptance region is $M_x \leq 1$

##### Display in GUI

To display results in GUI, expand NODE

Decomposition(X11) \> Quality Measures \> Summary

Results displayed in red indicate that the test failed.

![Text](All_images/RDimage46.png)

##### Retrieve in R

In version 2

```{r, echo = TRUE, eval = FALSE}
# this code snippet is not self-sufficient
model_sa$decomposition$mstats
```

In version 3

```{r, echo = TRUE, eval = FALSE}
# this code snippet is not self-sufficient
model_sa$decomposition$mstats
```

#### Detailed Quality measures {#a-sa-X11-q-meas}

In GUI all the diagnostics below can be displayed expanding the NODE

Decomposition(X11) \> Quality Measures \> Details

They are detailed in the [X11 method chapter](#m-X11)

In R (to be added): not directly available ?!

### Retrieve final filters {#a-sa-X11-final-p}

The following parameters are automatically chosen by the software as a result of the estimation process. They have no default value but can be set by the user.

-   **Final trend filter**: length of Henderson filter applied for final trend estimation (in the second part of the D step).

-   **Final seasonal filer**: length of final seasonal filter for seasonal component estimation (in the second part of the D step).

#### Display in GUI

Node Decomposition(X11) \> Final Filters

![Text](All_images/SA_X11_GUI_final_filters.png)

#### Retrieve in R

In version 2

```{r, echo = TRUE, eval = FALSE}
library("RJDemetra")
model_sa_v2 <- x13(raw_seriesa, spec = "RSA5c")
model_sa$decomposition$s_filter
model_sa$decomposition$t_filter
```

In version 3

```{r, echo = TRUE, eval = FALSE}
library("rjd3toolkit")
library("rjd3x13")
model_sa_v3 <- rjd3x13::x13(y_raw, spec = "RSA5")
model_sa_v3$result$decomposition$final_seasonal
model_sa_v3$result$decomposition$final_henderson
```

### User-defined parameters {#a-sa-X11-user-def}

The following parameters have default values, which will not be changed in the estimation process. They can be set by the user in a given range of admissible values.

#### General settings

-   **Mode**

    -   Undefined: automatically chosen between Multiplicative and Additive Options available only if no pre-processing:
    -   Additive: $Y=T+S+I$, $SA =Y-S=T+I$
    -   Multiplicative $Y=T*S*I$, $SA =Y/S=T*I$
    -   LogAdditive $Log(Y) = T + S + I$, $SA=exp(T+I)=Y/exp(S)$
    -   PseudoAdditive $Y=T*(S+I-1)$, $SA=T*I$
    


If X11 decomposition comes after a pre-processing, **mode** is set to undefined and will correspond to decomposition choice (link) made in the [pre-treatment](#sa-pre-t-mode): multiplicative if series log- transformed, additive otherwise.

-   **Seasonal component**

Option available only if no pre-processing: - yes (default), decomposition into $S$, $T$, $I$ - no, decomposition into $S$, $T$, $I$

-   **Forecasts horizon**

Length of the forecasts generated by the Reg-Arima model - in months (positive values) - years (negative values) - if set to is set to 0, the X11 procedure does not use any model-based forecasts but the original X11 type forecasts for one year. - default value: -1, thus one year from the Arima model

-   **Backcasts horizon**

Length of the backcasts generated by the Reg-Arima model - in months (positive values) - years (negative values) - default value: 0

##### Irregular correction

-   **LSigma**
    -   sets lower sigma (standard deviation) limit used to down-weight the extreme irregular values in the internal seasonal adjustment iterations
    -   values in $[0,Usigma]$
    -   default value is 1.5
-   **USigma**
    -   sets upper sigma (standard deviation)
    -   values in $[Lsigma,+\infty]$
    -   default value is 2.5
-   **Calendarsigma**

Allows to set different **LSigma** and **USigma** for each period - None (default) - All: standard errors used for the extreme values detection and adjustment computed separately for each calendar month/quarter - Signif: groups determined by Cochran test (check) - Sigmavec: set two customized groups of periods

-   **Excludeforecasts**

    -   ticked: forecasts and backcasts from the Reg-Arima model not used in Irregular Correction
    -   unticked (default): forecasts and backcasts used

##### Seasonality extraction filters choice {#a-X11-seaso-filters}

-   **Seasonal filter**

Specifies which filters will be used to estimate the seasonal factors for the entire series.

-   default value: *MSR* [Moving seasonality ratio](#m-X11-seaso-filters ), automatic choice of final seasonal filter, initial filters are $3\times 3$

-   choices: $3\times 1$, $3\times 3$, $3\times 5$, $3\times 9$, $3\times 15$ or Stable

-   "Stable": constant factor for each calendar period (simple moving average of a all $S+I$ values for each period)

User choices will be applied to final phase D step.

The seasonal filters can be selected for the entire series, or for a particular month or quarter.

-   **Details on seasonal filters**

Sets different seasonal filters by period in order to account for [seasonal heteroskedasticity](#m-X11-hetero)

-   default value: empty, same filter for all periods

##### Trend estimation filters {#a-X11-trend-filters}

-   **Automatic Henderson filter** or user-defined

    -   default: length 13
    -   unticked: user-defined length choice

-   **Henderson filter** length choice

    -   values: odd number in $[3,101]$
    -   default value: 13

Check: will user choice be applied to all steps or only to final phase D step

#### Parameter setting in GUI

All the parameters above can be set with in the [specification box](#gui-sa)

![Text](All_images/SA_X11_GUI_param_setting.png)

Setting details on seasonal filters

![**Seasonnal filters**](All_images/SA_X11_GUI_details_seaso_filter1.png)

Previously set values are displayed for each type of period, here they are all to default MSR choice.

Click on the right top button (show on image)

Another window appears in the top-left corner allowing to chose the filter period by period.

![Text](All_images/SA_X11_GUI_details_seaso_filter2.png)

#### Parameter setting in R packages

In version 2 using RJDemetra

```{r, echo = TRUE, eval = FALSE}
current_sa_model <- x13(raw_series, spec = current_spec)
# Creating a modified specification, customizing all available X11 parameters
modified_spec <- x13_spec(current_sa_model,
    X11.mode = NA,
    X11.seasonalComp = NA,
    X11.fcasts = -2,
    X11.bcasts = -1,
    X11.lsigma = 1.2,
    X11.usigma = 2.8,
    X11.calendarSigma = NA,
    X11.sigmaVector = NA,
    X11.excludeFcasts = NA,
    # filters
    X11.trendAuto = NA,
    X11.trendma = 23,
    X11.seasonalma = "S3X9"
)

# New SA estimation: apply modified_spec
modified_sa_model <- x13(raw_series, modified_spec)
```

In version 3 using rjd3x13

```{r, echo = TRUE, eval = FALSE}
# Creating a modified specification, customizing all available X11 parameters
library("RJDemetra")
model_sa_v2 <- x13(raw_series, spec = "RSA5c")
# Creating a modified specification from the current estimation model
# Customizing all available X11 parameters
modified_spec <- x13_spec(model_sa_v2,
    X11.fcasts = -2,
    X11.bcasts = -1,
    X11.lsigma = 1.2,
    X11.usigma = 2.8,
    X11.calendarSigma = NA,
    X11.sigmaVector = NA,
    X11.excludeFcasts = NA,
    # filters
    X11.trendAuto = NA,
    X11.trendma = 23,
    X11.seasonalma = "S3X9"
)

# New SA estimation: apply modified_spec

modified_sa_model <- x13(raw_series, modified_spec)


# For options available only in X11 mode
modified_spec <- x13_spec(model_sa_v2,
    # X11.mode="?",
    # X11.seasonalComp = "?",
    X11.fcasts = -2
)
```

#### Retrieving Parameters

How to see what parameters have actually been used.

In GUI: just open the [specification box](#sa-window) and navigate the options.

In R, print your model or navigate its elements.