# Finding Features in Data

When you are presented with longitudinal data, it is useful to summarise the data into a format where you have one row per key. That means one row per unique identifier of the data - if you aren’t sure what this means, see the vignette, “Longitudinal Data Structures”.

So, say for example you wanted to find features in the wages data, which looks like this:

    library(brolgar)
    wages
    #> # A tsibble: 6,402 x 9 [!]
    #> # Key:       id [888]
    #>       id ln_wages    xp   ged xp_since_ged black hispanic high_grade
    #>    <int>    <dbl> <dbl> <int>        <dbl> <int>    <int>      <int>
    #>  1    31     1.49 0.015     1        0.015     0        1          8
    #>  2    31     1.43 0.715     1        0.715     0        1          8
    #>  3    31     1.47 1.73      1        1.73      0        1          8
    #>  4    31     1.75 2.77      1        2.77      0        1          8
    #>  5    31     1.93 3.93      1        3.93      0        1          8
    #>  6    31     1.71 4.95      1        4.95      0        1          8
    #>  7    31     2.09 5.96      1        5.96      0        1          8
    #>  8    31     2.13 6.98      1        6.98      0        1          8
    #>  9    36     1.98 0.315     1        0.315     0        0          9
    #> 10    36     1.80 0.983     1        0.983     0        0          9
    #> # ℹ 6,392 more rows
    #> # ℹ 1 more variable: unemploy_rate <dbl>

You can return a dataset that has one row per key, with say the minimum value for `ln_wages`, for each key:

    #> # A tibble: 888 × 2
    #>       id   min
    #>    <int> <dbl>
    #>  1    31 1.43
    #>  2    36 1.80
    #>  3    53 1.54
    #>  4   122 0.763
    #>  5   134 2.00
    #>  6   145 1.48
    #>  7   155 1.54
    #>  8   173 1.56
    #>  9   206 2.03
    #> 10   207 1.58
    #> # ℹ 878 more rows

This then allows us to summarise these kinds of data, to say for example find the distribution of minimum values:

    library(ggplot2)
    ggplot(wages_min,
           aes(x = min)) +
      geom_density()

We call these summaries `features` of the data.

This vignette discusses how to calculate these features of the data.

# Calculating features

We can calculate `features` of longitudinal data using the `features` function (from `fabletools`, made available in `brolgar`).

`features` works by specifying the data, the variable to summarise, and the feature to calculate:

    features(<DATA>, <VARIABLE>, <FEATURE>)

or with the pipe:

    <DATA> %>% features(<VARIABLE>, <FEATURE>)

As an example, we can calculate a five number summary (minimum, 25th quantile, median, mean, 75th quantile, and maximum) of the data using `feat_five_num`, like so:

    wages_five <- wages %>%
      features(ln_wages, feat_five_num)

    wages_five
    #> # A tibble: 888 × 6
    #>       id   min   q25   med   q75   max
    #>    <int> <dbl> <dbl> <dbl> <dbl> <dbl>
    #>  1    31 1.43   1.48  1.73  2.02  2.13
    #>  2    36 1.80   1.97  2.32  2.59  2.93
    #>  3    53 1.54   1.58  1.71  1.89  3.24
    #>  4   122 0.763  2.10  2.19  2.46  2.92
    #>  5   134 2.00   2.28  2.36  2.79  2.93
    #>  6   145 1.48   1.58  1.77  1.89  2.04
    #>  7   155 1.54   1.83  2.22  2.44  2.64
    #>  8   173 1.56   1.68  2.00  2.05  2.34
    #>  9   206 2.03   2.07  2.30  2.45  2.48
    #> 10   207 1.58   1.87  2.15  2.26  2.66
    #> # ℹ 878 more rows

Here we are taking the `wages` data, piping it to `features`, and then telling it to summarise the `ln_wages` variable, using `feat_five_num`.

There are several handy functions for calculating features of the data that `brolgar` provides. These all start with `feat_`.

You can, for example, find those whose values only increase or decrease with `feat_monotonic`:

    wages_mono <- wages %>%
      features(ln_wages, feat_monotonic)

    wages_mono
    #> # A tibble: 888 × 5
    #>       id increase decrease unvary monotonic
    #>    <int> <lgl>    <lgl>    <lgl>  <lgl>
    #>  1    31 FALSE    FALSE    FALSE  FALSE
    #>  2    36 FALSE    FALSE    FALSE  FALSE
    #>  3    53 FALSE    FALSE    FALSE  FALSE
    #>  4   122 FALSE    FALSE    FALSE  FALSE
    #>  5   134 FALSE    FALSE    FALSE  FALSE
    #>  6   145 FALSE    FALSE    FALSE  FALSE
    #>  7   155 FALSE    FALSE    FALSE  FALSE
    #>  8   173 FALSE    FALSE    FALSE  FALSE
    #>  9   206 TRUE     FALSE    FALSE  TRUE
    #> 10   207 FALSE    FALSE    FALSE  FALSE
    #> # ℹ 878 more rows

These could then be used to identify individuals who only increase like so:

    library(dplyr)
    wages_mono %>%
      filter(increase)
    #> # A tibble: 50 × 5
    #>       id increase decrease unvary monotonic
    #>    <int> <lgl>    <lgl>    <lgl>  <lgl>
    #>  1   206 TRUE     FALSE    FALSE  TRUE
    #>  2   295 TRUE     FALSE    FALSE  TRUE
    #>  3   518 TRUE     FALSE    FALSE  TRUE
    #>  4  1508 TRUE     FALSE    FALSE  TRUE
    #>  5  2178 TRUE     FALSE    FALSE  TRUE
    #>  6  2194 TRUE     FALSE    FALSE  TRUE
    #>  7  2330 TRUE     FALSE    FALSE  TRUE
    #>  8  2456 TRUE     FALSE    FALSE  TRUE
    #>  9  2612 TRUE     FALSE    FALSE  TRUE
    #> 10  2890 TRUE     FALSE    FALSE  TRUE
    #> # ℹ 40 more rows

They could then be joined back to the data

    wages_mono_join <- wages_mono %>%
      filter(increase) %>%
      left_join(wages, by = "id")

    wages_mono_join
    #> # A tibble: 164 × 13
    #>       id increase decrease unvary monotonic ln_wages    xp   ged xp_since_ged
    #>    <int> <lgl>    <lgl>    <lgl>  <lgl>        <dbl> <dbl> <int>        <dbl>
    #>  1   206 TRUE     FALSE    FALSE  TRUE          2.03 1.87      0        0
    #>  2   206 TRUE     FALSE    FALSE  TRUE          2.30 2.81      0        0
    #>  3   206 TRUE     FALSE    FALSE  TRUE          2.48 4.31      0        0
    #>  4   295 TRUE     FALSE    FALSE  TRUE          1.79 2.03      0        0
    #>  5   295 TRUE     FALSE    FALSE  TRUE          1.81 3.12      0        0
    #>  6   295 TRUE     FALSE    FALSE  TRUE          2.11 4.16      0        0
    #>  7   295 TRUE     FALSE    FALSE  TRUE          2.13 5.08      0        0
    #>  8   295 TRUE     FALSE    FALSE  TRUE          2.31 6.58      0        0
    #>  9   518 TRUE     FALSE    FALSE  TRUE          1.27 0.525     1        0.525
    #> 10   518 TRUE     FALSE    FALSE  TRUE          1.61 1.93      1        1.93
    #> # ℹ 154 more rows
    #> # ℹ 4 more variables: black <int>, hispanic <int>, high_grade <int>,
    #> #   unemploy_rate <dbl>

And these could be plotted:

    ggplot(wages_mono_join,
           aes(x = xp,
               y = ln_wages,
               group = id)) +
      geom_line()

To get a sense of the data and where it came from, we could create a plot with `gghighlight` to highlight those that only increase, by using `gghighlight(increase)` - since `increase` is a logical, this tells `gghighlight` to highlight those that are TRUE.

    library(gghighlight)
    wages_mono %>%
      left_join(wages, by = "id") %>%
      ggplot(aes(x = xp,
                 y = ln_wages,
                 group = id)) +
      geom_line() +
      gghighlight(increase)

You can explore the available features, see the function References

# Creating your own Features

To create your own features or summaries to pass to `features`, you provide a named list of functions. For example:

    library(brolgar)
    feat_three <- list(min = min,
                       med = median,
                       max = max)

    feat_three
    #> $min
    #> function (..., na.rm = FALSE)  .Primitive("min")
    #>
    #> $med
    #> function (x, na.rm = FALSE, ...)
    #> UseMethod("median")
    #> <bytecode: 0x15853b8f8>
    #> <environment: namespace:stats>
    #>
    #> $max
    #> function (..., na.rm = FALSE)  .Primitive("max")

These are then passed to `features` like so:

    wages %>%
      features(ln_wages, feat_three)
    #> # A tibble: 888 × 4
    #>       id   min   med   max
    #>    <int> <dbl> <dbl> <dbl>
    #>  1    31 1.43   1.73  2.13
    #>  2    36 1.80   2.32  2.93
    #>  3    53 1.54   1.71  3.24
    #>  4   122 0.763  2.19  2.92
    #>  5   134 2.00   2.36  2.93
    #>  6   145 1.48   1.77  2.04
    #>  7   155 1.54   2.22  2.64
    #>  8   173 1.56   2.00  2.34
    #>  9   206 2.03   2.30  2.48
    #> 10   207 1.58   2.15  2.66
    #> # ℹ 878 more rows

    heights %>%
      features(height_cm, feat_three)
    #> # A tibble: 144 × 4
    #>    country       min   med   max
    #>    <chr>       <dbl> <dbl> <dbl>
    #>  1 Afghanistan  161.  167.  168.
    #>  2 Albania      168.  170.  170.
    #>  3 Algeria      166.  169   171.
    #>  4 Angola       159.  167.  169.
    #>  5 Argentina    167.  168.  174.
    #>  6 Armenia      164.  169.  172.
    #>  7 Australia    170   172.  178.
    #>  8 Austria      162.  167.  179.
    #>  9 Azerbaijan   170.  172.  172.
    #> 10 Bahrain      161.  164.  164
    #> # ℹ 134 more rows

Inside `brolgar`, the features are created with the following syntax:

    feat_five_num <- function(x, ...) {
      list(
        min = b_min(x, ...),
        q25 = b_q25(x, ...),
        med = b_median(x, ...),
        q75 = b_q75(x, ...),
        max = b_max(x, ...)
      )
    }

Here the functions `b_` are functions with a default of `na.rm = TRUE`, and in the cases of quantiles, they use `type = 8`, and `names = FALSE`.

# Accessing sets of features

If you want to run many or all features from a package on your data you can collect them all with `feature_set`. For example:

    library(fabletools)
    feat_brolgar <- feature_set(pkgs = "brolgar")
    length(feat_brolgar)
    #> [1] 6

You could then run these like so:

    wages %>%
      features(ln_wages, feat_brolgar)
    #> # A tibble: 888 × 46
    #>       id min...1 med...2 max...3 min...4 q25...5 med...6 q75...7 max...8 min...9
    #>    <int>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>
    #>  1    31   1.43     1.73    2.13   1.43     1.48    1.73    2.02    2.13   1.43
    #>  2    36   1.80     2.32    2.93   1.80     1.97    2.32    2.59    2.93   1.80
    #>  3    53   1.54     1.71    3.24   1.54     1.58    1.71    1.89    3.24   1.54
    #>  4   122   0.763    2.19    2.92   0.763    2.10    2.19    2.46    2.92   0.763
    #>  5   134   2.00     2.36    2.93   2.00     2.28    2.36    2.79    2.93   2.00
    #>  6   145   1.48     1.77    2.04   1.48     1.58    1.77    1.89    2.04   1.48
    #>  7   155   1.54     2.22    2.64   1.54     1.83    2.22    2.44    2.64   1.54
    #>  8   173   1.56     2.00    2.34   1.56     1.68    2.00    2.05    2.34   1.56
    #>  9   206   2.03     2.30    2.48   2.03     2.07    2.30    2.45    2.48   2.03
    #> 10   207   1.58     2.15    2.66   1.58     1.87    2.15    2.26    2.66   1.58
    #> # ℹ 878 more rows
    #> # ℹ 36 more variables: max...10 <dbl>, range_diff...11 <dbl>, iqr...12 <dbl>,
    #> #   var...13 <dbl>, sd...14 <dbl>, mad...15 <dbl>, iqr...16 <dbl>,
    #> #   min...17 <dbl>, max...18 <dbl>, median <dbl>, mean <dbl>, q25...21 <dbl>,
    #> #   q75...22 <dbl>, range1 <dbl>, range2 <dbl>, range_diff...25 <dbl>,
    #> #   sd...26 <dbl>, var...27 <dbl>, mad...28 <dbl>, iqr...29 <dbl>,
    #> #   increase...30 <dbl>, decrease...31 <dbl>, unvary...32 <dbl>, …

For more information see `?fabletools::feature_set`

# Registering a feature in a package

If you create features in your own package and want to make them accessible with `feature_set`, do the following.

Functions can be registered via `fabletools::register_feature()`. To register features in a package, I create a file called `zzz.R`, and use the `.onLoad(...)` function to set this up on loading the package:

    .onLoad <- function(...) {
      fabletools::register_feature(feat_three_num, c("summary"))
      # ... and as many as you want here!
    }