slide-almanac.pdf

Moving Averages and Calendars
Davis Vaughan
@dvaughan32
Software Engineer, RStudio
October 2019


View Slide

Window Functions
Schedules

View Slide

Window What?

View Slide

speakerdeck.com/davisvaughan/slide-almanac

View Slide

function applied
=
mean()

View Slide

function applied
=
mean()
function applied
=
sd()

View Slide

Function applied
is arbitrary

View Slide

Function applied
is arbitrary
rolling regression = lm()

View Slide

Types of windows:
1) Sliding
2) Expanding

View Slide

speakerdeck.com/davisvaughan/slide-almanac Idea from: https://eng.uber.com/forecasting-introduction/

View Slide

Moving averages, rolling regressions…
Idea from: https://eng.uber.com/forecasting-introduction/

View Slide

speakerdeck.com/davisvaughan/slide-almanac Idea from: https://eng.uber.com/forecasting-introduction/

View Slide

Cumulative sums, expanding window regression…
Idea from: https://eng.uber.com/forecasting-introduction/

View Slide

In R?

View Slide

So many attempts:
- zoo::rollapply()
- tibbletime::rollify()
- tsibble::slide() / stretch()
- data.table::frollapply() (2019-10-03)

View Slide

{slide}

View Slide

slide(1:4, ~.x, .before = 2)
3
2
1
[[3]]
2
1
[[2]]
4
3
2
[[4]]
1
[[1]]

View Slide

3
2
1
[[3]]
NULL
[[2]]
4
3
2
[[4]]
NULL
[[1]]
slide(1:4, ~.x, .before = 2, .complete = TRUE)
Ignore partial results

View Slide

slide(1:4, ~.x, .before = 1, .after = 1)
4
3
2
[[3]]
4
3
[[4]]
3
2
1
[[2]]
2
1
[[1]]
Center alignment

View Slide

slide(1:4, ~.x, .before = Inf)
3
2
1
[[3]]
2
1
[[2]]
4
3
2
1
[[4]]
1
[[1]]
Cumulative sliding

View Slide

slide()
slide_dbl()
slide_int()
...
Type Stability

View Slide

sales_vec
[1] 2 4 6 2
slide_dbl(sales_vec, mean, .before = 2)
[1] 2 3 4 4

View Slide

index_vec wday_vec company sales = sales_vec,
index = index_vec,
wday = wday_vec
)
4 2 Wed
2019-09-04
1
3
2
2019-09-03 Tue
6
2019-08-30 Fri
4
2 Thu
2019-08-29
sales wday
index

View Slide

“3 day rolling average?”
4 2 Wed
2019-09-04
1
3
2
2019-09-03 Tue
6
2019-08-30 Fri
4
2 Thu
2019-08-29
sales wday
index

View Slide


3
6
4
roll_day
2
4 2 Wed
2019-09-04
1
3
2
2019-09-03 Tue
6
2019-08-30 Fri
4
2 Thu
2019-08-29
sales wday
index

View Slide

company %
mutate(
roll_row = slide_dbl(sales, mean, .before = 2)
)
roll_row

3
4
4
2

3
6
4
roll_day
2
4 2 Wed
2019-09-04
1
3
2
2019-09-03 Tue
6
2019-08-30 Fri
4
2 Thu
2019-08-29
sales wday
index

View Slide

# Construct a regular index
full_index company,
index = full_seq(index, 1)
)
# Join with original data
company_full_raw full_index,
company
)
# Slide over this, then filter back down
company_three_day %
mutate(
roll_day = slide_dbl(
sales,
mean,
na.rm = TRUE,
.before = 2
)
) %>%
filter(
index %in% company$index
)
Solution?
7 2019-09-04
2
6 6 2019-09-03
2019-09-02
5 NA
4 NA 2019-09-01
1
3
2
2019-08-31
NA
2019-08-30
4
2 2019-08-29
sales index

3
6
4
roll_day
2
4 2 2019-09-04
1
3
2
2019-09-03
6
2019-08-30
4
2 2019-08-29
sales index

company_full_raw
company_three_day

View Slide

# Construct a regular index
full_index company,
index = full_seq(index, 1)
)
# Join with original data
company_full_raw full_index,
company
)
# Slide over this, then filter back down
company_three_day %
mutate(
roll_day = slide_dbl(
sales,
mean,
na.rm = TRUE,
.before = 2
)
) %>%
filter(
index %in% company$index
)
Solution?
7 2019-09-04
2
6 6 2019-09-03
2019-09-02
5 NA
4 NA 2019-09-01
1
3
2
2019-08-31
NA
2019-08-30
4
2 2019-08-29
sales index

3
6
4
roll_day
2
4 2 2019-09-04
1
3
2
2019-09-03
6
2019-08-30
4
2 2019-08-29
sales index

company_full_raw
company_three_day
I
JUST
WANT
A
3
DAY
AVERAGE

View Slide

slide(.x, .f, …)
slide_index(.x, .i, .f, …)

View Slide

slide_index(
.x = wday_vec,
.i = index_vec,
.f = ~.x,
.before = days(2)
)
Tue
[[3]]
Fri
Thu
[[2]]
Wed
Tue
[[4]]
Thu
[[1]]
slide(
.x = wday_vec,
.f = ~.x,
.before = 2
)
Tue
Fri
Thu
[[3]]
Fri
Thu
[[2]]
Wed
Tue
Fri
[[4]]
Thu
[[1]]

View Slide

company %
mutate(
roll_day = slide_index_dbl(sales, index, mean, .before = days(2))
)

3
6
4
roll_day
2
4 2 Wed
2019-09-04
1
3
2
2019-09-03 Tue
6
2019-08-30 Fri
4
2 Thu
2019-08-29
sales wday
index

View Slide


View Slide

“3 business day rolling average?”

View Slide

roll_day

3
4
6
2

3
5
4
roll_bday
2
4 2 Wed
2019-09-04
1
3
2
2019-09-03 Tue
6
2019-08-30 Fri
4
2 Thu
2019-08-29
sales wday
index

3 bday = [Fri, Mon, Tue]
3 day = [Sun, Mon, Tue]

View Slide

calendar company %
mutate(
)
company %
mutate(
roll_bday = slide_index_dbl(sales, index, mean, .before = bdays(2, calendar))
)
Ideally

View Slide

calendar company %
mutate(
)
company %
mutate(
)
Ideally
“knows” about custom holidays and weekends

View Slide

calendar company %
mutate(
)
company %
mutate(
)
Ideally
“knows” about custom holidays and weekends
“adjusts” dates relative to the calendar

View Slide

{almanac}

View Slide

Recurrence rule:
A set of conditions that define a
recurring event, such as a weekend
or holiday.

View Slide

on_labor_day %
recur_on_ymonth(“September”) %>%
recur_on_wday(“Monday”, nth = 1)

View Slide

on_labor_day %
Base frequency of the event

View Slide

on_labor_day %
Recurrence conditions

View Slide

on_labor_day %
sch_in(c("2019-09-02", "2019-09-03"), on_labor_day)
#> [1] TRUE FALSE

View Slide

on_labor_day %
sch_in(c("2019-09-02", "2019-09-03"), on_labor_day)
#> [1] TRUE FALSE
sch_seq("2017-01-01", "2019-12-31", on_labor_day)
#> [1] "2017-09-04" "2018-09-03" "2019-09-02"

View Slide

Schedule:
A collection of recurrence rules,
required dates, and exclusion
dates.

View Slide

on_labor_day %
on_christmas %
recur_on_ymonth(“December”) %>%
recur_on_mday(25)
on_weekends %
recur_on_weekends()
on_weekends_or_holidays %
sch_add_rrule(on_labor_day) %>%
sch_add_rrule(on_christmas) %>%
sch_add_rrule(on_weekends)

View Slide

sch_seq("2019-09-01", "2019-12-31", on_weekends_or_holidays)
#> [1] "2019-09-01" "2019-09-02" "2019-09-07" "2019-09-08" "2019-09-14"
#> [6] "2019-09-15" "2019-09-21" "2019-09-22" "2019-09-28" "2019-09-29"
#> ...
#> [31] "2019-12-14" "2019-12-15" "2019-12-21" "2019-12-22" "2019-12-25"
#> [36] "2019-12-28" "2019-12-29"

View Slide

* These will probably move to their own
hldy_christmas()
hldy_easter()
hldy_thanksgiving()
...
calendar_us_federal()
calendar_us_nyse()
Prebuilt holidays and calendars

View Slide

* These will probably move to their own
hldy_christmas()
hldy_easter()
hldy_thanksgiving()
...
calendar_us_federal()
calendar_us_nyse()
Prebuilt holidays and calendars
Particularly
challenging!

View Slide

# A Monday
labor_day # Find the next business day?
# - Sees labor day, adjust by 1 day
# - Lands on 2019-09-03, done!
sch_adjust(labor_day, on_weekends_or_holidays)
#> [1] "2019-09-03"
# - Sees labor day, adjust by -1 day
# - Lands on 2019-09-01, a Sunday, adjust by -1 day
# - Lands on 2019-08-31, a Saturday, adjust by -1 day
# - Lands on 2019-08-30, done!
sch_adjust(labor_day, on_weekends_or_holidays, -days(1))
#> [1] “2019-08-30"

View Slide

# A Monday
labor_day # Find the next business day?
# - Sees labor day, adjust by 1 day
# - Lands on 2019-09-03, done!
sch_adjust(labor_day, on_weekends_or_holidays)
#> [1] "2019-09-03"
# - Sees labor day, adjust by -1 day
# - Lands on 2019-09-01, a Sunday, adjust by -1 day
# - Lands on 2019-08-31, a Saturday, adjust by -1 day
# - Lands on 2019-08-30, done!
sch_adjust(labor_day, on_weekends_or_holidays, -days(1))
#> [1] “2019-08-30"
This can also be a function

View Slide

Modified following:
Choose the first business day
after x, unless it falls in a
different month, in which case the
first business day before x is
chosen instead.

View Slide

on_15th_and_last %
recur_on_mday(c(15, -1))
payments dates = sch_seq("2019-09-01", "2019-12-31", on_15th_and_last),
wday = wday(dates, label = TRUE)
)
2019-12-31 2019-12-31 Tue
8 Tue
2019-12-15
7 Sun Mon
2019-12-16
6 2019-11-30 Sat Fri
2019-11-29
2019-11-15 Fri 2019-11-15 Fri
5
adj_wday

Mon
Thu
Tue
Mon

2019-09-30
2019-10-15
2019-10-31
adj_dates
2019-09-16
4 Thu
2019-10-31
1
3
2
2019-10-15 Tue
2019-09-30 Mon
Sun
2019-09-15
wday
dates

View Slide

2019-12-31 2019-12-31 Tue
8 Tue
2019-12-15
7 Sun Mon
2019-12-16
6 2019-11-30 Sat Fri
2019-11-29
2019-11-15 Fri 2019-11-15 Fri
5
adj_wday

Mon
Thu
Tue
Mon

2019-09-30
2019-10-15
2019-10-31
adj_dates
2019-09-16
4 Thu
2019-10-31
1
3
2
2019-10-15 Tue
2019-09-30 Mon
Sun
2019-09-15
wday
dates

on_weekends % recur_on_weekends()
payments %>%
mutate(
adj_dates = sch_adjust(dates, on_weekends, adj_modified_following),
adj_wday = wday(adj_dates, label = TRUE)
)

View Slide

friday_before_labor_day # Move forward two business days?
# - Steps forward 1 day to Saturday 2019-08-31
# - Call sch_adjust(), adjusts to Tuesday 2019-09-03
# - Steps forward 1 day to Wednesday 2019-09-04
# - Call sch_adjust(), no adjustment needed
sch_step(
friday_before_labor_day,
n = 2,
schedule = on_weekends_or_holidays
)
#> [1] “2019-09-04"

View Slide

{slide} + {almanac}

View Slide


View Slide

calendar company %
mutate(
)
company %
mutate(
)
Ideally

View Slide

calendar company %
mutate(
)
company %
mutate(
)
Ideally
We can make this with {almanac}

View Slide

calendar company %
mutate(
)
company %
mutate(
)
Ideally
We can make this with {almanac}
This doesn’t exist yet, but would use sch_step()

View Slide

slide_index(.x, .i, .f, …)
slide_between(.x, .i, .starts, .stops, .f, …)

View Slide

company %
mutate(
)
company %
mutate(
starts = index - days(2),
stops = index,
roll_day = slide_between_dbl(sales, index, mean, .starts = starts, .stops = stops)
)

3
6
4
roll_day
2
4 2 Wed
2019-09-04
1
3
2
2019-09-03 Tue
6
2019-08-30 Fri
4
2 Thu
2019-08-29
sales wday
index

View Slide

company %
mutate(
)
company %
mutate(
starts = index - days(2),
stops = index,
roll_day = slide_between_dbl(sales, index, mean, .starts = starts, .stops = stops)
)

3
6
4
roll_day
2
4 2 Wed
2019-09-04
1
3
2
2019-09-03 Tue
6
2019-08-30 Fri
4
2 Thu
2019-08-29
sales wday
index

This is where we solve our problem

View Slide

company %
mutate(
starts = sch_step(index, n = -2, schedule = on_weekends),
stops = index,
roll_bday = slide_between_dbl(sales, index, mean, .starts = starts, .stops = stops)
)
roll_day

3
4
6
2

3
5
4
roll_bday
2
4 2 Wed
2019-09-04
1
3
2
2019-09-03 Tue
6
2019-08-30 Fri
4
2 Thu
2019-08-29
sales wday
index

View Slide


View Slide

In conclusion…

View Slide

{slide} for window functions
slide_index() to roll relative to an index

View Slide

{almanac} to build schedules and adjust dates

View Slide

{slide} + {almanac}
=
Flexible rolling computations!
{almanac} to build schedules and adjust dates

View Slide

Special Thanks
JavaScript: rrule
https://github.com/jakubroztocil/rrule
James Laird-Smith: gs
https://github.com/jameslairdsmith/gs
Jeroen Ooms: V8
https://github.com/jeroen/V8

View Slide

Questions?
{almanac}
GitHub
https://github.com/DavisVaughan/almanac
Website
https://davisvaughan.github.io/almanac
{slide}
GitHub
https://github.com/DavisVaughan/slide
Website
https://davisvaughan.github.io/slide

View Slide

slide-almanac.pdf

slide-almanac.pdf

Davis Vaughan

More Decks by Davis Vaughan

Featured

Transcript