Engineering {shiny} with {golem} - WhyR

Workshop at WhyR 2020

"Engineering {shiny} with {golem}"

Transcript

1. Engineering {shiny} with {golem} whyR 2020 2020-09-09 Colin Fay - ThinkR Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 1 / 84
2. Today's menu Program Part 00 & 01: Introduction Part 02: Developing a golem app Part 03: Testing & Sending to prod Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 2 / 84
3. PART 00 - INTRODUCTION Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 3 / 84
4. $ whoami Colin FAY Data Scientist & R-Hacker at ThinkR, a french company focused on Data Science & R. Hyperactive open source developer, lead developer of the {golem} project. https://thinkr.fr https://rtask.thinkr.fr https://twitter.com/_colinfay https://github.com/colinfay https://colinfay.me Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 4 / 84
5. ThinkR Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 5 / 84
6. Data Science engineering, focused on R. Training Software Engineering R in production Consulting ThinkR Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 6 / 84
7. Logistics https://whyr.slack.com/ - #workshop-shiny-advanced @_ColinFay @whyRconf https://2020.whyr.pl/#Code%20of%20Conduct Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 7 / 84
8. PART 01 - ABOUT GOLEM Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 8 / 84
9. Here comes {golem} {golem} is an R package that contains a framework for building production-ready Shiny Applications. Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 9 / 84
10. Here comes {golem} golem <- cranlogs::cran_downloads("golem", from = "2019-08-01") sum(golem$count) [1] 36787 Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 10 / 84
11. Why using {golem}? Automate the boring stuff repetitive tasks Work with reliable tools Gain time developing Simplify deployment Standardize team work About {golem} at ThinkR: Internal need, used on a daily basis Need reliable tooling for deploying to our clients' environments Build and share good practices globally Promote R & Shiny in production Why {golem}? Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 11 / 84
12. {golem} central philosophy Shiny App As a Package What's a "prod-ready" Shiny App? Has meta data (DESCRIPTION) Divided in functions (R/) Tested (tests/) With dependencies (NAMESPACE) Documented (man/ & vignettes) Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 12 / 84
13. {golem} central philosophy Shiny App As a Package The plus side: everything you know about package development works with {golem}. Notably: Documentation Testing Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 13 / 84
14. Understanding {golem} Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 14 / 84
15. ⚠ Warning ⚠ packageVersion("golem") [1] '0.3.0' remotes::install_github( "thinkr-open/golem" ) Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 15 / 84
16. Create a {golem} 02:00 Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 16 / 84
17. Understanding {golem} fs::dir_tree("golex") golex ├── DESCRIPTION ├── NAMESPACE ├── R │ ├── app_config.R │ ├── app_server.R │ ├── app_ui.R │ └── run_app.R ├── dev │ ├── 01_start.R │ ├── 02_dev.R │ ├── 03_deploy.R │ └── run_dev.R ├── inst │ ├── app │ │ └── www │ │ └── favicon.ico │ └── golem-config.yml └── man └── run_app.Rd Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 17 / 84
18. Understanding {golem} Standard package files (i.e. not {golem}-specific): DESCRIPTION: meta-data NAMESPACE: exported functions + functions from other R/: functions (everything in {golem} is a function) man/: documentation Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 18 / 84
19. app_server.R #' The application server-side #' #' @param input,output,session Internal parameters for {shiny}. #' DO NOT REMOVE. #' @import shiny #' @noRd app_server <- function( input, output, session ) { # Your application server logic } server logic can be thought of as a drop in replacement of server.R series of callModule() / module_server() (if used) Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 19 / 84
20. app_ui.R #' @param request Internal parameter for `{shiny}`. #' DO NOT REMOVE. #' @import shiny #' @noRd app_ui <- function(request) { tagList( # Leave this function for adding external resources golem_add_external_resources(), # Your application UI logic fluidPage( h1("golex") ) ) } UI counterpart put UI content after the # Your application UI logic line Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 20 / 84
21. app_ui.R golem_add_external_resources <- function(){ add_resource_path( 'www', app_sys('app/www') ) tags$head( favicon(), bundle_resources( path = app_sys('app/www'), app_title = 'golex' ) # Add here other external resources # for example, you can add shinyalert::useShinyalert() ) } Used to add external resources Will integrate CSS and JS in the app Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 21 / 84
22. app_config.R #' Access files in the current app #' #' NOTE: If you manually change your package name in the DESCRIPTION, #' don't forget to change it here too, and in the config file. #' For a safer name change mechanism, use the `golem::set_golem_name()` function. #' #' @param ... character vectors, specifying subdirectory and file(s) #' within your package. The default, none, returns the root of the app. #' #' @noRd app_sys <- function(...){ system.file(..., package = "golex") } app_sys("x") will refer to inst/x Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 22 / 84
23. app_config.R get_golem_config <- function( value, config = Sys.getenv("R_CONFIG_ACTIVE", "default"), use_parent = TRUE ){ config::get( value = value, config = config, # Modify this if your config file is somewhere else: file = app_sys("golem-config.yml"), use_parent = use_parent ) } Retrieves elements from inst/golem-config.yml Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 23 / 84
24. golem-config.yml golem_version: 0.0.0.9000 app_prod: no production: app_prod: yes dev: Uses the {config} format Can be safely ignored if you don't feel like you need it Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 24 / 84
25. run_app.R #' @param ... arguments to pass to golem_opts #' @inheritParams shiny::shinyApp #' #' @export #' @importFrom shiny shinyApp #' @importFrom golem with_golem_options run_app <- function( onStart = NULL, options = list(), enableBookmarking = NULL, ... ) { with_golem_options( app = shinyApp( ui = app_ui, server = app_server, onStart = onStart, options = options, enableBookmarking = enableBookmarking ), golem_opts = list(...) ) } Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 25 / 84
26. run_app.R About with_golem_options Allows to pass arguments to run_app() will later be callable with golem::get_golem_options() Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 26 / 84
27. Understanding {golem} inst/app/www/ Host external files, notably the one created with: golem::add_css_file() golem::add_js_file() golem::add_js_handler() golem::add_js_input_binding() golem::add_js_output_binding() golem::use_favicon() (More on the later...) Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 27 / 84
28. About the dev/ folder Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 28 / 84
29. About the dev/ folder fs::dir_tree("golex/dev") golex/dev ├── 01_start.R ├── 02_dev.R ├── 03_deploy.R └── run_dev.R Four files that bundle the golem workflow: 01_start.R: run once at the beginning of the project 02_dev.R: day to day development 03_deploy.R: to use before sending to prod run_dev.R: to relaunch your app during development Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 29 / 84
30. 01_start.R Fill the DESCRIPTION file golem::fill_desc( pkg_name = "golex", # The Name of the package containing the App pkg_title = "PKG_TITLE", # The Title of the package containing the App pkg_description = "PKG_DESC.", # The Description of the package containing the App author_first_name = "AUTHOR_FIRST", # Your First Name author_last_name = "AUTHOR_LAST", # Your Last Name author_email = "AUTHOR@MAIL.COM", # Your Email repo_url = NULL # The (optional) URL of the GitHub Repo ) 04:00 Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 30 / 84
31. 01_start.R To be launched for setting elements: golem::set_golem_options() Fills the yaml file golem::use_recommended_tests() Sets common dependencies Use golem::set_golem_name() to globally change your app name Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 31 / 84
32. 01_start.R {usethis} commonly used calls usethis::use_mit_license( name = "Golem User" ) # You can set another license here usethis::use_readme_rmd( open = FALSE ) usethis::use_code_of_conduct() usethis::use_lifecycle_badge( "Experimental" ) usethis::use_news_md( open = FALSE ) usethis::use_git() Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 32 / 84
33. 01_start.R golem::use_recommended_deps() golem::use_favicon() golem::use_utils_ui() & golem::use_utils_server() Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 33 / 84
34. PART 02 - DEV Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 34 / 84
35. STRUCTURING YOUR APPLICATION Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 35 / 84
36. Application logic vs business logic Application logic: the things that make your Shiny app interactive Business logic: core algorithms and functions that make your application specific to your area of work Keep them separated! Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 36 / 84
37. Structuring your app Naming convention: app_*: global infrastructure functions fct_*: business logic functions mod_*: file with ONE module (ui + server) utils_*: cross module utilitarian functions *_ui_* / *_server_*: relates to UI and SERVER Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 37 / 84
38. : R/connect.R : R/summary.R : R/plot.R : R/odbc.R Why bother? Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 38 / 84
39. : R/connect.R : R/summary.R : R/plot.R : R/odbc.R : R/fct_connect.R : R/mod_summary.R : R/utils_plot.R : R/fct_odbc.R Why bother? Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 39 / 84
40. Why bother? Separation of business & app logic is crucial: Easier to work on functions when they are not inside the app (you don't need to relaunch the app every time) You can test your business logic with classical package testing tools You can use the business logic functions outside the app Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 40 / 84
41. 02_dev.R golem::add_fct( "helpers" ) golem::add_utils( "data_ui" ) ✓ File created at R/fct_helpers.R • Go to R/fct_helpers.R ✓ File created at R/utils_data_ui.R • Go to R/utils_data_ui.R Creates fct_* and utils_* files golem::add_module( name = "my_first_module") ✓ File created at R/mod_my_first_module.R • Go to R/mod_my_first_module.R Builds a skeleton for a Shiny Module Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 41 / 84
42. USING SHINY MODULES Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 42 / 84
43. About Shiny Modules "Pieces" of Shiny Apps Always come in pair (UI + Server) Manage the unique IDs necessity of Shiny "Functionnalize" your app Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 43 / 84
44. One million “Validate” buttons library(shiny) ui <- function(request){ fluidPage( sliderInput("choice1", "choice 1", 1, 10, 5), actionButton("validate1", "Validate choice 1"), sliderInput("choice2", "choice 2", 1, 10, 5), actionButton("validate2", "Validate choice 2") ) } server <- function(input, output, session){ observeEvent( input$validate1 , { print(input$choice1) }) observeEvent( input$validate2 , { print(input$choice2) }) } shinyApp(ui, server) Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 44 / 84
45. Why Shiny Modules Functionalizing Shiny elements: name_ui <- function(id){ ns <- NS(id) tagList( sliderInput(ns("choice"), "Choice", 1, 10, 5), actionButton(ns("validate"), "Validate Choice") ) } name_server <- function(id){ moduleServer(id, function(input, output, session) { observeEvent( input$validate , { print(input$choice) }) }) } Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 45 / 84
46. Why Shiny Modules Functionalizing Shiny elements: library(shiny) ui <- function(request){ fluidPage( name_ui("name_ui_1"), name_ui("name_ui_2") ) } server <- function(input, output, session){ name_server("name_ui_1") name_server("name_ui_2") } shinyApp(ui, server) Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 46 / 84
47. Why Shiny Modules id <- "name_ui_1" ns <- NS(id) ns("choice") [1] "name_ui_1-choice" name_ui <- function(id, butname){ ns <- NS(id) tagList( actionButton(ns("validate"), butname) ) } name_ui("name_ui_1", "Validate Choice") name_ui("name_ui_2", "Validate Choice, again") <button id="name_ui_1-validate" type="button" class="btn btn-default action- button">Validate Choice</button> <button id="name_ui_2-validate" type="button" class="btn btn-default action- button">Validate Choice, again</button> Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 47 / 84
48. 02_dev.R golem::add_module( name = "my_first_module") Builds a skeleton for a Shiny Module Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 48 / 84
49. 02_dev.R #' my_first_module UI Function #' #' @description A shiny Module. #' #' @param id,input,output,session Internal parameters for {shiny}. #' #' @noRd #' #' @importFrom shiny NS tagList mod_my_first_module_ui <- function(id){ ns <- NS(id) tagList( ) } Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 49 / 84
50. 02_dev.R #' my_first_module Server Functions #' #' @noRd mod_my_first_module_server <- function(id){ moduleServer( id, function(input, output, session){ ns <- session$ns }) } ## To be copied in the UI # mod_my_first_module_ui("my_first_module_ui_1") ## To be copied in the server # mod_my_first_module_server("my_first_module_ui_1") Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 50 / 84
51. ⚠ DON'T FORGET THE NS() IN THE UI ⚠ Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 51 / 84
52. Your turn Create one shiny module with one slider one output that display the value of this slider 00:00 Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 52 / 84
53. UNDERSTANDING NAMESPACE & DEPENDENCIES Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 53 / 84
54. About the NAMESPACE file One of the most important files of your package ⚠ NEVER EDIT BY HAND ⚠ Describes how your package interacts with R, and with other packages Lists functions that are exported (from your app) and imported (from other packages) Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 54 / 84
55. What's a dependency? Your app needs external functions (at least from {shiny}) The DESCRIPTION file contains the package dependencies Are added to the DESCRIPTION with: usethis::use_package("attempt") Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 55 / 84
56. What's a dependency? You also need to add tags on top of each functions that specify what deps are imported Either with @import (a whole package) and @importFrom (a specific function). golem built modules, by default, import elements from {shiny}: #' @importFrom shiny NS tagList Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 56 / 84
57. Do this for EACH function #' @import magrittr #' @importFrom stats na.omit mean_no_na <- function(x){ x <- x %>% na.omit() sum(x)/length(x) } You can use import or importFrom. The better is to use importFrom, for preventing namespace conflict. Add to EACH function. It will take some time, but it's better on the long run. Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 57 / 84
58. ABOUT DATA Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 58 / 84
59. Adding datasets You might need internal data inside your app Call usethis::use_data_raw from dev/02_dev.R ## Add internal datasets ---- ## If you have data in your package usethis::use_data_raw( name = "dataset" ) Creates data-raw/dataset.R Inside this file: ## code to prepare `dataset` dataset goes here usethis::use_data("dataset") Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 59 / 84
60. Adding datasets ## code to prepare `dataset` dataset goes here library(tidyverse) my_app_dataset <- read.csv("bla/bla/bla.csv") my_app_dataset <- my_app_dataset %>% filter(this == "that") %>% arrange(on_that) %>% select( -contains("this") ) usethis::use_data(my_app_dataset) Now available as my_app_dataset inside your app. Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 60 / 84
61. External data (database) If your app is large or changes frequently, it's better to use a database as a backend. Choice Update Size Package data Never to very rare Low to medium Reading files Uploaded by Users Preferably low External DataBase Never to Streaming Low to Big Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 61 / 84
62. PART 03 - INCLUDING EXTERNAL FILES Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 62 / 84
63. golem_add_external_resources() Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 63 / 84
64. golem_add_external_resources() In app_ui.R, the golem_add_external_resources() functions add to the app every .css and .js file contained in inst/app/www golem_add_external_resources <- function(){ add_resource_path( 'www', app_sys('app/www') ) tags$head( favicon(), bundle_resources( path = app_sys('app/www'), app_title = 'golex' ) # Add here other external resources # for example, you can add shinyalert::useShinyalert() ) } Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 64 / 84
65. Use external resources golem::use_external_js_file() golem::use_external_css_file() golem::use_external_html_template() golem::use_external_file() Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 65 / 84
66. Your turn In your app, create: One h1 One h2 One fluidRow Inside this fluidRow, insert an h3 Use the CSS located online ☝ The link to the CSS will be pasted in the conference chat ☝ 05:00 Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 66 / 84
67. PART 04 TEST AND SEND Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 67 / 84
68. Everything which is not tested will, at last, break. Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 68 / 84
69. Using unit tests Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 69 / 84
70. Why automate code testing? To save time! Work with others Transfer the project Long term stability Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 70 / 84
71. What do we test Focus on business logic testing: it's more important to test that the algorithm is still accurate than testing the UI As we've separated businness logic from application logic, we use package development testing tools We'll use the {testthat} Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 71 / 84
72. 01_start.R ## Init Testing Infrastructure ---- ## Create a template for tests golem::use_recommended_tests() Recommended tests 02_dev.R ## Tests ---- ## Add one line by test you want to create usethis::use_test( "app" ) Add custom test Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 72 / 84
73. Safely collaborate & change elements on a project Making changes should be pain-free Bugs should be detected quickly New collaborators should be able to integrate a team smoothly Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 73 / 84
74. Safely serve application Your users should not be your unit tests Serving application cost money (corollary) You shouldn't spend 1 million bucks on AWS (Jeff Bezos is rich enough) You shouldn't DoS your own server Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 74 / 84
75. What the users see What the users interact with General front-end/design What - User Interface Your time is limited, so if you have to choose don't focus too much on testint the UI only Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 75 / 84
76. Core algorithms that make your app "unique" Business knowledge What your users rely on What - Business logic Try to test business logic as extensively as possible Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 76 / 84
77. How much CPU & RAM does your application need Bad estimate will lead to slow application performances If the app needs to scale, it's crucial to kow it upfront What - Application Load Poor app performances lead to bad UX, and potentially cost Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 77 / 84
78. -> Leverage standard testing frameworks test_that("The meaning of life is 42", { expect_equal( meaning_of_life(), 42 ) }) Shiny App as a package Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 78 / 84
79. UI regressions {shinytests} Test visual regression of your application puppeteer Command line tool to mock a web session, in NodeJS {crrry} R tool to drive a {shiny} session Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 79 / 84
80. Testing the app load {shinyloadtest} : native R package + Cli to record and replay load tests {dockerstats} : get Docker stats inside R {crrry} + {dockerstats} : replay session and watch the Docker stats Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 80 / 84
81. SEND TO PRODUCTION Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 81 / 84
82. ## RStudio ---- ## If you want to deploy on RStudio related platforms golem::add_rstudioconnect_file() golem::add_shinyappsio_file() golem::add_shinyserver_file() ## Docker ---- ## If you want to deploy via a generic Dockerfile golem::add_dockerfile() ## If you want to deploy to ShinyProxy golem::add_dockerfile_shinyproxy() ## If you want to deploy to Heroku golem::add_dockerfile_heroku() Send to prod Once everything is tested, you can send to prod using {golem} Deploy on a server, or build as a tar.gz with pkgbuild::build() Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 82 / 84
83. engineering-shiny.org Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 83 / 84
84. Thx! Questions? Colin Fay Online colin@thinkr.fr http://twitter.com/_colinfay http://twitter.com/thinkr_fr https://github.com/ColinFay https://thinkr.fr/ https://rtask.thinkr.fr/ https://colinfay.me/ Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 84 / 84