An overview of testing in R

Talk at the AI Testing online meetup

Transcript

1. An overview of testing in R rdev::secure() 2020-09-09 Colin Fay - ThinkR Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 1 / 41
2. $ whoami Colin FAY Data Scientist & R-Hacker at ThinkR, a french company focused on Data Science & R. Hyperactive open source developer, lead developer of the {golem} project. https://thinkr.fr https://rtask.thinkr.fr https://twitter.com/_colinfay https://github.com/colinfay https://colinfay.me Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 2 / 41
3. ThinkR Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 3 / 41
4. Data Science engineering, focused on R.  Training  Software Engineering  R in production  Consulting ThinkR Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 4 / 41
5. An overview of testing in R Securing youR development Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 5 / 41
6. Today's menu A (non-comprehensive) overview of testing technologies in the R world Program Testing in R, why? Testing R packages Benchmark and loadtest Disclaimer I know nothing about AI Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 6 / 41
7. An overview of testing in R Part 0, The Current State of Unit Test Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 7 / 41
8. The two states of unit test Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 8 / 41
9. The two states of unit test Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 9 / 41
10. An overview of testing in R Part 1, Why Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 10 / 41
11. Everything That's Not Tested Will Eventually Break Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 11 / 41
12. Don't let your users be your unit test Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 12 / 41
13. Control your software load See also: Don't DoS your own server Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 13 / 41
14. Welcome new collaborators Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 14 / 41
15. Make your code reproducible Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 15 / 41
16. An overview of testing in R Part 2, Testing R packages Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 16 / 41
17. R CMD check devtools::check() rcmdcheck::rcmdcheck() Command line interface & can be used on CI/CD Testing R packages Package infrastructure comes with native testing Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 17 / 41
18. {testthat} test_that("The meaning of life is 42", { expect_equal( meaning_of_life(), 42 ) }) Testing R packages Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 18 / 41
19. Testing R packages {testthat} Comes with a lot of native expectations library(testthat) ls("package:testthat", pattern = "^expect_") [1] "expect_condition" "expect_cpp_tests_pass" "expect_equal" [4] "expect_equal_to_reference" "expect_equivalent" "expect_error" [7] "expect_failure" "expect_false" "expect_gt" [10] "expect_gte" "expect_identical" "expect_invisible" [13] "expect_is" "expect_known_failure" "expect_known_hash" [16] "expect_known_output" "expect_known_value" "expect_length" [19] "expect_less_than" "expect_lt" "expect_lte" [22] "expect_mapequal" "expect_match" "expect_message" [25] "expect_more_than" "expect_named" "expect_null" [28] "expect_output" "expect_output_file" "expect_reference" [31] "expect_s3_class" "expect_s4_class" "expect_setequal" [34] "expect_silent" "expect_success" "expect_that" [37] "expect_true" "expect_type" "expect_vector" [40] "expect_visible" "expect_warning" Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 19 / 41
20. Testing R packages {testthat} Custom expectations can be defined expect_file_exists <- function(file) { act <- quasi_label(rlang::enquo(file), arg = "fls") act$val <- file.exists(file) expect( isTRUE(act$val), sprintf("File %s doesn't exist.", fls) ) invisible(act$val) } Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 20 / 41
21. Testing R packages Other tools to define tests for functions {testit}, which comes with two functions (assert() and test_pkg()) {testthis}, to enhance unit testing (use_testdata(), test_coverage()) {tinytest}, a lightweight and dependency free unit testing framework Complementary tools {webmockr} and {vcr}, to test for HTTP requests {httptest}: webtesting through {httr} Document {covr} & {covrpage}, to document test coverage {testdown}: {testthat} results as a bookdown Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 21 / 41
22. CI/CD Github Actions https://github.com/r-lib/actions GitHub Actions for the R language - uses: r-lib/actions/setup-r@v1 - uses: r-lib/actions/setup-pandoc@v1 - uses: r-lib/actions/setup-tinytex@master - name: Test run: Rscript -e 'rcmdcheck::rcmdcheck()' Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 22 / 41
23. CI/CD Travis https://docs.travis-ci.com/user/languages/r/ Set up R specific dependencies, and test the package language: r before_install: - tlmgr install index install: - R -e 'devtools::install_deps(dep = T)' script: - R CMD build . - R CMD check *tar.gz Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 23 / 41
24. CI/CD GitLab Some docker images can be used as a base: image: colinfay/r-ci-tidyverse:3.6.0 test: script: - R -e 'remotes::install_local()' - R -e 'devtools::check()' Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 24 / 41
25. {shiny} specific {shinytests} Test visual regression of your application puppeteer Command line tool to mock a web session, in NodeJS {crrry} R tool to drive a {shiny} session Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 25 / 41
26. Everything is a package Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 26 / 41
27. Everything is a package Analysis as a package Applications as a package Tools as a package Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 27 / 41
28. An overview of testing in R Part 3, benchmark and load Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 28 / 41
29. library(profvis) top <- function(){ pause(0.1) lapply(1:10, function(x){ x * 10 }) middle() } middle <- function(){ pause(0.2); 1e4 * 9 } profvis({ top() }) Benchmark and load Profiling R code with {profvis} Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 29 / 41
30. Benchmark and load Profiling memory with {profmem} library(profmem) p <- profmem({ x <- raw(1000) A <- matrix(rnorm(100), ncol = 10) }) p2 <- subset(p, bytes > 1000) print(p2) Rprofmem memory profiling of: { x <- raw(1000) A <- matrix(rnorm(100), ncol = 10) } Memory allocations: what bytes calls 1 alloc 1048 raw() Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 30 / 41
31. Benchmark and load Timing with {bench} res <- bench::mark( plus = 1+1, sum = sum(1, 1), iterations = 10000 ) res # A tibble: 2 x 6 expression min median `itr/sec` mem_alloc `gc/sec` <bch:expr> <bch:tm> <bch:tm> <dbl> <bch:byt> <dbl> 1 plus 71ns 101ns 4561272. 0B 0 2 sum 222ns 266ns 1733893. 0B 0 Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 31 / 41
32. Benchmark and load ggplot2::autoplot(res) Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 32 / 41
33. Benchmark and load in {shiny} {shinyloadtest} : native R package + Cli to record and replay load tests {dockerstats} : get Docker stats inside R {crrry} + {dockerstats} : replay session and watch the Docker stats Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 33 / 41
34. Wrapper around docker stats Turns the stats from Docker into an R dataframe Can be called recursively Focus - {dockerstats} Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 34 / 41
35. How - {dockerstats} # Launch the container from the terminal docker run --name hexmake --rm -p 2811:80 colinfay/hexmake # Call dockerstats, only selecting specific columns dockerstats::dockerstats("hexmake")[, c("Name", "MemUsage", "CPUPerc", "MemPerc", "record_time")] Name MemUsage CPUPerc MemPerc record_time 1 hexmake 137MiB 0.08 1.72 2020-09-09 21:08:48 Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 35 / 41
36. {dockerstats} Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 36 / 41
37. An overview of testing in R Conclusion Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 37 / 41
38. To conclude Test often, test soon Everything can, and should be scripted Testing is not just about functions, it's also about load and speed R Packages make our life easier Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 38 / 41
39. devtools.r-lib.org github.com/r-lib/rcmdcheck testthat.r-lib.org github.com/yihui/testit github.com/s-fleck/testthis github.com/markvanderloo/tinytest github.com/ropensci/webmockr github.com/ropensci/vcr covr.r-lib.org/ github.com/yonicd/covrpage github.com/ThinkR-open/testdown github.com/nealrichardson/httptest rstudio.github.io/shinytest pptr.dev github.com/ColinFay/crrry rstudio.github.io/profvis github.com/HenrikBengtsson/profmem bench.r-lib.org rstudio.github.io/shinyloadtest github.com/ColinFay/dockerstats Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 39 / 41
40. engineering-shiny.org Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 40 / 41
41. Thx! Questions? Colin Fay Online colin@thinkr.fr http://twitter.com/_colinfay http://twitter.com/thinkr_fr https://github.com/ColinFay https://thinkr.fr/ https://rtask.thinkr.fr/ https://colinfay.me/ Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 41 / 41