$30 off During Our Annual Pro Sale. View Details »

An overview of testing in R

Colin Fay
September 09, 2020

An overview of testing in R

Talk at the AI Testing online meetup

Colin Fay

September 09, 2020
Tweet

More Decks by Colin Fay

Other Decks in Programming

Transcript

  1. An overview of testing in R
    rdev::secure()
    2020-09-09
    Colin Fay - ThinkR
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 1 / 41

    View Slide

  2. $ whoami
    Colin FAY
    Data Scientist & R-Hacker at ThinkR, a french company focused on Data Science & R.
    Hyperactive open source developer, lead developer of the {golem} project.
    https://thinkr.fr
    https://rtask.thinkr.fr
    https://twitter.com/_colinfay
    https://github.com/colinfay
    https://colinfay.me
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 2 / 41

    View Slide

  3. ThinkR
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 3 / 41

    View Slide

  4. Data Science engineering, focused on R.
     Training
     Software Engineering
     R in production
     Consulting
    ThinkR
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 4 / 41

    View Slide

  5. An overview of testing in R
    Securing youR development
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 5 / 41

    View Slide

  6. Today's menu
    A (non-comprehensive) overview of testing technologies in the R world
    Program
    Testing in R, why?
    Testing R packages
    Benchmark and loadtest
    Disclaimer
    I know nothing about AI
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 6 / 41

    View Slide

  7. An overview of testing in R
    Part 0, The Current State of Unit Test
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 7 / 41

    View Slide

  8. The two states of unit test
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 8 / 41

    View Slide

  9. The two states of unit test
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 9 / 41

    View Slide

  10. An overview of testing in R
    Part 1, Why
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 10 / 41

    View Slide

  11. Everything That's Not Tested
    Will Eventually Break
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 11 / 41

    View Slide

  12. Don't let your users be your unit test
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 12 / 41

    View Slide

  13. Control your software load
    See also: Don't DoS your own server
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 13 / 41

    View Slide

  14. Welcome new collaborators
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 14 / 41

    View Slide

  15. Make your code reproducible
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 15 / 41

    View Slide

  16. An overview of testing in R
    Part 2, Testing R packages
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 16 / 41

    View Slide

  17. R CMD check
    devtools::check()
    rcmdcheck::rcmdcheck()
    Command line interface &
    can be used on CI/CD
    Testing R packages
    Package infrastructure comes with native testing
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 17 / 41

    View Slide

  18. {testthat}
    test_that("The meaning of life is
    42", {
    expect_equal(
    meaning_of_life(),
    42
    )
    })
    Testing R packages
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 18 / 41

    View Slide

  19. Testing R packages
    {testthat}
    Comes with a lot of native expectations
    library(testthat)
    ls("package:testthat", pattern = "^expect_")
    [1] "expect_condition" "expect_cpp_tests_pass" "expect_equal"
    [4] "expect_equal_to_reference" "expect_equivalent" "expect_error"
    [7] "expect_failure" "expect_false" "expect_gt"
    [10] "expect_gte" "expect_identical" "expect_invisible"
    [13] "expect_is" "expect_known_failure" "expect_known_hash"
    [16] "expect_known_output" "expect_known_value" "expect_length"
    [19] "expect_less_than" "expect_lt" "expect_lte"
    [22] "expect_mapequal" "expect_match" "expect_message"
    [25] "expect_more_than" "expect_named" "expect_null"
    [28] "expect_output" "expect_output_file" "expect_reference"
    [31] "expect_s3_class" "expect_s4_class" "expect_setequal"
    [34] "expect_silent" "expect_success" "expect_that"
    [37] "expect_true" "expect_type" "expect_vector"
    [40] "expect_visible" "expect_warning"
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 19 / 41

    View Slide

  20. Testing R packages
    {testthat}
    Custom expectations can be defined
    expect_file_exists <- function(file) {
    act <- quasi_label(rlang::enquo(file), arg = "fls")
    act$val <- file.exists(file)
    expect(
    isTRUE(act$val),
    sprintf("File %s doesn't exist.", fls)
    )
    invisible(act$val)
    }
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 20 / 41

    View Slide

  21. Testing R packages
    Other tools to define tests for functions
    {testit}, which comes with two functions (assert() and test_pkg())
    {testthis}, to enhance unit testing (use_testdata(), test_coverage())
    {tinytest}, a lightweight and dependency free unit testing framework
    Complementary tools
    {webmockr} and {vcr}, to test for HTTP requests
    {httptest}: webtesting through {httr}
    Document
    {covr} & {covrpage}, to document test coverage
    {testdown}: {testthat} results as a bookdown
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 21 / 41

    View Slide

  22. CI/CD
    Github Actions
    https://github.com/r-lib/actions
    GitHub Actions for the R language
    - uses: r-lib/actions/setup-r@v1
    - uses: r-lib/actions/setup-pandoc@v1
    - uses: r-lib/actions/setup-tinytex@master
    - name: Test
    run: Rscript -e 'rcmdcheck::rcmdcheck()'
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 22 / 41

    View Slide

  23. CI/CD
    Travis
    https://docs.travis-ci.com/user/languages/r/
    Set up R specific dependencies, and test the package
    language: r
    before_install:
    - tlmgr install index
    install:
    - R -e 'devtools::install_deps(dep = T)'
    script:
    - R CMD build .
    - R CMD check *tar.gz
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 23 / 41

    View Slide

  24. CI/CD
    GitLab
    Some docker images can be used as a base:
    image: colinfay/r-ci-tidyverse:3.6.0
    test:
    script:
    - R -e 'remotes::install_local()'
    - R -e 'devtools::check()'
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 24 / 41

    View Slide

  25. {shiny} specific
    {shinytests}
    Test visual regression of your application
    puppeteer
    Command line tool to mock a web session, in NodeJS
    {crrry}
    R tool to drive a {shiny} session
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 25 / 41

    View Slide

  26. Everything is a package
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 26 / 41

    View Slide

  27. Everything is a package
    Analysis as a package
    Applications as a package
    Tools as a package
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 27 / 41

    View Slide

  28. An overview of testing in R
    Part 3, benchmark and load
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 28 / 41

    View Slide

  29. library(profvis)
    top <- function(){
    pause(0.1)
    lapply(1:10, function(x){
    x * 10
    })
    middle()
    }
    middle <- function(){
    pause(0.2); 1e4 * 9
    }
    profvis({
    top()
    })
    Benchmark and load
    Profiling R code with {profvis}
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 29 / 41

    View Slide

  30. Benchmark and load
    Profiling memory with {profmem}
    library(profmem)
    p <- profmem({
    x <- raw(1000)
    A <- matrix(rnorm(100), ncol = 10)
    })
    p2 <- subset(p, bytes > 1000)
    print(p2)
    Rprofmem memory profiling of:
    {
    x <- raw(1000)
    A <- matrix(rnorm(100), ncol = 10)
    }
    Memory allocations:
    what bytes calls
    1 alloc 1048 raw()
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 30 / 41

    View Slide

  31. Benchmark and load
    Timing with {bench}
    res <- bench::mark(
    plus = 1+1,
    sum = sum(1, 1),
    iterations = 10000
    )
    res
    # A tibble: 2 x 6
    expression min median `itr/sec` mem_alloc `gc/sec`

    1 plus 71ns 101ns 4561272. 0B 0
    2 sum 222ns 266ns 1733893. 0B 0
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 31 / 41

    View Slide

  32. Benchmark and load
    ggplot2::autoplot(res)
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 32 / 41

    View Slide

  33. Benchmark and load in {shiny}
    {shinyloadtest}
    : native R package + Cli to record and replay load tests
    {dockerstats}
    : get Docker stats inside R
    {crrry} + {dockerstats}
    : replay session and watch the Docker stats
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 33 / 41

    View Slide

  34. Wrapper around docker
    stats
    Turns the stats from Docker
    into an R dataframe
    Can be called recursively
    Focus - {dockerstats}
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 34 / 41

    View Slide

  35. How - {dockerstats}
    # Launch the container from the terminal
    docker run --name hexmake --rm -p 2811:80 colinfay/hexmake
    # Call dockerstats, only selecting specific columns
    dockerstats::dockerstats("hexmake")[, c("Name", "MemUsage", "CPUPerc",
    "MemPerc", "record_time")]
    Name MemUsage CPUPerc MemPerc record_time
    1 hexmake 137MiB 0.08 1.72 2020-09-09 21:08:48
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 35 / 41

    View Slide

  36. {dockerstats}
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 36 / 41

    View Slide

  37. An overview of testing in R
    Conclusion
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 37 / 41

    View Slide

  38. To conclude
    Test often, test soon
    Everything can, and should be scripted
    Testing is not just about functions, it's also about load and speed
    R Packages make our life easier
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 38 / 41

    View Slide

  39. devtools.r-lib.org
    github.com/r-lib/rcmdcheck
    testthat.r-lib.org
    github.com/yihui/testit
    github.com/s-fleck/testthis
    github.com/markvanderloo/tinytest
    github.com/ropensci/webmockr
    github.com/ropensci/vcr
    covr.r-lib.org/
    github.com/yonicd/covrpage
    github.com/ThinkR-open/testdown
    github.com/nealrichardson/httptest
    rstudio.github.io/shinytest
    pptr.dev
    github.com/ColinFay/crrry
    rstudio.github.io/profvis
    github.com/HenrikBengtsson/profmem
    bench.r-lib.org
    rstudio.github.io/shinyloadtest
    github.com/ColinFay/dockerstats

    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 39 / 41

    View Slide

  40. engineering-shiny.org
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 40 / 41

    View Slide

  41. Thx! Questions?
    Colin Fay
    Online
    [email protected]
    http://twitter.com/_colinfay
    http://twitter.com/thinkr_fr
    https://github.com/ColinFay
    https://thinkr.fr/
    https://rtask.thinkr.fr/
    https://colinfay.me/
    Colin FAY (@_ColinFay) - https://rtask.thinkr.fr 41 / 41

    View Slide