$30 off During Our Annual Pro Sale. View Details »

Analyze Rails CI

Analyze Rails CI

Rails Developers Meetup 2019

https://railsdm.github.io/

Fumiaki MATSUSHIMA

March 23, 2019
Tweet

More Decks by Fumiaki MATSUSHIMA

Other Decks in Programming

Transcript

  1. #railsdm2019 Analyze Rails CI
    Analyze Rails CI
    @mtsmfm
    Fumiaki Matsushima
    RailsDM 2019 #railsdm2019

    View Slide

  2. #railsdm2019 Analyze Rails CI
    ➔ Web Dev at Quipper
    ➔ Ruby, Mahjong, Dead by
    Daylight
    ➔ Nishi-nippori.rb organizer
    ➔ GraphQL Tokyo organizer
    @mtsmfm.inspect

    View Slide

  3. #railsdm2019 Analyze Rails CI
    https://studysapuri.jp/

    View Slide

  4. #railsdm2019 Analyze Rails CI

    View Slide

  5. #railsdm2019 Analyze Rails CI
    https://nishinipporirb.doorkeeper.jp/events/88540
    Nishi-nippori.rb meetup #53

    View Slide

  6. #railsdm2019 Analyze Rails CI
    https://nishinipporirb.doorkeeper.jp/events/88540
    Meetup
    place
    Nishi-
    nippori

    View Slide

  7. #railsdm2019 Analyze Rails CI
    I gave a talk about flaky tests before
    https://speakerdeck.com/mtsmfm/how-do-e2e-tests-fail-randomly

    View Slide

  8. #railsdm2019 Analyze Rails CI
    I’ve been writing flaky tests

    View Slide

  9. #railsdm2019 Analyze Rails CI
    I’m going to write new
    variation of flaky tests

    View Slide

  10. #railsdm2019 Analyze Rails CI
    “100% is the wrong
    reliability target for
    basically everything”
    Site Reliability Engineering
    Edited by Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy
    https://landing.google.com/sre/sre-book/chapters/introduction/

    View Slide

  11. #railsdm2019 Analyze Rails CI
    The same with flaky tests

    View Slide

  12. #railsdm2019 Analyze Rails CI
    97.3 %

    View Slide

  13. #railsdm2019 Analyze Rails CI
    The percentage of passed
    jobs on master branch
    in Feb 2019
    in rails/rails repo

    View Slide

  14. #railsdm2019 Analyze Rails CI
    https://github.com/rails/rails/pull/33829#issuecomment-420586191
    Rails also has flaky tests

    View Slide

  15. #railsdm2019 Analyze Rails CI

    View Slide

  16. #railsdm2019 Analyze Rails CI
    https://has-it-failed.herokuapp.com

    View Slide

  17. #railsdm2019 Analyze Rails CI
    https://has-it-failed.herokuapp.com/?q=test_invoke_when_generator_is_not_found

    View Slide

  18. #railsdm2019 Analyze Rails CI
    What I’d like to talk
    - I’ve published Rails CI result as a public dataset on
    BigQuery
    https://console.cloud.google.com/bigquery?p=rails-travis
    -result&d=rails_travis_result&page=dataset
    - You can find the test was failed before or not on Rails
    CI via http://has-it-failed.herokuapp.com
    - There’s a room for improvement if we have CI result
    database
    (e.g. Make CI result passed if failed one is a flaky test)

    View Slide

  19. #railsdm2019 Analyze Rails CI
    Browser
    rails-ci-result-importer
    BigQuery
    PostgreSQL
    has-it-failed
    TravisCI API
    Overview of systems I created for this talk

    View Slide

  20. #railsdm2019 Analyze Rails CI
    Agenda
    1. rails-ci-result-importer
    2. has-it-failed app
    3. Future work

    View Slide

  21. #railsdm2019 Analyze Rails CI
    1. rails-ci-result-importer

    View Slide

  22. #railsdm2019 Analyze Rails CI
    rails-ci-result-importer
    BigQuery
    TravisCI API

    View Slide

  23. #railsdm2019 Analyze Rails CI
    https://github.com/mtsmfm/rails-ci-result-importer

    View Slide

  24. #railsdm2019 Analyze Rails CI
    Heroku
    Scheduler
    Embulk
    BigQuery
    Redis
    diff.yml
    TravisCI API
    bin/import
    embulk-input-travis
    embulk-output-bigquery

    View Slide

  25. #railsdm2019 Analyze Rails CI
    https://www.embulk.org/docs/

    View Slide

  26. #railsdm2019 Analyze Rails CI
    Why embulk?
    - Just curious
    - Embulk plugin can be written in (J)Ruby

    View Slide

  27. #railsdm2019 Analyze Rails CI
    Heroku
    Scheduler
    Embulk
    BigQuery
    Redis
    diff.yml
    TravisCI API
    bin/import
    embulk-input-travis
    embulk-output-bigquery

    View Slide

  28. #railsdm2019 Analyze Rails CI
    https://github.com/mtsmfm/embulk-input-travis

    View Slide

  29. #railsdm2019 Analyze Rails CI
    Resources on TravisCI
    Commit
    d8d6bd5
    Build
    59740
    Job
    59740.1
    GEM=railties
    Ruby: 2.5.3
    $ git push xxx
    Repository
    rails/rails Job
    59740.2
    GEM=railties
    Ruby: 2.6.0
    Job
    59740.3
    GEM=ap,ac
    Ruby: 2.6.0
    ...

    View Slide

  30. #railsdm2019 Analyze Rails CI
    Table schema on BigQuery
    Field name Type Description
    id INTEGER Job id on TravisCI
    data STRING Job attributes in JSON
    log STRING Job log
    started_at TIMESTAMP When this job started
    build_number INTEGER Build number job belongs to
    build_data STRING Build attributes job belongs to
    commit_data STRING Commit attributes job belongs to

    View Slide

  31. #railsdm2019 Analyze Rails CI
    Important notice
    - This dataset doesn’t have all results (for now)
    - Some rows don’t have commit_data (for now)
    - I paid to store this dataset but you need to
    pay money to run query

    View Slide

  32. #railsdm2019 Analyze Rails CI
    Table schema on BigQuery
    Field name Type Description
    id INTEGER Job id on TravisCI
    data STRING Job attributes in JSON
    log STRING Job log
    started_at TIMESTAMP When this job started
    build_number INTEGER Build number job belongs to
    build_data STRING Build attributes job belongs to
    commit_data STRING Commit attributes job belongs to
    Partitioned by started_at

    View Slide

  33. #railsdm2019 Analyze Rails CI
    Be sure to query with “WHERE started_at”

    View Slide

  34. #railsdm2019 Analyze Rails CI
    40 / 1024 * 5 ≒ 0.2 (USD)
    https://cloud.google.com/bigquery/pricing

    View Slide

  35. #railsdm2019 Analyze Rails CI
    Important notice
    - This dataset doesn’t have all results (for now)
    - Some rows don’t have commit_data (for now)
    - I paid to store this dataset but you need to
    pay money to run query
    - Be sure to query with “WHERE started_at”
    - Full scan costs only $0.2 for now but amount of data
    keeps increasing

    View Slide

  36. #railsdm2019 Analyze Rails CI
    2. has-it-failed app

    View Slide

  37. #railsdm2019 Analyze Rails CI
    https://github.com/mtsmfm/has-it-failed

    View Slide

  38. #railsdm2019 Analyze Rails CI
    Browser
    BigQuery
    PostgreSQL
    Find all failed tests
    via SQL
    ImportJob
    Store PG
    (cache)
    Web server

    View Slide

  39. #railsdm2019 Analyze Rails CI
    Browser
    BigQuery
    PostgreSQL
    Find all failed tests
    via SQL
    ImportJob
    Store PG
    (cache)
    Web server

    View Slide

  40. #railsdm2019 Analyze Rails CI
    It’s just raw log...
    /home/travis/.rvm/rubies/ruby-head/bin/ruby -w -Itest -Ilib -I../activesupport/lib -I../actionpack/lib
    -I../actionview/lib -I../activemodel/lib test/commands/server_test.rb
    Run options: --seed 10052
    # Running:
    ....F
    Failure:
    Rails::Command::ServerCommandTest#test_using_server_mistype_without_suggestion
    [test/commands/server_test.rb:38]:
    Expected /Maybe you meant/ to not match "Could not find server \"t\". Maybe you meant \"cgi\"?\nRun
    `rails server --help` for more options.\n".
    rails test test/commands/server_test.rb:35
    .....................
    Finished in 0.760100s, 34.2060 runs/s, 71.0433 assertions/s.
    26 runs, 54 assertions, 1 failures, 0 errors, 0 skips
    ^^^ +++
    --- test/configuration/middleware_stack_proxy_test.rb

    View Slide

  41. #railsdm2019 Analyze Rails CI
    It’s just raw log...
    /home/travis/.rvm/rubies/ruby-head/bin/ruby -w -Itest -Ilib -I../activesupport/lib -I../actionpack/lib
    -I../actionview/lib -I../activemodel/lib test/commands/server_test.rb
    Run options: --seed 10052
    # Running:
    ....F
    Failure:
    Rails::Command::ServerCommandTest#test_using_server_mistype_without_suggestion
    [test/commands/server_test.rb:38]:
    Expected /Maybe you meant/ to not match "Could not find server \"t\". Maybe you meant \"cgi\"?\nRun
    `rails server --help` for more options.\n".
    rails test test/commands/server_test.rb:35
    .....................
    Finished in 0.760100s, 34.2060 runs/s, 71.0433 assertions/s.
    26 runs, 54 assertions, 1 failures, 0 errors, 0 skips
    ^^^ +++
    --- test/configuration/middleware_stack_proxy_test.rb
    Test class,
    test method
    Failed info

    View Slide

  42. #railsdm2019 Analyze Rails CI
    Browser
    BigQuery
    PostgreSQL
    Find all failed tests
    via SQL
    ImportJob
    Store PG
    (cache)
    Web server

    View Slide

  43. #railsdm2019 Analyze Rails CI
    SELECT * FROM(
    SELECT DISTINCT id, data, build_number, build_data,
    extractCiResult(log) AS parse_result
    FROM `rails-travis-result.rails_travis_result.jobs`
    WHERE "#{from.iso8601}" < started_at
    AND started_at < "#{to.iso8601}"
    ) WHERE parse_result <> "error"
    AND JSON_EXTRACT_SCALAR(data, "$.state") = "failed"
    AND JSON_ARRAY_LENGTH(
    JSON_EXTRACT(parse_result, "$.failedTests")
    ) > 0

    View Slide

  44. #railsdm2019 Analyze Rails CI
    SELECT * FROM(
    SELECT DISTINCT id, data, build_number, build_data,
    extractCiResult(log) AS parse_result
    FROM `rails-travis-result.rails_travis_result.jobs`
    WHERE "#{from.iso8601}" < started_at
    AND started_at < "#{to.iso8601}"
    ) WHERE parse_result <> "error"
    AND JSON_EXTRACT_SCALAR(data, "$.state") = "failed"
    AND JSON_ARRAY_LENGTH(
    JSON_EXTRACT(parse_result, "$.failedTests")
    ) > 0

    View Slide

  45. #railsdm2019 Analyze Rails CI
    CREATE TEMP FUNCTION extractCiResult (log STRING) RETURNS STRING LANGUAGE js AS
    """
    try {
    const railsCiResult = (TravisResultParser.parse(log)).find(
    command => command.includes('[Travis CI]')
    );
    return JSON.stringify(RailsCiParser.parse(railsCiResult));
    } catch {
    return 'error';
    }
    """
    OPTIONS
    (
    library=[
    "gs://rails-travis-result/parser/v0.1.0/rails_ci.js",
    "gs://rails-travis-result/parser/v0.1.0/travis_result.js"
    ]
    );

    View Slide

  46. #railsdm2019 Analyze Rails CI
    BigQuery supports JavaScript UDF
    - Let’s write a parser in JavaScript and run on
    BigQuery directly!

    View Slide

  47. #railsdm2019 Analyze Rails CI
    PEG and PEG.js
    - Parsing Expression Grammar
    - PEG.js is one of the most popular parser
    generator
    - Export parser from PEG
    - Online editor

    View Slide

  48. #railsdm2019 Analyze Rails CI
    https://pegjs.org/online

    View Slide

  49. #railsdm2019 Analyze Rails CI
    Result
    = (Noise command:Command Noise { return command; })+
    Noise
    = (!Command .)*
    Command
    = TimeStart commandBody:CommandBody TimeEnd {
    return commandBody;
    }
    CommandBody
    = chars:(!TimeEnd char:. {return char})* {
    return chars.join('').replace(/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]/g, '').replace(/\r\n/g, "\n")
    }
    TimeStart
    = "travis_time:start:" (!"\n" .)+ "\n"
    TimeEnd
    = "travis_time:end:" (!"\n" .)+ "\n"

    View Slide

  50. #railsdm2019 Analyze Rails CI
    How to parse Rails CI log
    1. Find output from entrypoint (ci/travis.rb)
    2. Find test command
    e.g. $ ruby foo_test.rb
    3. Find finished messages

    View Slide

  51. #railsdm2019 Analyze Rails CI
    CREATE TEMP FUNCTION extractCiResult (log STRING) RETURNS STRING LANGUAGE js AS
    """
    try {
    const railsCiResult = (TravisResultParser.parse(log)).find(
    command => command.includes('[Travis CI]')
    );
    return JSON.stringify(RailsCiParser.parse(railsCiResult));
    } catch {
    return 'error';
    }
    """
    OPTIONS
    (
    library=[
    "gs://rails-travis-result/parser/v0.1.0/rails_ci.js",
    "gs://rails-travis-result/parser/v0.1.0/travis_result.js"
    ]
    );

    View Slide

  52. #railsdm2019 Analyze Rails CI
    Log contains infrastructure info
    https://api.travis-ci.org/v3/job/505038092/log.txt
    ...

    View Slide

  53. #railsdm2019 Analyze Rails CI
    Result
    = (Noise command:Command Noise { return command; })+
    Noise
    = (!Command .)*
    Command
    = TimeStart commandBody:CommandBody TimeEnd {
    return commandBody;
    }
    CommandBody
    = chars:(!TimeEnd char:. {return char})* {
    return chars.join('').replace(/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]/g, '').replace(/\r\n/g, "\n")
    }
    TimeStart
    = "travis_time:start:" (!"\n" .)+ "\n"
    TimeEnd
    = "travis_time:end:" (!"\n" .)+ "\n"

    View Slide

  54. #railsdm2019 Analyze Rails CI
    travis_fold:start:worker_info

    travis_fold:end:worker_info

    travis_time:start:08cc8690
    $ bundle install --jobs 3 --retry 3
    travis_time:end:08cc8690

    ...
    travis_time:start:262254e0
    $ ci/travis.rb
    [Travis CI] railties
    Running command: bundle exec rake test
    travis_time:end:262254e0

    Done. Your build exited with 0.
    Result
    = (Noise command:Command Noise { return command; })+

    View Slide

  55. #railsdm2019 Analyze Rails CI
    travis_fold:start:worker_info

    travis_fold:end:worker_info

    travis_time:start:08cc8690
    $ bundle install --jobs 3 --retry 3
    travis_time:end:08cc8690

    ...
    travis_time:start:262254e0
    $ ci/travis.rb
    [Travis CI] railties
    Running command: bundle exec rake test
    travis_time:end:262254e0

    Done. Your build exited with 0.
    Noise
    Result
    = (Noise command:Command Noise { return command; })+
    Noise
    Command
    Noise
    Noise
    Command

    View Slide

  56. #railsdm2019 Analyze Rails CI
    CREATE TEMP FUNCTION extractCiResult (log STRING) RETURNS STRING LANGUAGE js AS
    """
    try {
    const railsCiResult = (TravisResultParser.parse(log)).find(
    command => command.includes('[Travis CI]')
    );
    return JSON.stringify(RailsCiParser.parse(railsCiResult));
    } catch {
    return 'error';
    }
    """
    OPTIONS
    (
    library=[
    "gs://rails-travis-result/parser/v0.1.0/rails_ci.js",
    "gs://rails-travis-result/parser/v0.1.0/travis_result.js"
    ]
    );

    View Slide

  57. #railsdm2019 Analyze Rails CI
    https://github.com/rails/rails/blob/d8d6bd5e63a9a4a6c06a4dde3c7137ee2be105fd/ci/travis.rb#L45

    View Slide

  58. #railsdm2019 Analyze Rails CI
    CREATE TEMP FUNCTION extractCiResult (log STRING) RETURNS STRING LANGUAGE js AS
    """
    try {
    const railsCiResult = (TravisResultParser.parse(log)).find(
    command => command.includes('[Travis CI]')
    );
    return JSON.stringify(RailsCiParser.parse(railsCiResult));
    } catch {
    return 'error';
    }
    """
    OPTIONS
    (
    library=[
    "gs://rails-travis-result/parser/v0.1.0/rails_ci.js",
    "gs://rails-travis-result/parser/v0.1.0/travis_result.js"
    ]
    );

    View Slide

  59. #railsdm2019 Analyze Rails CI
    RailsCiResult
    = Noise tests:Test+ {
    return {
    failedTests: tests.filter(t => t.summary).filter(({summary: {failuresCount, errorsCount}}) => failuresCount + errorsCount > 0),
    coreDumpedTests: tests.filter(t => t.dump),
    noises: tests.filter(t => t.noise)
    }
    }
    Noise
    = (!Test .)*
    Test
    = command:TestCommand
    result:(
    results:TestFailureOrErrorResult* summary:TestFinished TestOutputNoise { return { summary, results } }
    / results:TestSuccessResult summary:TestFinished TestOutputNoise { return { summary, results } }
    / dump:RubyCoreDump { return { dump } }
    / noise:TestOutputNoise { return { noise } }
    ) {
    return { command, ...result };
    }
    TestCommand
    = command:$("/home/travis/.rvm" $(!("/bin/" "j"? "ruby -w" / "\n") .)* "/bin/" "j"? "ruby -w" $(!"\n" .)*)
    "\n" { return command }
    TestOutputNoise
    = (!TestCommand .)*
    TestSuccessResult
    = message:$(!TestFinished .)* { return { message } }
    TestFailureOrErrorResult
    = (!(TestFailureOrErrorResultKeyword / TestFinished) .)* meta:TestFailureOrErrorResultKeyword message:TestFailureMessage {
    return { ...meta, message }
    }
    TestFailureMessage
    = $(!(TestFailureOrErrorResultKeyword / TestFinished) .)*
    TestFailureOrErrorResultKeyword
    = "\n" type:("Failure" / "Error") ":\n" testClass:TestClass "#" method:TestMethod (" [" file:TestFile ":" line:Int "]")? ":\n" {
    return { type, testClass, method }
    }
    TestClass
    = $[a-zA-Z:]+
    TestMethod
    = $[a-zA-Z_]+
    TestFile
    = $[a-z/\._]+
    TestFinished
    = "\n" runsCount:Int " runs, " assertionsCount:Int " assertions, " failuresCount:Int " failures, " errorsCount:Int " errors, " skipsCount:Int " skips" "\n" {
    return { runsCount, assertionsCount, failuresCount, errorsCount, skipsCount }
    }
    RubyCoreDump
    = $((!RubyCoreDumpNote .)* RubyCoreDumpNote)
    RubyCoreDumpNote
    = "\n"
    "[NOTE]\n"
    "You may have encountered a bug in the Ruby interpreter or extension libraries.\n"
    "Bug reports are welcome.\n"
    "For details: https://www.ruby-lang.org/bugreport.html\n"
    "\n"
    "Aborted (core dumped)\n"
    Int
    = chars:$[0-9]+ { return parseInt(chars) }

    View Slide

  60. #railsdm2019 Analyze Rails CI
    RailsCiResult
    = Noise tests:Test+ {
    return {
    failedTests: tests.filter(t => t.summary).filter(({summary: {failuresCount,
    errorsCount}}) => failuresCount + errorsCount > 0),
    coreDumpedTests: tests.filter(t => t.dump),
    noises: tests.filter(t => t.noise)
    }
    }

    View Slide

  61. #railsdm2019 Analyze Rails CI
    Test
    = command:TestCommand
    result:(
    TestFailureOrErrorResult* TestFinished TestOutputNoise
    / TestSuccessResult TestFinished TestOutputNoise
    / RubyCoreDump
    / TestOutputNoise
    )

    View Slide

  62. #railsdm2019 Analyze Rails CI
    TestCommand
    = command:$("/home/travis/.rvm" $(!("/bin/" "j"? "ruby -w" / "\n") .)* "/bin/" "j"? "ruby -w" $(!"\n" .)*) "\n"
    TestFailureOrErrorResult
    = (!(TestFailureOrErrorResultKeyword / TestFinished) .)* TestFailureOrErrorResultKeyword TestFailureMessage
    TestFailureOrErrorResultKeyword
    = "\n" type:("Failure" / "Error") ":\n" testClass:TestClass "#" method:TestMethod (" [" file:TestFile ":" line:Int "]")? ":\n"
    TestClass
    = $[a-zA-Z:]+
    TestMethod
    = $[a-zA-Z_]+
    TestFile
    = $[a-z/\._]+
    TestFinished
    = "\n" runsCount:Int " runs, " assertionsCount:Int " assertions, " failuresCount:Int " failures, " errorsCount:Int " errors, "
    skipsCount:Int " skips" "\n"

    View Slide

  63. #railsdm2019 Analyze Rails CI
    /home/travis/.rvm/rubies/ruby-head/bin/ruby -w -Itest -Ilib -I../activesupport/lib -I../actionpack/lib
    -I../actionview/lib -I../activemodel/lib test/commands/server_test.rb
    Run options: --seed 10052
    # Running:
    ....F
    Failure:
    Rails::Command::ServerCommandTest#test_using_server_mistype_without_suggestion
    [test/commands/server_test.rb:38]:
    Expected /Maybe you meant/ to not match "Could not find server \"t\". Maybe you meant \"cgi\"?\nRun
    `rails server --help` for more options.\n".
    rails test test/commands/server_test.rb:35
    .....................
    Finished in 0.760100s, 34.2060 runs/s, 71.0433 assertions/s.
    26 runs, 54 assertions, 1 failures, 0 errors, 0 skips
    ^^^ +++
    --- test/configuration/middleware_stack_proxy_test.rb
    /home/travis/.rvm/rubies/ruby-head/bin/ruby -w -Itest -Ilib -I../activesupport/lib -I../actionpack/lib
    -I../actionview/lib -I../activemodel/lib test/configuration/middleware_stack_proxy_test.rb

    View Slide

  64. #railsdm2019 Analyze Rails CI
    /home/travis/.rvm/rubies/ruby-head/bin/ruby -w -Itest -Ilib -I../activesupport/lib -I../actionpack/lib
    -I../actionview/lib -I../activemodel/lib test/commands/server_test.rb
    Run options: --seed 10052
    # Running:
    ....F
    Failure:
    Rails::Command::ServerCommandTest#test_using_server_mistype_without_suggestion
    [test/commands/server_test.rb:38]:
    Expected /Maybe you meant/ to not match "Could not find server \"t\". Maybe you meant \"cgi\"?\nRun
    `rails server --help` for more options.\n".
    rails test test/commands/server_test.rb:35
    .....................
    Finished in 0.760100s, 34.2060 runs/s, 71.0433 assertions/s.
    26 runs, 54 assertions, 1 failures, 0 errors, 0 skips
    ^^^ +++
    --- test/configuration/middleware_stack_proxy_test.rb
    /home/travis/.rvm/rubies/ruby-head/bin/ruby -w -Itest -Ilib -I../activesupport/lib -I../actionpack/lib
    -I../actionview/lib -I../activemodel/lib test/configuration/middleware_stack_proxy_test.rb
    TestCommand
    Noise
    TestFailureOrErrorResultKeyword
    FailureMessage
    TestFinished

    View Slide

  65. #railsdm2019 Analyze Rails CI
    {
    "failedTests": [
    {
    "command": "/home/travis/.rvm/rubies/ruby-head/bin/ruby -w -Itest -Ilib -I../activesupport/lib -I../actionpack/lib
    -I../actionview/lib -I../activemodel/lib test/commands/server_test.rb",
    "summary": {
    "runsCount": 26,
    "assertionsCount": 54,
    "failuresCount": 1,
    "errorsCount": 0,
    "skipsCount": 0
    },
    "results": [
    {
    "type": "Failure",
    "testClass": "Rails::Command::ServerCommandTest",
    "method": "test_using_server_mistype_without_suggestion",
    "message": "Expected /Maybe you meant/ to not match \"Could not find server \\\"t\\\". Maybe you meant \\\"cgi\\\"?\\nRun
    `rails server --help` for more options.\\n\".\n\nrails test test/commands/server_test.rb:35\n\n.....................\n\nFinished in
    0.760100s, 34.2060 runs/s, 71.0433 assertions/s."
    } ...

    View Slide

  66. #railsdm2019 Analyze Rails CI
    Yay!

    View Slide

  67. #railsdm2019 Analyze Rails CI
    Browser
    BigQuery
    PostgreSQL
    Find all failed tests
    via SQL
    ImportJob
    Store PG
    (cache)
    Web server

    View Slide

  68. #railsdm2019 Analyze Rails CI
    has-it-failed web
    - Rails 6.0.0.beta3
    - Heroku
    - Nothing special

    View Slide

  69. #railsdm2019 Analyze Rails CI
    3. Future work

    View Slide

  70. #railsdm2019 Analyze Rails CI
    CREATE TEMP FUNCTION extractCiResult (log STRING) RETURNS STRING LANGUAGE js AS
    """
    try {
    const railsCiResult = (TravisResultParser.parse(log)).find(
    command => command.includes('[Travis CI]')
    );
    return JSON.stringify(RailsCiParser.parse(railsCiResult));
    } catch {
    return 'error';
    }
    """
    OPTIONS
    (
    library=[
    "gs://rails-travis-result/parser/v0.1.0/rails_ci.js",
    "gs://rails-travis-result/parser/v0.1.0/travis_result.js"
    ]
    );

    View Slide

  71. #railsdm2019 Analyze Rails CI
    This parser is also flaky

    View Slide

  72. #railsdm2019 Analyze Rails CI
    It’s much better to collect
    test report in future

    View Slide

  73. #railsdm2019 Analyze Rails CI
    https://docs.travis-ci.com/user/uploading-artifacts/
    TravisCI doesn’t provide the place to store artifacts

    View Slide

  74. #railsdm2019 Analyze Rails CI
    Buildkite can store artifacts
    https://buildkite.com/pricing

    View Slide

  75. #railsdm2019 Analyze Rails CI
    https://github.com/rails/rails/pull/35698

    View Slide

  76. #railsdm2019 Analyze Rails CI
    [Idea] Make CI passed if failed one is flaky
    1. Generate JUnit format XML test report
    2. Send test report to the service
    3. The service collects flaky tests data and
    returns 0 or 1
    4. Use the value in step 3 as exit code

    View Slide

  77. #railsdm2019 Analyze Rails CI
    $ bin/ruby -I test || true
    Flaky tests
    Data
    Web server
    $ test $(curl --form
    "[email protected]" https://server) =
    ‘0’
    result.xml

    View Slide

  78. #railsdm2019 Analyze Rails CI
    $ bin/ruby -I test || true
    Flaky tests
    Data
    1. Generate XML and always
    returns 0
    Web server
    $ test $(curl --form
    "[email protected]" https://server) =
    ‘0’
    result.xml

    View Slide

  79. #railsdm2019 Analyze Rails CI
    $ bin/ruby -I test || true
    Flaky tests
    Data
    1. Generate XML and always
    returns 0
    Web server
    $ test $(curl --form
    "[email protected]" https://server) =
    ‘0’
    result.xml
    2. Send test report to the service

    View Slide

  80. #railsdm2019 Analyze Rails CI
    $ bin/ruby -I test || true
    Flaky tests
    Data
    1. Generate XML and always
    returns 0
    Web server
    $ test $(curl --form
    "[email protected]" https://server) =
    ‘0’
    result.xml
    3. Service returns 0 or 1
    2. Send test report to the service

    View Slide

  81. #railsdm2019 Analyze Rails CI
    $ bin/ruby -I test || true
    Flaky tests
    Data
    1. Generate XML and always
    returns 0
    Web server
    $ test $(curl --form
    "[email protected]" https://server) =
    ‘0’
    result.xml
    3. Service returns 0 or 1
    4. Verify returned value
    2. Send test report to the service

    View Slide

  82. #railsdm2019 Analyze Rails CI
    Conclusion
    - I’ve published Rails CI result as a public dataset on
    BigQuery
    https://console.cloud.google.com/bigquery?p=rails-travis
    -result&d=rails_travis_result&page=dataset
    - You can find the test was failed before or not on Rails
    CI via http://has-it-failed.herokuapp.com
    - There’s a room for improvement if we have CI result
    database
    (e.g. Make CI result passed if failed one is a flaky test)

    View Slide