Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Find and Replace Code based on AST

Find and Replace Code based on AST

This is my presentation on RubyKaigi 2023

synvert

flyerhzm

May 13, 2023
Tweet

More Decks by flyerhzm

Other Decks in Technology

Transcript

  1. Find and Replace
    Code based on AST
    Richard Huang
    @flyerhzm

    View full-size slide

  2. What is AST?
    Abstract Syntax Tree
    class RubyKaigi
    def year
    2023
    end
    def location
    'Matsumoto'
    end
    end
    s(:class,
    s(:const, nil, :RubyKaigi), nil,
    s(:begin,
    s(:def, :year,
    s(:args),
    s(:int, 2023)),
    s(:def, :location,
    s(:args),
    s(:str, "Matsumoto"))))

    View full-size slide

  3. Why based on AST?
    Precise
    # match puts
    puts "hello world"
    puts"hello world"
    puts("hello world")
    puts(
    "hello world"
    )
    puts\
    "hello world"
    # not match puts
    'puts "hello world"'
    <<~HEREDOC
    puts "hello world"
    HEREDOC
    custom_puts "hello world"

    View full-size slide

  4. Why based on AST?
    Precise
    Powerful
    Find hash pair where key and value are identical
    and delete the value
    Find old hash rocket syntax and replace with new
    hash syntax
    Find minitest setup methods that do not call
    super and prepend the super

    View full-size slide

  5. What tools based on AST?
    Reek
    Rubocop
    Rufo

    View full-size slide

  6. Synvert
    Write snippet code to rewrite your source code
    https://synvert.net/

    View full-size slide

  7. Original Idea
    Simplify the process of upgrading a Rails project

    View full-size slide

  8. RubyKaigi 2014
    Write ruby code to change ruby code

    View full-size slide

  9. New Idea
    Give everyone the ability to
    find and replace code based on AST

    View full-size slide

  10. Node Query
    NQL - css like node query language
    Node Rules - hash object
    https://github.com/xinminlabs/node-query-ruby

    View full-size slide

  11. Find debug code
    puts "hello world"
    p current_user
    # NQL
    '.send[receiver=nil][message IN (puts p)]'
    # Node Rules
    { node_type: 'send', receiver: nil, message: { in: %w[puts p] } }

    View full-size slide

  12. Find two string arguments
    'slug from title'.gsub(' ', '_')
    # NQL
    '.send[message=gsub]
    [arguments.size=2]
    [arguments.first=.str]
    [arguments.last=.str]'
    # Node Rules
    { node_type: 'send',
    message: 'gsub',
    arguments: {
    size: 2,
    first: { node_type: 'str' },
    last: { node_type: 'str' } } }

    View full-size slide

  13. Find hash rocket key
    { :foo => 'bar' }
    # NQL
    %q{.pair[key=.sym][key=~/\A:([^'"]+)\z/]}
    # Node Rules
    { node_type: 'pair', key: /\A:([^'"]+)\z/ }

    View full-size slide

  14. Find hash pairs where the key and value
    are identical
    a: a,
    c: c,
    some_method(
    1
    2
    b: b + 2,
    3
    4
    d: d + 4
    5
    )
    6
    # NQL
    '.pair[key="{{value}}"]'
    # Node Rules
    { node_type: 'pair', key: "{{value}}" }

    View full-size slide

  15. Find minitest setup methods that do not
    call super
    def setup
    do_something
    end
    class TestMeme < Minitest::Test
    1
    2
    3
    4
    end
    5
    # NQL
    '.class[parent_class=Minitest::Test]
    .def[name=setup]:not_has(> .super)'
    # Node Rules
    { node_type: 'class', parent_class: 'Minitest::Test' } # first step
    { node_type: 'def', name: 'setup' } # second step
    children.none? { |node| node.type == 'super' } # third step

    View full-size slide

  16. Find hash value
    render nothing: true, status: :created
    render nothing: true
    1
    2
    # NQL
    '.hash[status_value!=nil]'
    # Node Rules
    { node_type: 'hash', status_value: { not: nil } }

    View full-size slide

  17. Node Mutation
    Provides a set of APIs to rewrite node source code
    https://github.com/xinminlabs/node-mutation-ruby

    View full-size slide

  18. replace_with
    errors[:base] = "author not present"
    1
    2
    errors.add(:base, "author not present")
    3
    replace_with node,
    '{{receiver}}.add({{arguments.0}}, {{arguments.1}})'

    View full-size slide

  19. replace_with
    errors[:base] = "author not present"
    1
    2
    errors.add(:base, "author not present")
    3 errors.add(:base, "author not present")
    errors[:base] = "author not present"
    1
    2
    3
    replace_with node,
    '{{receiver}}.add({{arguments.0}}, {{arguments.1}})'

    View full-size slide

  20. replace
    class Post < ActiveRecord::Base
    end
    1
    2
    3
    class Post < ApplicationRecord
    4
    end
    5
    replace node, :parent_class, with: 'ApplicationRecord'

    View full-size slide

  21. replace
    class Post < ActiveRecord::Base
    end
    1
    2
    3
    class Post < ApplicationRecord
    4
    end
    5
    class Post < ApplicationRecord
    end
    class Post < ActiveRecord::Base
    1
    end
    2
    3
    4
    5
    replace node, :parent_class, with: 'ApplicationRecord'

    View full-size slide

  22. insert
    open('http://test.com')
    1
    2
    URI.open('http://test.com')
    3
    insert node, 'URI.', at: 'beginning'

    View full-size slide

  23. insert
    open('http://test.com')
    1
    2
    URI.open('http://test.com')
    3 URI.open('http://test.com')
    open('http://test.com')
    1
    2
    3
    insert node, 'URI.', at: 'beginning'

    View full-size slide

  24. prepend
    def setup
    do_something
    end
    class TestMeme < Minitest::Test
    1
    2
    3
    4
    end
    5
    6
    class TestMeme < Minitest::Test
    7
    def setup
    8
    super
    9
    do_something
    10
    end
    11
    end
    12
    prepend node, 'super'

    View full-size slide

  25. prepend
    def setup
    do_something
    end
    class TestMeme < Minitest::Test
    1
    2
    3
    4
    end
    5
    6
    class TestMeme < Minitest::Test
    7
    def setup
    8
    super
    9
    do_something
    10
    end
    11
    end
    12
    def setup
    super
    do_something
    end
    class TestMeme < Minitest::Test
    1
    def setup
    2
    do_something
    3
    end
    4
    end
    5
    6
    class TestMeme < Minitest::Test
    7
    8
    9
    10
    11
    end
    12
    prepend node, 'super'

    View full-size slide

  26. append
    class TestMeme < Minitest::Test
    def teardown
    clean_something
    end
    end
    1
    2
    3
    4
    5
    6
    class TestMeme < Minitest::Test
    7
    def teardown
    8
    clean_something
    9
    super
    10
    end
    11
    end
    12
    append node, 'super'

    View full-size slide

  27. append
    class TestMeme < Minitest::Test
    def teardown
    clean_something
    end
    end
    1
    2
    3
    4
    5
    6
    class TestMeme < Minitest::Test
    7
    def teardown
    8
    clean_something
    9
    super
    10
    end
    11
    end
    12
    class TestMeme < Minitest::Test
    def teardown
    clean_something
    super
    end
    end
    class TestMeme < Minitest::Test
    1
    def teardown
    2
    clean_something
    3
    end
    4
    end
    5
    6
    7
    8
    9
    10
    11
    12
    append node, 'super'

    View full-size slide

  28. delete
    FactoryBot.create(:user)
    1
    2
    create(:user)
    3
    delete node, :receiver, :dot

    View full-size slide

  29. delete
    FactoryBot.create(:user)
    1
    2
    create(:user)
    3 create(:user)
    FactoryBot.create(:user)
    1
    2
    3
    delete node, :receiver, :dot

    View full-size slide

  30. remove
    puts(
    "hello"\
    "world"
    )
    1
    2
    3
    4
    5
    # nothing
    6
    remove node

    View full-size slide

  31. remove
    puts(
    "hello"\
    "world"
    )
    1
    2
    3
    4
    5
    # nothing
    6 # nothing
    puts(
    1
    "hello"\
    2
    "world"
    3
    )
    4
    5
    6
    remove node

    View full-size slide

  32. Hash node
    after_commit :add_to_index_later, on: :create
    1
    2
    after_create_commit :add_to_index_later
    3
    replace node, :message, with:
    'after_{{arguments.-1.on_value.to_value}}_commit'
    delete node, 'arguments.-1.on_pair', and_comma: true

    View full-size slide

  33. Hash node
    after_commit :add_to_index_later, on: :create
    1
    2
    after_create_commit :add_to_index_later
    3 after_create_commit :add_to_index_later
    after_commit :add_to_index_later, on: :create
    1
    2
    3
    replace node, :message, with:
    'after_{{arguments.-1.on_value.to_value}}_commit'
    delete node, 'arguments.-1.on_pair', and_comma: true

    View full-size slide

  34. node_query + node_mutation
    require 'parser/current'
    require 'parser_node_ext'
    require 'node_query'
    require 'node_mutation'
    1
    2
    3
    4
    5
    source = "foo\nputs foo\n"
    6
    node = Parser::CurrentRuby.parse(source)
    7
    query = NodeQuery.new('.send[receiver=nil][message IN (p puts)]')
    8
    mutation = NodeMutation.new(source)
    9
    query.query_nodes(node).each do |matching_node|
    10
    mutation.remove(matching_node)
    11
    end
    12
    result = mutation.process
    13
    result.new_source # foo\n
    14

    View full-size slide

  35. node_query + node_mutation
    require 'parser/current'
    require 'parser_node_ext'
    require 'node_query'
    require 'node_mutation'
    1
    2
    3
    4
    5
    source = "foo\nputs foo\n"
    6
    node = Parser::CurrentRuby.parse(source)
    7
    query = NodeQuery.new('.send[receiver=nil][message IN (p puts)]')
    8
    mutation = NodeMutation.new(source)
    9
    query.query_nodes(node).each do |matching_node|
    10
    mutation.remove(matching_node)
    11
    end
    12
    result = mutation.process
    13
    result.new_source # foo\n
    14
    source = "foo\nputs foo\n"
    node = Parser::CurrentRuby.parse(source)
    query = NodeQuery.new('.send[receiver=nil][message IN (p puts)]')
    query.query_nodes(node).each do |matching_node|
    end
    require 'parser/current'
    1
    require 'parser_node_ext'
    2
    require 'node_query'
    3
    require 'node_mutation'
    4
    5
    6
    7
    8
    mutation = NodeMutation.new(source)
    9
    10
    mutation.remove(matching_node)
    11
    12
    result = mutation.process
    13
    result.new_source # foo\n
    14

    View full-size slide

  36. node_query + node_mutation
    require 'parser/current'
    require 'parser_node_ext'
    require 'node_query'
    require 'node_mutation'
    1
    2
    3
    4
    5
    source = "foo\nputs foo\n"
    6
    node = Parser::CurrentRuby.parse(source)
    7
    query = NodeQuery.new('.send[receiver=nil][message IN (p puts)]')
    8
    mutation = NodeMutation.new(source)
    9
    query.query_nodes(node).each do |matching_node|
    10
    mutation.remove(matching_node)
    11
    end
    12
    result = mutation.process
    13
    result.new_source # foo\n
    14
    source = "foo\nputs foo\n"
    node = Parser::CurrentRuby.parse(source)
    query = NodeQuery.new('.send[receiver=nil][message IN (p puts)]')
    query.query_nodes(node).each do |matching_node|
    end
    require 'parser/current'
    1
    require 'parser_node_ext'
    2
    require 'node_query'
    3
    require 'node_mutation'
    4
    5
    6
    7
    8
    mutation = NodeMutation.new(source)
    9
    10
    mutation.remove(matching_node)
    11
    12
    result = mutation.process
    13
    result.new_source # foo\n
    14
    source = "foo\nputs foo\n"
    mutation = NodeMutation.new(source)
    mutation.remove(matching_node)
    result = mutation.process
    result.new_source # foo\n
    require 'parser/current'
    1
    require 'parser_node_ext'
    2
    require 'node_query'
    3
    require 'node_mutation'
    4
    5
    6
    node = Parser::CurrentRuby.parse(source)
    7
    query = NodeQuery.new('.send[receiver=nil][message IN (p puts)]')
    8
    9
    query.query_nodes(node).each do |matching_node|
    10
    11
    end
    12
    13
    14

    View full-size slide

  37. Adapter
    parser
    syntax_tree (under development)

    View full-size slide

  38. Synvert
    Synvert::Rewriter.new 'ruby', 'remove_debug_code' do
    within_files Synvert::ALL_RUBY_FILES do
    find_node '.send[receiver=nil][message IN (puts p)]' do
    remove
    end
    end
    end

    View full-size slide

  39. Check dependencies
    if_ruby '3.0.0'
    if_gem 'rails', '~> 6.0.0'

    View full-size slide

  40. Manipulate files
    add_file 'app/models/application_record.rb', <<~EOS
    class ApplicationRecord < ActiveRecord::Base
    self.abstract_class = true
    end
    EOS
    remove_file 'config/initializers/secret_token.rb'

    View full-size slide

  41. Add other snippets
    add_snippet 'minitest', 'assert_empty'
    add_snippet 'minitest/assert_instance_of'
    add_snippet '/Users/flyerhzm/.synvert-ruby/lib/minitest/assert_match.
    add_snippet 'https://github.com/xinminlabs/synvert-snippets-ruby/blob

    View full-size slide

  42. Support View Files (Erb, Haml, Slim)
    # <%= h user.login %>
    # =>
    # <%= user.login %>
    within_files Synvert::RAILS_VIEW_FILES do
    with_node type: 'send', receiver: nil, message: 'h' do
    replace_with '{{arguments}}'
    end
    end

    View full-size slide

  43. Synvert CLI
    synvert-ruby --run rspec/use_new_syntax
    synvert-ruby --run ~/.synvert-ruby/lib/rspec/use_new_syntax.rb
    synvert-ruby --run https://raw.githubusercontent.com/xinminlabs/synve

    View full-size slide

  44. Synvert GUI
    https://synvert.net/

    View full-size slide

  45. Automatically generate snippet

    View full-size slide

  46. List Snippets

    View full-size slide

  47. Use Synvert without writing any code
    Demo

    View full-size slide

  48. Compare with Rubocop

    View full-size slide

  49. Synvert::Rewriter.new 'ruby', 'perfer_not_nil' do
    within_files Synvert::ALL_RUBY_FILES do
    find_node '.send[message=!=][arguments.size=1][arguments.0=nil]' do
    replace_with '!{{receiver}}.nil?'
    end
    end
    end

    View full-size slide

  50. Synvert VSCode Extension
    https://synvert.net/

    View full-size slide

  51. Synvert Web (Playground)
    https://playground.synvert.net/

    View full-size slide

  52. Synvert Javascript
    new Synvert.Rewriter("typescript", "array-type", () => {
    configure({ parser: Synvert.Parser.TYPESCRIPT });
    // const z: Array = ['a', 'b'];
    // =>
    // const z: (string|number)[] = ['a', 'b'];
    withinFiles(Synvert.ALL_TS_FILES, function () {
    findNode(
    `.TypeReference
    [typeName.escapedText=Array]
    [typeArguments.0=.UnionType]`,
    () => {
    replaceWith("({{typeArguments}})[]");
    }
    );

    View full-size slide

  53. Video Tutorials
    https://synvert.substack.com/

    View full-size slide

  54. Contact Us
    https://synvert.net/contact_us

    View full-size slide