Find and Replace Code based on AST

This is my presentation on RubyKaigi 2023

synvert

Transcript

1. Find and Replace Code based on AST Richard Huang @flyerhzm
2. What is AST? Abstract Syntax Tree class RubyKaigi def year 2023 end def location 'Matsumoto' end end s(:class, s(:const, nil, :RubyKaigi), nil, s(:begin, s(:def, :year, s(:args), s(:int, 2023)), s(:def, :location, s(:args), s(:str, "Matsumoto"))))
3. Why based on AST? Precise # match puts puts "hello world" puts"hello world" puts("hello world") puts( "hello world" ) puts\ "hello world" # not match puts 'puts "hello world"' <<~HEREDOC puts "hello world" HEREDOC custom_puts "hello world"
4. Why based on AST? Precise Powerful Find hash pair where key and value are identical and delete the value Find old hash rocket syntax and replace with new hash syntax Find minitest setup methods that do not call super and prepend the super
5. What tools based on AST? Reek Rubocop Rufo
6. Synvert Write snippet code to rewrite your source code https://synvert.net/
7. Original Idea Simplify the process of upgrading a Rails project
8. RubyKaigi 2014 Write ruby code to change ruby code
10. New Idea Give everyone the ability to find and replace code based on AST
13. Node Query NQL - css like node query language Node Rules - hash object https://github.com/xinminlabs/node-query-ruby
14. Find debug code puts "hello world" p current_user # NQL '.send[receiver=nil][message IN (puts p)]' # Node Rules { node_type: 'send', receiver: nil, message: { in: %w[puts p] } }
15. Find two string arguments 'slug from title'.gsub(' ', '_') # NQL '.send[message=gsub] [arguments.size=2] [arguments.first=.str] [arguments.last=.str]' # Node Rules { node_type: 'send', message: 'gsub', arguments: { size: 2, first: { node_type: 'str' }, last: { node_type: 'str' } } }
16. Find hash rocket key { :foo => 'bar' } # NQL %q{.pair[key=.sym][key=~/\A:([^'"]+)\z/]} # Node Rules { node_type: 'pair', key: /\A:([^'"]+)\z/ }
17. Find hash pairs where the key and value are identical a: a, c: c, some_method( 1 2 b: b + 2, 3 4 d: d + 4 5 ) 6 # NQL '.pair[key="{{value}}"]' # Node Rules { node_type: 'pair', key: "{{value}}" }
18. Find minitest setup methods that do not call super def setup do_something end class TestMeme < Minitest::Test 1 2 3 4 end 5 # NQL '.class[parent_class=Minitest::Test] .def[name=setup]:not_has(> .super)' # Node Rules { node_type: 'class', parent_class: 'Minitest::Test' } # first step { node_type: 'def', name: 'setup' } # second step children.none? { |node| node.type == 'super' } # third step
19. Find hash value render nothing: true, status: :created render nothing: true 1 2 # NQL '.hash[status_value!=nil]' # Node Rules { node_type: 'hash', status_value: { not: nil } }
20. NQL rex racc
21. Node Mutation Provides a set of APIs to rewrite node source code https://github.com/xinminlabs/node-mutation-ruby
22. replace_with errors[:base] = "author not present" 1 2 errors.add(:base, "author not present") 3 replace_with node, '{{receiver}}.add({{arguments.0}}, {{arguments.1}})'
23. replace_with errors[:base] = "author not present" 1 2 errors.add(:base, "author not present") 3 errors.add(:base, "author not present") errors[:base] = "author not present" 1 2 3 replace_with node, '{{receiver}}.add({{arguments.0}}, {{arguments.1}})'
24. replace class Post < ActiveRecord::Base end 1 2 3 class Post < ApplicationRecord 4 end 5 replace node, :parent_class, with: 'ApplicationRecord'
25. replace class Post < ActiveRecord::Base end 1 2 3 class Post < ApplicationRecord 4 end 5 class Post < ApplicationRecord end class Post < ActiveRecord::Base 1 end 2 3 4 5 replace node, :parent_class, with: 'ApplicationRecord'
26. insert open('http://test.com') 1 2 URI.open('http://test.com') 3 insert node, 'URI.', at: 'beginning'
27. insert open('http://test.com') 1 2 URI.open('http://test.com') 3 URI.open('http://test.com') open('http://test.com') 1 2 3 insert node, 'URI.', at: 'beginning'
28. prepend def setup do_something end class TestMeme < Minitest::Test 1 2 3 4 end 5 6 class TestMeme < Minitest::Test 7 def setup 8 super 9 do_something 10 end 11 end 12 prepend node, 'super'
29. prepend def setup do_something end class TestMeme < Minitest::Test 1 2 3 4 end 5 6 class TestMeme < Minitest::Test 7 def setup 8 super 9 do_something 10 end 11 end 12 def setup super do_something end class TestMeme < Minitest::Test 1 def setup 2 do_something 3 end 4 end 5 6 class TestMeme < Minitest::Test 7 8 9 10 11 end 12 prepend node, 'super'
30. append class TestMeme < Minitest::Test def teardown clean_something end end 1 2 3 4 5 6 class TestMeme < Minitest::Test 7 def teardown 8 clean_something 9 super 10 end 11 end 12 append node, 'super'
31. append class TestMeme < Minitest::Test def teardown clean_something end end 1 2 3 4 5 6 class TestMeme < Minitest::Test 7 def teardown 8 clean_something 9 super 10 end 11 end 12 class TestMeme < Minitest::Test def teardown clean_something super end end class TestMeme < Minitest::Test 1 def teardown 2 clean_something 3 end 4 end 5 6 7 8 9 10 11 12 append node, 'super'
32. delete FactoryBot.create(:user) 1 2 create(:user) 3 delete node, :receiver, :dot
33. delete FactoryBot.create(:user) 1 2 create(:user) 3 create(:user) FactoryBot.create(:user) 1 2 3 delete node, :receiver, :dot
34. remove puts( "hello"\ "world" ) 1 2 3 4 5 # nothing 6 remove node
35. remove puts( "hello"\ "world" ) 1 2 3 4 5 # nothing 6 # nothing puts( 1 "hello"\ 2 "world" 3 ) 4 5 6 remove node
36. Hash node after_commit :add_to_index_later, on: :create 1 2 after_create_commit :add_to_index_later 3 replace node, :message, with: 'after_{{arguments.-1.on_value.to_value}}_commit' delete node, 'arguments.-1.on_pair', and_comma: true
37. Hash node after_commit :add_to_index_later, on: :create 1 2 after_create_commit :add_to_index_later 3 after_create_commit :add_to_index_later after_commit :add_to_index_later, on: :create 1 2 3 replace node, :message, with: 'after_{{arguments.-1.on_value.to_value}}_commit' delete node, 'arguments.-1.on_pair', and_comma: true
38. node_query + node_mutation require 'parser/current' require 'parser_node_ext' require 'node_query' require 'node_mutation' 1 2 3 4 5 source = "foo\nputs foo\n" 6 node = Parser::CurrentRuby.parse(source) 7 query = NodeQuery.new('.send[receiver=nil][message IN (p puts)]') 8 mutation = NodeMutation.new(source) 9 query.query_nodes(node).each do |matching_node| 10 mutation.remove(matching_node) 11 end 12 result = mutation.process 13 result.new_source # foo\n 14
39. node_query + node_mutation require 'parser/current' require 'parser_node_ext' require 'node_query' require 'node_mutation' 1 2 3 4 5 source = "foo\nputs foo\n" 6 node = Parser::CurrentRuby.parse(source) 7 query = NodeQuery.new('.send[receiver=nil][message IN (p puts)]') 8 mutation = NodeMutation.new(source) 9 query.query_nodes(node).each do |matching_node| 10 mutation.remove(matching_node) 11 end 12 result = mutation.process 13 result.new_source # foo\n 14 source = "foo\nputs foo\n" node = Parser::CurrentRuby.parse(source) query = NodeQuery.new('.send[receiver=nil][message IN (p puts)]') query.query_nodes(node).each do |matching_node| end require 'parser/current' 1 require 'parser_node_ext' 2 require 'node_query' 3 require 'node_mutation' 4 5 6 7 8 mutation = NodeMutation.new(source) 9 10 mutation.remove(matching_node) 11 12 result = mutation.process 13 result.new_source # foo\n 14
40. node_query + node_mutation require 'parser/current' require 'parser_node_ext' require 'node_query' require 'node_mutation' 1 2 3 4 5 source = "foo\nputs foo\n" 6 node = Parser::CurrentRuby.parse(source) 7 query = NodeQuery.new('.send[receiver=nil][message IN (p puts)]') 8 mutation = NodeMutation.new(source) 9 query.query_nodes(node).each do |matching_node| 10 mutation.remove(matching_node) 11 end 12 result = mutation.process 13 result.new_source # foo\n 14 source = "foo\nputs foo\n" node = Parser::CurrentRuby.parse(source) query = NodeQuery.new('.send[receiver=nil][message IN (p puts)]') query.query_nodes(node).each do |matching_node| end require 'parser/current' 1 require 'parser_node_ext' 2 require 'node_query' 3 require 'node_mutation' 4 5 6 7 8 mutation = NodeMutation.new(source) 9 10 mutation.remove(matching_node) 11 12 result = mutation.process 13 result.new_source # foo\n 14 source = "foo\nputs foo\n" mutation = NodeMutation.new(source) mutation.remove(matching_node) result = mutation.process result.new_source # foo\n require 'parser/current' 1 require 'parser_node_ext' 2 require 'node_query' 3 require 'node_mutation' 4 5 6 node = Parser::CurrentRuby.parse(source) 7 query = NodeQuery.new('.send[receiver=nil][message IN (p puts)]') 8 9 query.query_nodes(node).each do |matching_node| 10 11 end 12 13 14
41. Adapter parser syntax_tree (under development)
42. Synvert Synvert::Rewriter.new 'ruby', 'remove_debug_code' do within_files Synvert::ALL_RUBY_FILES do find_node '.send[receiver=nil][message IN (puts p)]' do remove end end end
43. Check dependencies if_ruby '3.0.0' if_gem 'rails', '~> 6.0.0'
44. Manipulate files add_file 'app/models/application_record.rb', <<~EOS class ApplicationRecord < ActiveRecord::Base self.abstract_class = true end EOS remove_file 'config/initializers/secret_token.rb'
45. Add other snippets add_snippet 'minitest', 'assert_empty' add_snippet 'minitest/assert_instance_of' add_snippet '/Users/flyerhzm/.synvert-ruby/lib/minitest/assert_match. add_snippet 'https://github.com/xinminlabs/synvert-snippets-ruby/blob
46. Support View Files (Erb, Haml, Slim) # <%= h user.login %> # => # <%= user.login %> within_files Synvert::RAILS_VIEW_FILES do with_node type: 'send', receiver: nil, message: 'h' do replace_with '{{arguments}}' end end
47. Synvert CLI synvert-ruby --run rspec/use_new_syntax synvert-ruby --run ~/.synvert-ruby/lib/rspec/use_new_syntax.rb synvert-ruby --run https://raw.githubusercontent.com/xinminlabs/synve
49. Synvert GUI https://synvert.net/
50. Run Snippet
51. Automatically generate snippet
53. List Snippets
55. Use Synvert without writing any code Demo
56. Compare with Rubocop
58. Synvert::Rewriter.new 'ruby', 'perfer_not_nil' do within_files Synvert::ALL_RUBY_FILES do find_node '.send[message=!=][arguments.size=1][arguments.0=nil]' do replace_with '!{{receiver}}.nil?' end end end
59. Synvert VSCode Extension https://synvert.net/
61. Synvert Web (Playground) https://playground.synvert.net/
62. Synvert Javascript new Synvert.Rewriter("typescript", "array-type", () => { configure({ parser: Synvert.Parser.TYPESCRIPT }); // const z: Array<string|number> = ['a', 'b']; // => // const z: (string|number)[] = ['a', 'b']; withinFiles(Synvert.ALL_TS_FILES, function () { findNode( `.TypeReference [typeName.escapedText=Array] [typeArguments.0=.UnionType]`, () => { replaceWith("({{typeArguments}})[]"); } );
63. Video Tutorials https://synvert.substack.com/
64. Contact Us https://synvert.net/contact_us
65. Thank You!