RBS meets LLMs - Type inference using LLM

RubyKaigi 2023での発表資料です。以下、公式サイトに掲載した概要となります。

---

Recently, Large Language Models (LLMs), such as ChatGPT, have been rapidly evolving. LLMs are characterized by their ability to interpret word meanings and contexts, and by their very wide range of applications.

By the way, common static analyzers and execution systems treat class and variable names as mere labels, but they are important clues when humans read code. By using LLMs, we can interpret those intentions and use them for type inference.

Also, static analysis is not good at dynamic definitions such as define_method, which cannot be distinguished from a simple method call in Ruby syntax. LLMs may be able to successfully generate type definitions in these cases.

In this session, we will introduce our efforts to use ChatGPT for RBS generation and discuss the possibility of incorporating LLMs into development tools.

Transcript

1. RBS meets LLMs Leaner Technologies, Inc. 黒曜 (@kokuyouwind) ~ Type inference using LLM ~
2. Talk: Japanese Slides: English (+ Japanese)
3. $ whoami 黒曜 / @kokuyouwind Work at: Leaner Technologies, Inc. Ruby Sponsor Day 1: Sponsor Talk (Done) Day 2: Leaner Drinkup
4. The idea is covered by Matz's Keynote, but this is the first time you've heard of it! Matz のKeynote で出た話と発想が被っていますが、 初めて聞いたことにしてください！
5. Large Language Model (LLM) Machine learning models trained on large amounts of text data. ChatGPT Bard Bing Chat ⼤量のテキストデータを使ってトレーニングされた機械学習モデル ⼤規模⾔語モデル
6. Example: ChatGPT
7. Can we do something fun with this? 🤔 なにか⾯⽩いことに使えないかな？
8. Type Inference user = User.new # `user` is an User type variable name = user.name # `name` is a String type variable(?) 1 2 3 4 5 Humans sometimes infer type from word meanings. ⼈間は単語の意味から型を推測することがある
9. With ChatGPT, can we infer the type from the meaning of a word? ChatGPT を使えば、単語の意味から 型を推測できるのでは？
10. Let's see some cases! Tool Settings model: ChatGPT(gpt-3.5-turbo) Temperature: 0 GUI: BetterChatGPT
11. Case 1: User Builder user = UserBuilder.new.name('kokuyouwind').build # We infer types as follows: # * UserBuilder#name returns an UserBuilder # * UserBuilder#build returns an User 1 2 3 4 5
12. Case 1: User Builder
13. Case 1: User Builder
14. (There is extra output, but) Perfect!!! 👏
15. Case 2: Company Repository company = CompanyRepository.new.find(1).name # We infer types as follows: # * CompanyRepository#find returns a Company # * Company#name returns a String # Note. # This expression has the same syntactic form with: # user = UserBuilder.new.name('kokuyouwind').build 1 2 3 4 5 6 7 8 9
16. Case 2: Company Repository
17. (There is extra output, but) Perfect, again!!! 👏
18. Problems RBS sometimes broken ( たまにRBS の構⽂がおかしい) Extra Outputs ( 不要な出⼒がある) Only RBS output needed (RBS だけ出⼒させたい) Single RBS (RBS がひとまとめになっている) Separate User's RBS to user.rbs, and so on (User のRBS はuser.rbs に、などクラスごとに分割したい)
19. Improve: FewShot Demonstrate type inference for UserBuilder, then ask about the CompanyRepository case. UserBuilder の模範解答を⼊⼒してから、CompanyRepository について推論させる Demo Input Main Question Output 模範解答 メインの問い合わせ 出⼒
20. Improve: FewShot Demo Input Main Question 模範解答を⼊⼒ メインの問い合わせ
22. Really Perfect!!! 👏
23. Please talk to me at parties, etc. ぜひドリンクアップなどで話しかけてください！ I've tried to infer types for metaprogramming, refine existing RBS, etc., but I don't have time to talks. So, メタプログラミングコードに対する型の推測や既存RBS の詳細化なども試しましたが 今回は話す時間が⾜りないので、
24. Future Works CommandLine Tool ( コマンドラインツール化) Fine tuning for type inference ( 型推測⽤のファインチューニング) Agent-style autonomous drive ( エージェントスタイルの⾃律駆動) Refer to the necessary files ( 必要なファイルの参照) Correction with type check results ( 型チェックを元にした修正) Run with LLMs locally ( ローカルで動くLLM を使った動作) Alpaca.cpp, ChatRWKV, or else