IntelliJ Elixir Works Even if your Code Doesn't

Abstract:

IntelliJ Elixir is the Elixir plugin for JetBrains IDEs. It is built around the assumption that your editor should work even your code doesn’t compile. By statically analyzing code using its own parser, it is able to resolve and complete code in broken files. Go To Definition and Find Usages show paths through uses, imports, and macros that are thrown out by the Elixir compiler, which allows you to understand macro heavy code. Heuristics find definitions in DSL for Phoenix and Ecto and quote blocks.

Talk objectives:

Developers will learn how IntelliJ Elixir can be used to understand their code easier and faster even in the face of errors or gnarly macros.

Target audience:

Developers looking for a richer editor/IDE experience or struggling with complex code bases. Anyone that wants to see how source becomes .beams or macros inject code.

Transcript

1. IntelliJ Elixir Works Even if your Code Doesn't Elle Imho ff Principal Software Engineer at DockYard, Inc. Hi, I'm Elle Imho ff . 
 I'm known across the internet as KronicDeth because I'm full of diseases! I got new ones now and they're not all auto-immune anymore! I'm a member of the Lumen Core team, a Principal Software Engineer at DockYard, and most importantly for this talk, the creator of the IntelliJ Elixir plugin.
2. INTELLIJ ELIXIR WHAT IS INTELLIJ ELIXIR? IDE Project Language Add-on IntelliJ Elixir Any AppCode Objective-C Elixir AppCode Swift Elixir CLion C Elixir CLion C++ Elixir DataGrip SQL Elixir GoLand Go Elixir PHPStorm PHP Elixir PyCharm Python Elixir Rider .NET Elixir RubyMine Ruby Elixir WebStorm JS Elixir IntelliJ Elixir is a plugin for JetBrains IDEs: IntelliJ (Community and Ultimate Edition), of course, which supports any language, but it also works for those coming from an Objective-C background using AppCode, C/C++ using CLion, SQL using DataGrip, Go using GoLand, PHP using PHPStorm, Python using PyCharm, .NET using Rider, Ruby using Rubymine, or JS using WebStorm.
3. INTELLIJ ELIXIR WHEN IS INTELLIJ ELIXIR? •Started 2014-07-27 •87 releases •Updates every 33 days •Last update 2022-06-01 In the almost 8 years since I started the project, I've released 87 updates, which averages out to 33 days between updates with the last update only 9 days ago on June 1st.
4. INTELLIJ ELIXIR WHY IS INTELLIJ ELIXIR? •Used RubyMine for Ruby development •Wanted vim key bindings •Wanted Cmd+Click Go To De fi nition for Elixir •Wanted Search Everywhere for Elixir I originally considered starting plugin development after seeing Dave Thomas talk about Elixir at Lone Star Ruby and it was 2 days after the fi rst Elixir Conf that I did the fi rst commit. I thought: "Elixir's syntax is similar to Ruby's and RubyMine exists, so it can't be that hard to make an Elixir plugin".
5. INTELLIJ ELIXIR HOW IS INTELLIJ ELIXIR? •Custom lexer using JFlex •Custom parser using GrammarKit •Built on top of JetBrains' Open API Unlike the other editors that use the Elixir Language Server, the IntelliJ Elixir plugin actually predates ElixirLS by 3 years. IntelliJ Elixir has a custom lexer using JFlex, a custom parser using GrammarKit, and uses the JetBrains OpenAPI to deliver features you'd expect in JetBrain IDEs. This means that I'm writing Java and Kotlin to analyze Elixir. There turns out to be almost a 3rd of the project in Elixir due to the extensive testing. Early in the project my grammar and lexer were actually revealing bugs and inconsistencies in the lexer and parser in Elixir itself. The independent lexer and parser also means that code analysis works even if your code won't compile!
6. JETBRAINS OPEN API CODE IN MARKDOWN •Start blocks with language tag:
 ```elixir Why is building on JetBrains OpenAPI useful? You get cool features for free like integration with the Markdown support when you use language tagged code blocks
7. CODE IN MARKDOWN SYNTAX HIGHLIGHTING So, when writing your guides syntax highlighting still works in the code blocks
8. CODE IN MARKDOWN PARAMETER AND VARIABLE MATCHING Parameter highlighting still works
9. CODE IN MARKDOWN COMPLETION Completion still works
10. CODE IN MARKDOWN GO TO DEFINITION Go To De fi nition even works.
11. GO TO DEFINITION WAIT... WHY DOES THAT WORK? As you can see the code block, `Repo` isn't aliased. You as a human reading the code know that Repo means your app's MyApp.Repo, but the code itself isn't valid as Repo isn't a top-level module, Ecto.Repo is, so why does the plugin fi nd a result? It's again because of JetBrains Open API. Part of the resolving system in the API is that a result can be marked as valid or invalid. Invalid matches are a crucial concept in editors as you want completion and Go To Declaration to work while you're typing.
12. GO TO DEFINITION VALID AND INVALID MATCHES Module Name Typing? Arity Interval Match Exact Unknown Yes Out of Interval Invalid Exact Unknown Don't Care In Interval Invalid Exact No Pre fi x Don't Care Don't Care No Exact Pre fi x Don't Care Don't Care Invalid Exact Exact Don't Care Out of Interval Invalid Exact Exact Don't Care In Interval Valid Any No Pre fi x Don't Care Don't Care No Any Pre fi x Yes Don't Care Invalid Any Pre fi x No Don't Care No Any Exact Don't Care Out of Interval No Any Exact Don't Care In Interval Invalid The matching logic needs to handle whether the module matches, how well the name matches, which includes unknown names for derived function names, whether the user is typing, and if the call is in the arity interval of the de fi nition. I say arity interval and not range because the special forms like `with` take an unlimited number of arguments, which for `with` is an interval from 1 to in fi nity. If there are no valid Exact module matches, then IntelliJ Elixir uses the stub index to fi nd any matches that match the name or arity regardless of module.
13. GO TO DEFINITION GO TO INVALID MATCHES IN ANY MODULE So we can navigate from Markdown even without the `alias` because the indexes built of the project let us fi nd invalid matches with the same name and arity even if in the wrong module.
14. GO TO DEFINITION GO TO CALLBACK If we go to the second entry you'll see that what IntelliJ Elixir counts as a de fi nition is more than just that which uses def or defmacro: it can resolves callbacks too!
15. INTELLIJ ELIXIR FEATURES •BEAM Files •Builder •Code Folding •Commenter •Completion •Credo Integration •Debugger •Dialyzer Integraton •EEx Templates •Find Usages •Formatting •Go To Declaration/Related/Symbol •Inspections •Live Template •Quick Fixes •Refactor •Run/Debug Con fi guration •Show Parameters •SDK •Structure •Syntax Highlighting IntelliJ Elixir has tons of features that I've implemented in the past 8 years, but that's too much to talk about in the time we have here today, so...
16. INTELLIJ ELIXIR ☣ CHANGES Releases 19 Commits 1015 Days 859 ... let's cover what's new since the last time I spoke in February 2020. It should be no problem, it's only 19 releases over 2 years and we have less than 40 minutes!
17. INTELLIJ ELIXIR V11.5.0 •More completions for invalid matches •Debugger expression evaluator in Elixir 1.9.2 As hinted at in the explanation of Go To De fi nition, IntelliJ Elixir fi nds as many de fi nitions as it can that could be related to the current call. Version 11.5.0 extended that...
18. V11.5.0 MORE COMPLETIONS FOR INVALID MATCHES •Quali fi ed calls (Process.flag(...)) •Type Specs (@spec foo(...)) •Aliases (alias Foo...) •Module Attributes (@attribute) •Variables (foo) •Protocol reserved module attributes (@protocol and @for) ... with quali fi ed calls, specs, aliases, module attributes, and variables with special support for the @protocol and @for module attributes de fi ned implicitly in Protocols. Expanding support in the plugin reminds me a lot of how exploit development works, you fi nd one bug in a system and then you keep exploring around the area to fi nd similar problem.
19. V11.5.0 DEBUGGER EXPRESSION EVALUATOR IN ELIXIR 1.9.2 •elixir_erl record changed José Valim has said that the debugger and IEx API are unsupported and not public, but that I can use it, so I do, but that does mean chasing after changes made to the internals. In Elixir 1.9.2, the elixir_erl record changed. Because records are just tuples and not maps, any change makes the code I wrote in the debugger break, so..
20. V11.5.0 DEBUGGER EXPRESSION EVALUATOR IN ELIXIR 1.9.2 ... I needed to change my version of it in the debug server. It's not guaranteed that the SDK shipped with the Erlang header fi les and not just the `.beam` fi les, so I can't depend on the same headers as Elixir itself.
21. INTELLIJ ELIXIR V11.6.0 •LEEx highlighting like EEx Version 11.6.0 was released in the long-long ago of April 2020, so before HEEx was part of Phoenix LiveView, so LEEx highlighting was added.
22. V11.6.0 LEEX HIGHLIGHTING LIKE EEX How LEEx and EEx works is once again based on JetBrains OpenAPI. In this case, support for template languages.
23. LEEX HIGHLIGHTING LIKE EEX A SMALL CHANGE FOR EEX The original EEx support was a simple and small change...
24. A SMALL CHANGE FOR EEX EEX GRAMMAR ... a lot of that change was regenerating the Elixir parser to support interleaving EEx template tags, but EEx as a language itself is really simple. It's either DATA or a tag. LEEx is the same since the actual syntax is not di ff erent, only the semantics and how it is compiled.
25. A SMALL CHANGE FOR EEX DATA IN TEMPLATEDATA That DATA tag is then hooked up as a TemplateDataElementType, TemplateData, which...
26. A SMALL CHANGE FOR EEX TEMPLATEDATA ...is used in...
27. A SMALL CHANGE FOR EEX TEMPLATEDATA IN ELEMENTTYPE ... getting the ViewProvider elementType
28. A SMALL CHANGE FOR EEX VIEWPROVIDER ELEMENTTYPE The ViewProvider elementType is then...
29. A SMALL CHANGE FOR EEX VIEWPROVIDER CREATEFILE ... called to create the fi le, but why does it know that the `.html.leex` fi le is HTML in addition to LEEx?!
30. A SMALL CHANGE FOR EEX VIEWPROVIDER Well, that happens when the ViewProvider is created. It calls templateDataLanguage
31. A SMALL CHANGE FOR EEX VIEWPROVIDER ...
32. A SMALL CHANGE FOR EEX VIEWPROVIDER There a lot of stu ff about caching here
33. A SMALL CHANGE FOR EEX VIEWPROVIDER ... and fallbacks if the template language isn't known, but ...
34. A SMALL CHANGE FOR EEX VIEWPROVIDER The actual logic of fi nding the template language from the fi le
35. A SMALL CHANGE FOR EEX VIEWPROVIDER Let's focus on onlyTemplateDateFileType
36. A SMALL CHANGE FOR EEX EEX FILE TYPE .. which is in the EEx File Type
37. A SMALL CHANGE FOR EEX EEX FILE TYPE This function is making sure there is only a single known fi le type, so let's see if how the set of fi le types are found...
38. A SMALL CHANGE FOR EEX EEX FILE TYPE In templateDataFileTypeSet
39. A SMALL CHANGE FOR EEX EEX FILE TYPE TemplateDataFileTypeSet fi nds the fi le types associated with the virtualFIle.
40. A SMALL CHANGE FOR EEX EEX FILE TYPE ... then gets those fi le types that have a fi xed extension...
41. A SMALL CHANGE FOR EEX EEX FILE TYPE ... that are then removed from the path...
42. A SMALL CHANGE FOR EEX EEX FILE TYPE ... and fi nally checking if that new fi le name has a known fi le type
43. A SMALL CHANGE FOR EEX EEX FILE TYPE 1.Start with full fi le name index.html.eex 2..eex is the EEx fi le type 3.Strip .eex: index.html 4..html is the HTML fi le type 5.HTML is the template data language How would this work for a real fi le? Say we have an `index.html.eex`, it's an EEx fi le type, which has `.eex` as its extension. The code strips that extension o ff , leaving `index.html`, which matches the HTML language, so...
44. A SMALL CHANGE FOR EEX HTML FROM JETBRAINS ❤ EEX FROM INTELLIJ ELIXIR ... we get HTML support developed by JetBrains mixed with the EEx I developed as you can see from the CSS selector breadcrumbs
45. A SMALL CHANGE FOR EEX SVG FROM JETBRAINS ❤ EEX FROM INTELLIJ ELIXIR Because I don't hard code the supported template languages you can template stu ff besides HTML, like SVG
46. A SMALL CHANGE FOR EEX ERL FROM INTELLIJ ERLANG ❤ EEX FROM INTELLIJ ELIXIR It also means it works with language support from other plugins, say templating Erlang source from the IntelliJ Erlang plugin using EEx.
47. INTELLIJ ELIXIR V11.8.0 •Quick Documentation •Gutter Icons for ExUnit Version 11.8.0 was a great release because all the features were implemented by community contributors.
48. V11.8.0 QUICK DOCUMENTATION •Implemented by Marcin Dawidziuk (@marcindawidziuk) Quick Documentation is the docs that pop up as hovers over the call being documented.
49. QUICK DOCUMENTATION ACTIVATING •F1 on keyboard You can trigger it with F1, but since not all of you are lucky enough to need a Kinesis Advantage 2, which has function keys, for RSI like I do, you can also activate it from the menu: go to View > Quick Documentation and you'll get the docs you're used to seeing in Hex docs.
50. V11.8.0 GUTTER ICONS FOR EXUNIT •Implemented by Niko Strijbol The gutter icons are a single play button for individual tests and fast forward buttons for groups like modules and describe. If you click on one of the icons it will run the module, describe, or test.
51. GUTTER ICONS FOR EXUNIT RESULTS The icons for groups or tests that pass will remain the same, but for tests that have errors, such as this test that incorrectly is checking for the "Sign out" message when it should be "Sign in", then the error is marked with an alert exclamation point behind the play button to re-run the test.
52. INTELLIJ ELIXIR V11.8.1 •JInterface 1.11 •Graphical Debugger on Windows Version 11.8.1 had graphical debugger fi xes.
53. V11.8.1 GRAPHICAL DEBUGGER COMMUNICATION PROTOCOL •No Pipe •Uses Distributed Erlang •IDE is a Java Node •Debugged is normal BEAM Node The graphical debugger in IntelliJ Elixir doesn't communicate with the BEAM using a pipe, but instead uses Distributed Erlang with the IDE acting as one node.
54. V11.8.1 JINTERFACE •Can't depend on SDK •Can't depend on build •Use Java Package from Maven To make a Java Node, the IDE needs to load the JInterface library. However, most package managers don't install JInterface and by default the erlang build doesn't build JInterface. It's one of those thing like SSL, XML, and doc support that it will silently skip building if it can't fi nd the dependency. So, to be a good Java application I had Jinterface pulling from the public Maven repository. Maven is like Hex, but for Java.
55. V11.8.1 JINTERFACE 1.6.1 💔 OTP 19.0 •BIG_CREATION •NEW_PID_EXT •NEW_PORT_EXT •NEWER_REFERENCE_EXT Jinterface doesn't get updated as often as Erlang itself because it only needs to change when the Distributed Erlang protocol changes. In OTP 19.0, the protocol changed to support something called BIG_CREATION. BIG_CREATION was a future proo fi ng to allow changes to how PIDs, Ports, and References were transmitted.
56. V11.8.1 JINTERFACE 1.6.1 💔 OTP 19.0 •BIG_CREATION •NEW_PID_EXT •NEW_PORT_EXT •NEWER_REFERENCE_EXT 🚫 The important point was that it was unused in OTP 19.0.
57. V11.8.1 MAVEN CENTRAL JINTERFACE •OTP Team didn't maintain it! •Basho maintained it! So even though this incompatibility existed since June of 2016, it didn't actually break my usage of JInterface 1.6.1 until May 2020 when OTP 24 was out. It turns out Jinterface wasn't being updated on Maven Centeral because the OTP team wasn't the publisher: Basho was and they were just putting the OTP team data in it the package information since Bash only did the build and publish
58. V11.8.1 BASHO •Creators of Riak •Fell over in 2017 ... oh, but Basho fell over in 2017, which explains why their publishing of Jinterface stopped. So, IntelliJ Elixir happened to start using the Maven Central repository just a few months before Riak fell over when Jake Becker converted the build from Ant to Gradle. I'd like to thank Jake because he's also the contributor that fi rst added the Graphical Debugger to IntelliJ Elixir before moving on to all his work on ElixirLS.
59. V11.8.1 JINTERFACE 1.11 ❤ OTP 24 🚫 OTP Team 🚫 Basho 
 🤦 Me So, OTP team doesn't want it and Basho can't do it, so I'm back to maintaining JInterface myself, I built it against OTP 24.0.3 and that's what's still being used to this day.
60. INTELLIJ ELIXIR V11.9.0 •Documentation Provider Performance Improvements In version 11.9.0, after only adding the documentaton provider in 11.8.0 Marcin came back and improved the performance
61. V11.9.0 DOCUMENTATION PROVIDER PERFORMANCE IMPROVEMENTS •Fast enough for mouse over •Read .beam or source directly (no more mix) •EEP-48 The performance was improved by not calling mix in another process, but instead reading the `.beam` fi les directly using the BEAM handling originally implemented for the decompiler.
62. DOCUMENTATION PROVIDER PERFORMANCE IMPROVEMENTS EEP-48 EEP-48 is the standard that introduced the Docs chunk to the BEAM format.
63. DOCUMENTATION PROVIDER PERFORMANCE IMPROVEMENTS EEP-48 The data is stored in a tuple. Remember this is an _Erlang_ Enhancement Proposal so lowercase words are atoms and titlecase words are variables.
64. EEP-48 "DOCS" FORMAT The Metadata portion gives us some extra tags to apply to docs, like deprecated and since.
65. DOCUMENTATION PROVIDER PERFORMANCE IMPROVEMENTS RENDERED METADATA That deprecated is rendered as a separate section
66. DOCUMENTATION PROVIDER PERFORMANCE IMPROVEMENTS USE DOCS FOR BETTER DECOMPILATION Since the faster documentation provider had to read the Docs chunks anyway, Marcin also added those docs to the decompiled code!
67. INTELLIJ ELIXIR V11.9.2 •Decompiler Enhancements •Fix for a 6 year old 🤦 bug In 11.9.2 there were even more decompiler enhancements
68. V11.9.2 DECOMPILER ENHANCEMENTS = HIGHLIGHTING BUGS 1.Decompile documentation with """ in it 2.Kill parsing of that decompiled module 3.Kill highlighting that references that module The addition of new features often lead to new bugs in old features due to new interactions, in the case of the Quick Documentation adding docs to the decompiled code, it meant that errors in the docs could break parsing those decompiled fi les, which meant it couldn't resolve references to those modules.
69. V11.9.2 MODULE PARSING FAILURE = HIGHLIGHTING BUGS This a ff ected Module, which is referenced by a lot of stu ff . So the example @doc in the @moduledoc for Module made all of Module's functions and macros unparseable
70. V11.9.2 MODULE PARSING FAILURE = HIGHLIGHTING BUGS So why wasn't this a problem in the original source for Module?
71. V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK Well it uses the capital S sigil and triple single quotes to avoid needing to escape the inner double quotes. The Docs chunk doesn't preserve that information since it's just a binary string that all BEAM languages can understand on purpose.
72. V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK So, what's the fi x? We need to calculate a safe promoter and terminator. The promoter is the symbol that start a quote while the terminator is the symbol that ends a quote. Sometimes they are the same, but sometimes it's matched curly braces. I got those terms from genetics because I couldn't think of the opposite of a terminator in computer science when writing the plugin.
73. V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK The safePromoterTerminator calculates the safe promoter...
74. V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK By checking for either triple single or double quotes
75. V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK If it has both nothing is safe!
76. V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK But if it has only one, then the other is safe
77. V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK If it has neither, it just defaults to triple double quotes
78. V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK With the safe promoter found or no safe promoter terminator at all, we use that to choose the outer promoter terminator.
79. V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK Then we use the the safePromoterTerminator to reformat the original text when appending
80. V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK ...
81. V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK Only when we don't have a safe promoter terminator do we actually need to reformat the documentation text to escape the existing triple double quotes, which fi xes the docs and the parsing and the references.
82. V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG Tomoki Odaka, long time excellent bug reporter and sponsor of the project noticed that sometimes the highlighting would just give up while typing invalid code that was then fi xed. You want to know how good of a bug reporter Tomoki is? Tomoki links to similar bugs, gives reproduction steps and screenshots, but wait...
83. V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG .... That's not just a picture, the screenshot is a movie!
84. V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG So even though the map on line 3 gets highlighted correctly, line two never recovers.
85. V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG 1.Lexer parses code: Map.put(%{}, :) 2.Parser fi nds no matching element for :) 3.Parser tells lexer to advance to past : to ) Part of the reason I originally picked using JetBrain's lexing and parsing libraries instead of Elixir is that it has automatic error recovery: the parser will automatically shift the lexing around an error, but there's a hesitation and users can see the lexer and parser take time to recovery. You can see the editor still thinks the code is messed up with the red squiggle even though the code is valid and it triggers an error report
86. V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG 1.Lexer parses code: Map.put(%{}, :) 2.Parser fi nds no matching element for :) 3.Parser tells lexer to advance to past : to ) 4.If the lexer is in YYINITIAL state... It's ok to restore from that point! So, why that delay where it think there's still an error in the code? It turns out there are some unwritten rules about how the parser uses the lexer: if the lexer is in it's initial state then it is OK to only rewind the lexer to that point in the fi le.
87. V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG State Stack Token Text Token Text Remaining YYINITIAL Empty " STRING_PROMOTER #{{}}" GROUP 1.YYINITIAL #{ INTERPOLATION_START {}}" INTERPOLATION 1.GROUP 2.YYINITIAL { OPENING_CURLY }}" MULTILINE_WHITE_SPACE_MAYBE 1.YYINITIAL 2.INTERPOLATION 3.GROUP 4.YYINITIAL None None }}" YYINITIAL 1.INTERPOLATION 2.GROUP 3.YYINITIAL } CLOSING_CURLY }" ADDITION_OR_SUBTRACTION_MAYBE 1.INTERPOLATION 2.GROUP 3.YYINITIAL None None }" INTERPOLATION 1.GROUP 2.YYINITIAL } INTERPOLATION_END " GROUP 1.YYINITIAL " STRING_TERMINATOR YYINITIAL Empty Why does that cause a problem? Well, whenever a `{` is encountered, this might be in interpolation, but we don't want a closing curly brace to close an interpolation if it's actually closing a tuple
88. V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG State Stack Token Text Token Text Remaining YYINITIAL Empty " STRING_PROMOTER #{{}}" GROUP 1.YYINITIAL #{ INTERPOLATION_START {}}" INTERPOLATION 1.GROUP 2.YYINITIAL { OPENING_CURLY }}" MULTILINE_WHITE_SPACE_MAYBE 1.YYINITIAL 2.INTERPOLATION 3.GROUP 4.YYINITIAL None None }}" YYINITIAL 1.INTERPOLATION 2.GROUP 3.YYINITIAL } CLOSING_CURLY }" ADDITION_OR_SUBTRACTION_MAYBE 1.INTERPOLATION 2.GROUP 3.YYINITIAL None None }" INTERPOLATION 1.GROUP 2.YYINITIAL } INTERPOLATION_END " GROUP 1.YYINITIAL " STRING_TERMINATOR YYINITIAL Empty When in the initial state, YYINITIAL the closing curly brace needs to count as just a closing curly brace for a tuple or map, but...
89. V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG State Stack Token Text Token Text Remaining YYINITIAL Empty " STRING_PROMOTER #{{}}" GROUP 1.YYINITIAL #{ INTERPOLATION_START {}}" INTERPOLATION 1.GROUP 2.YYINITIAL { OPENING_CURLY }}" MULTILINE_WHITE_SPACE_MAYBE 1.YYINITIAL 2.INTERPOLATION 3.GROUP 4.YYINITIAL None None }}" YYINITIAL 1.INTERPOLATION 2.GROUP 3.YYINITIAL } CLOSING_CURLY }" ADDITION_OR_SUBTRACTION_MAYBE 1.INTERPOLATION 2.GROUP 3.YYINITIAL None None }" INTERPOLATION 1.GROUP 2.YYINITIAL } INTERPOLATION_END " GROUP 1.YYINITIAL " STRING_TERMINATOR YYINITIAL Empty ... when in the INTERPOLATION state it needs to count as the end of the interpolation
90. V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG State Stack Token Text Token Text Remaining YYINITIAL Empty " STRING_PROMOTER #{{}}" GROUP 1.YYINITIAL #{ INTERPOLATION_START {}}" INTERPOLATION 1.GROUP 2.YYINITIAL { OPENING_CURLY }}" MULTILINE_WHITE_SPACE_MAYBE 1.YYINITIAL 2.INTERPOLATION 3.GROUP 4.YYINITIAL None None }}" YYINITIAL 1.INTERPOLATION 2.GROUP 3.YYINITIAL } CLOSING_CURLY }" ADDITION_OR_SUBTRACTION_MAYBE 1.INTERPOLATION 2.GROUP 3.YYINITIAL None None }" INTERPOLATION 1.GROUP 2.YYINITIAL } INTERPOLATION_END " GROUP 1.YYINITIAL " STRING_TERMINATOR YYINITIAL Empty If the lexer is in YYINITIAL state, it's ok to restore from that point! So if we look, when we're in the middle of interpolation, we're in YYINITIAL again, so when we combine that with assumption that Open API makes that YYINITIAL is OK to resume from, we can resume from the middle of interpolation when there is an error.
91. V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG What's the problem with resuming from there? The reset function doesn't restore the stack!
92. V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG State Stack Token Text Token Text Remaining YYINITIAL Empty " STRING_PROMOTER #{{}}" GROUP 1.YYINITIAL #{ INTERPOLATION_START {}}" INTERPOLATION 1.GROUP 2.YYINITIAL { OPENING_CURLY }}" MULTILINE_WHITE_SPACE_MAYBE 1.YYINITIAL 2.INTERPOLATION 3.GROUP 4.YYINITIAL None None }}" YYINITIAL 1.INTERPOLATION 2.GROUP 3.YYINITIAL } CLOSING_CURLY }" ADDITION_OR_SUBTRACTION_MAYBE 1.INTERPOLATION 2.GROUP 3.YYINITIAL None None }" INTERPOLATION 1.GROUP 2.YYINITIAL } INTERPOLATION_END " GROUP 1.YYINITIAL " STRING_TERMINATOR YYINITIAL Empty ... which means this is missing
93. FINDING A 6 YEAR OLD 🤦 BUG ERLANG VS FLEX Erlang Flex Language Class Decidable Regular Computational Model Turing machine Finite Automaton Supports In fi nite Nesting (for interpolation)? YES NO Why is the lexer not restoring its own stack?! Well, I kinda added the stack to make it work for interpolation. Since the Elixir lexer is written as Erlang functions, it can be a full Turing machine, but Flex can't. Flex is able to generate a very e ffi cient grammar by translating regular expressions, not extended regular expressions, to a fi nite automaton. A fi nite automaton lacks one important feature needed for Elixir: it cannot track in fi nite recursion of the language.
94. FINDING A 6 YEAR OLD 🤦 BUG INFINITE INTERPOLATION "#{"#{"#{"#{"#{"#{..}"}"}"}"}"}" Elixir requires this as you can nest any expression inside of interpolation, including a string or charlist, which supports interpolation. Although this is unlikely to happen in actual code for readability reasons, I didn't want to make the editor's grammar more strict than the real grammar as I believe Editor grammars need to keep working while you're typing and writing bad code.
95. FINDING A 6 YEAR OLD 🤦 BUG PUSHDOWN AUTOMATON Finite Automaton + Stack = Pushdown Automaton Flex, thankfully, has an escape hatch. Because arbitrary Java code can be run on each regular expression match, the Java code can keep track of a stack, which elevates the computational model from a Finite Automaton to a Pushdown Automaton. A Pushdown Automaton is the minimal complexity needed to track anything that requires balanced nesting like parentheses or interpolation braces.
96. FINDING A 6 YEAR OLD 🤦 BUG ERLANG VS FLEX Erlang Flex + Stack Language Class Decidable Decidable Computational Model Turing machine Pushdown Automaton Supports In fi nite Nesting (for interpolation)? YES YES Now, the lexers were equivalent for handling interpolation
97. V11.9.2 FIXING 🎉 A 6 YEAR OLD BUG •Don't match curly braces in YYINITIAL •Don't re-enter YYINITIAL with a stack •Clear stack on reset So the lexer needed a fi x. There were 3 ways the bug was fi xed and prevented from happening again
98. FIXING 🎉 A 6 YEAR OLD BUG DON'T MATCH CURLY BRACES IN YYINITIAL State Stack Token Text Token Next State Next Stack YYINTIAL Empty { OPENING_CURLY YYINITIAL Empty YYINITIAL Empty } CLOSING_CURLY YYINITIAL Empty INTERPOLATION 1.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL INTERPOLATION 1.YYINITIAL } INTERPOLATION_END YYINITIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL } CLOSING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION 2. YYINTIIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL } CLOSING_CURLY INTERPOLATION 1. YYINTIIAL The problem was ALWAYS counting nested curly braces even if it didn't a ff ect interpolation, so for YYINITIAL the lexer no longer adds to the stack
99. FIXING 🎉 A 6 YEAR OLD BUG DON'T MATCH CURLY BRACES IN YYINITIAL State Stack Token Text Token Next State Next Stack YYINTIAL Empty { OPENING_CURLY YYINITIAL Empty YYINITIAL Empty } CLOSING_CURLY YYINITIAL Empty INTERPOLATION 1.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL INTERPOLATION 1.YYINITIAL } INTERPOLATION_END YYINITIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL } CLOSING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION 2. YYINTIIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL } CLOSING_CURLY INTERPOLATION 1. YYINTIIAL For INTERPOLATION we enter a new state...
100. FIXING 🎉 A 6 YEAR OLD BUG DON'T MATCH CURLY BRACES IN YYINITIAL State Stack Token Text Token Next State Next Stack YYINTIAL Empty { OPENING_CURLY YYINITIAL Empty YYINITIAL Empty } CLOSING_CURLY YYINITIAL Empty INTERPOLATION 1.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL INTERPOLATION 1.YYINITIAL } INTERPOLATION_END YYINITIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL } CLOSING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION 2. YYINTIIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL } CLOSING_CURLY INTERPOLATION 1. YYINTIIAL INTERPOLATION_CURLY
101. FIXING 🎉 A 6 YEAR OLD BUG DON'T MATCH CURLY BRACES IN YYINITIAL State Stack Token Text Token Next State Next Stack YYINTIAL Empty { OPENING_CURLY YYINITIAL Empty YYINITIAL Empty } CLOSING_CURLY YYINITIAL Empty INTERPOLATION 1.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL INTERPOLATION 1.YYINITIAL } INTERPOLATION_END YYINITIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL } CLOSING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION 2. YYINTIIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL } CLOSING_CURLY INTERPOLATION 1. YYINTIIAL In INTERPOLATION _CURLY...
102. FIXING 🎉 A 6 YEAR OLD BUG DON'T MATCH CURLY BRACES IN YYINITIAL State Stack Token Text Token Next State Next Stack YYINTIAL Empty { OPENING_CURLY YYINITIAL Empty YYINITIAL Empty } CLOSING_CURLY YYINITIAL Empty INTERPOLATION 1.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL INTERPOLATION 1.YYINITIAL } INTERPOLATION_END YYINITIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL } CLOSING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION 2. YYINTIIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL } CLOSING_CURLY INTERPOLATION 1. YYINTIIAL ... it recursively adds to the stack for each opening curly brace, but...
103. FIXING 🎉 A 6 YEAR OLD BUG DON'T MATCH CURLY BRACES IN YYINITIAL State Stack Token Text Token Next State Next Stack YYINTIAL Empty { OPENING_CURLY YYINITIAL Empty YYINITIAL Empty } CLOSING_CURLY YYINITIAL Empty INTERPOLATION 1.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL INTERPOLATION 1.YYINITIAL } INTERPOLATION_END YYINITIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL } CLOSING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION 2. YYINTIIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL } CLOSING_CURLY INTERPOLATION 1. YYINTIIAL ... we pop each from the stack on the matching closing curly...
104. FIXING 🎉 A 6 YEAR OLD BUG DON'T MATCH CURLY BRACES IN YYINITIAL State Stack Token Text Token Next State Next Stack YYINTIAL Empty { OPENING_CURLY YYINITIAL Empty YYINITIAL Empty } CLOSING_CURLY YYINITIAL Empty INTERPOLATION 1.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL INTERPOLATION 1.YYINITIAL } INTERPOLATION_END YYINITIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL } CLOSING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION 2. YYINTIIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL } CLOSING_CURLY INTERPOLATION 1. YYINTIIAL ... until the fi nal one get us back into INTERPOLATION
105. FIXING 🎉 A 6 YEAR OLD BUG DON'T RE-ENTER YYINITIAL WITH A STACK The next part of the fi x was to not re-enter YYINITIAL with a stack, so if `pushAndBegin` is ever called with YYINITIAL throw an exception.
106. FIXING 🎉 A 6 YEAR OLD BUG DON'T RE-ENTER YYINITIAL WITH A STACK And also if the Stack itself is ever pushed another YYINITIAL when it isn't empty
107. FIXING 🎉 A 6 YEAR OLD BUG CLEAR STACK ON RESET The 3rd part of the fi x is to clear the stack on reset
108. FIXING 🎉 A 6 YEAR OLD BUG CLEAR STACK ON RESET ... I could easily lose this crucial line when making lexer updates in the future, so protect that with a test that reminds what to do if I mess up.
109. INTELLIJ ELIXIR V11.10.0 •OTP 23 and 24 opcodes •ASDF SDKs •Dialyzer Inspection After fi nding a 6 year old bug, version 11.10.0 was much more chill. It only involved expanding support for newer versions of OTP and SDKs and porting the credo inspection to work with dialyzer too.
110. V11.10.0 OTP 23 AND 24 OPCODES •BEAM Code chunk is not ETF •Opcodes don't include argument count •Every Code chunk includes the max opcode used IntelliJ Elixir doesn't just have a decompiler, but also a dissassembler, when OTP adds new opcodes I need to explicitly add support for them to the disassembler because the binary format does not include the expected argument count (that would take more space), but the disassembler can protect from incorrect disassembly because the Code chunk includes the max opcode used in its metadata.
111. OTP 23 AND 24 OPCODES ERTS/EMULATOR/BEAM/OPS.TAB Once I know beam opcodes are missing, it's easy to fi nd them because no opcode is ever removed and all the opcodes are in the ops.tab fi le. In BEAM itself this .tab fi le is transformed with Perl to produce the C code for the VM.
112. OTP 23 AND 24 OPCODES SRC/ORG/ELIXIR_LANG/BEAM/CHUNK/CODE/OPERATION/CODE.KT When adding the opcodes to the disassembler the important part is the number of arguments
113. OTP 23 AND 24 OPCODES SRC/ORG/ELIXIR_LANG/BEAM/CHUNK/CODE/OPERATION/CODE.KT ... but since in the Code chunk the arguments are just numbers, it can be useful to specialize the disassemlbly to make the dissassembler show more human-readable results like FAIL_LABELs
114. OTP 23 AND 24 OPCODES SRC/ORG/ELIXIR_LANG/BEAM/CHUNK/CODE/OPERATION/CODE.KT FAIL_LABEL is an Argument...
115. OTP 23 AND 24 OPCODES SRC/ORG/ELIXIR_LANG/BEAM/CHUNK/CODE/OPERATION/CODE.KT ... that should inline labels.
116. OTP 23 AND 24 OPCODES SRC/ORG/ELIXIR_LANG/BEAM/CHUNK/CODE/OPERATION/CODE.KT Options supports treating arguments as labels, but also atoms, functions, integers, lines, literals, local calls, and strings, which all reference other places in the beam fi le. When we combine all these options the disassembler can toggle the inlining on and o ff .
117. OTP 23 AND 24 OPCODES DISASSEMBLER INLINING TOGGLES Here's how that looks in the actual disassembler. I have a .beam fi le open in the editor and I went to the Chunks tab before going to the Code chunk. If I turn o ff all the inlining you get the bare binary format syntax without any semantic.
118. V11.10.0 ASDF SDK Adding a new package manager's SDK has two parts in IntelliJ Elixir: knowing where to look for it and the priority it should have compared to other installed package managers. This order mostly comes down to what is convenient on client projects, but feel free to open PRs placing your favorite package manager in the prime location.
119. V11.10.0 DETECTED SDK If you don't like the suggested directory, you can pick your own or use one of the other suggestions when you go to Preferences > Project Structure > Platform Settings > SDKs and click +
120. V11.10.0 DIALYZER INSPECTION •Implemented by Lutosław (@jacekgajek) Dialyzer inspection support was a community submission
121. V11.10.0 MIX DIALYZER BATCH INSPECTION In order to run dialyzer you need dialyxir or another project that supplies the dialyzer mix task in your project...
122. V11.10.0 MIX DIALYZER BATCH INSPECTION Then go to Code > Analyze Code > Run Inspection by Name...
123. V11.10.0 MIX DIALYZER BATCH INSPECTION ... and type Dialyzer to fi nd the inspection. Importantly, make sure to select the one for Elixir if you also have the IntelliJ Erlang plugin installed.
124. V11.10.0 MIX DIALYZER BATCH INSPECTION Since Elixir has more than one package for running Dialyzer, if your package doesn't call the mix tasks "dialyzer", you can customize the arguments in Perferences > Editor > Inspections > Dialyzer.
125. INTELLIJ ELIXIR V11.11.0 •JetBrains 2020.2 changed an API that broke dynamic languages plugins. Version 11.11.0 started users reporting weird changes in to Go To De fi nition in JetBrains 2020.2 IDEs. It broke multiple dynamic language plugins, including Bash, Clojure, and my own Elixir, and I took that personally...
126. V11.11.0 RESOLVING REFERENCES So, not only did I work-around the break, I also vastly improved the reference resolution because there's no engineering like over-engineering. So, now each Alias in a quali fi ed alias can resolve separately.
127. V11.11.0 RESOLVING REFERENCES - PERFORMANCE Indexed AllName ModularName defmodule ✅ ✅ defimpl ✅ ✅ defprotocol ✅ ✅ def ✅ ❌ defp ✅ ❌ defmacro ✅ ❌ defmacrop ✅ ❌ defdelegate ✅ ❌ @spec ✅ ❌ @callback ✅ ❌ @macrocallback ✅ ❌ Previously, completions for aliases used the larger, AllName index, which is slower...
128. V11.11.0 RESOLVING REFERENCES - PERFORMANCE Indexed AllName ModularName defmodule ✅ ✅ defimpl ✅ ✅ defprotocol ✅ ✅ def ✅ ❌ defp ✅ ❌ defmacro ✅ ❌ defmacrop ✅ ❌ defdelegate ✅ ❌ @spec ✅ ❌ @callback ✅ ❌ @macrocallback ✅ ❌ ... than using the smaller ModularName index it does now.
129. V11.11.0 RESOLVING REFERENCES - PONCHOS AND MONOREPOS Module Component allProject moduleWithDependenciesAndLibraryScope Web Source ✅ ✅ Web Libraries ✅ ✅ Web Dependencies ✅ ✅ Hardware Source ✅ ❌ Hardware Libraries ✅ ❌ Hardware Dependencies ✅ ❌ In monorepos and poncho projects, there can be multiple independent libraries and sources for each IntelliJ Module
130. V11.11.0 RESOLVING REFERENCES - PONCHOS AND MONOREPOS Module Component allProject moduleWithDependenciesAndLibraryScope Web Source ✅ ✅ Web Libraries ✅ ✅ Web Dependencies ✅ ✅ Hardware Source ✅ ❌ Hardware Libraries ✅ ❌ Hardware Dependencies ✅ ❌ If we want to fi nd aliases for completions in web
131. V11.11.0 RESOLVING REFERENCES - PONCHOS AND MONOREPOS Module Component allProject moduleWithDependenciesAndLibraryScope Web Source ✅ ✅ Web Libraries ✅ ✅ Web Dependencies ✅ ✅ Hardware Source ✅ ❌ Hardware Libraries ✅ ❌ Hardware Dependencies ✅ ❌ We don't want hardware modules being given as options
132. V11.11.0 RESOLVING REFERENCES - PONCHOS AND MONOREPOS Module Component allProject moduleWithDependenciesAndLibraryScope Web Source ✅ ✅ Web Libraries ✅ ✅ Web Dependencies ✅ ✅ Hardware Source ✅ ❌ Hardware Libraries ✅ ❌ Hardware Dependencies ✅ ❌ .. but before 11.11.0, allProject was used as the search scope, which would pull in everything
133. V11.11.0 RESOLVING REFERENCES - PONCHOS AND MONOREPOS Module Component allProject moduleWithDependenciesAndLibraryScope Web Source ✅ ✅ Web Libraries ✅ ✅ Web Dependencies ✅ ✅ Hardware Source ✅ ❌ Hardware Libraries ✅ ❌ Hardware Dependencies ✅ ❌ ... instead in 11.11.0, the completions are scoped to just Web and actual declared deps with moduleWithDependenciesAndLibraryScope
134. V11.11.0 RESOLVING REFERENCES - PREFERRED RESULTS 1.Valid 2.Same module 3.Source In the past, users have said they'd prefer not to see both source and decompiled results, so in v11.11.0, preference is given to better results unless no results would be shown
135. V11.11.0 RESOLVING REFERENCES - LEEX TEMPLATES •Assigns to assign/2,3, assign_new/3 calls •@myself •Assigns in live_component calls •Assigns in live_modal calls •@socket •@ fl ash •@inner_content Version 11.11.0 was released in June 2021 and Phoenix was still using LEEx instead of HEEx, so projects I had access to had LEEx I could use to vastly improve the resolving
136. RESOLVING REFERENCES - LEEX TEMPLATES ASSIGN/2,3 CALLS So @changeset is used in this form template, it is found in 3 places
137. RESOLVING REFERENCES - LEEX TEMPLATES ASSIGN/2 CALLS In the second case you can see the it is resolving to a key...
138. RESOLVING REFERENCES - LEEX TEMPLATES ASSIGN/2 CALLS But the variable name is fi nal_changeset, so it really is checking the key and not the variable name
139. RESOLVING REFERENCES - LEEX TEMPLATES ASSIGN/3 CALLS Then 3rd chageset is resolved as as argument to assign/3
140. RESOLVING REFERENCES - LEEX TEMPLATES @MYSELF @myself isn't de fi ned in a macro and instead buried in the libraries, so IntelliJ Elixir fi nd it there.
141. RESOLVING REFERENCES - LEEX TEMPLATES @SOCKET @socket in the template is resolved to the socket in handle_params
142. V11.11.0 RESOLVING REFERENCES - CODE_RELOADING? In the generated code for a Phoenix Endpoint, the `code_reloading?` is used to con fi g the code reloading socket. It's resolved with a the var! Declaration in the library code.
143. V11.11.0 RESOLVING REFERENCES - ECTO •Query •Query.API •Schema After LEEx, the next big library for reference improvements was Ecto. Ecto's macros are super weird, most of the words and variables aren't real functions or variables.
144. RESOLVING REFERENCES - ECTO QUERY For Ecto.Query, the p declared in a from in can be resolved in a select; the binding in functional join, where, group_by, ...
145. RESOLVING REFERENCES - ECTO QUERY ... order_by, having, select_merge, or distinct.
146. RESOLVING REFERENCES - ECTO QUERY.API Ecto.Query.API is a house of lies! The de fi nitions in the actual module are just there to allow documentation. If you import and try to use it directly.
147. RESOLVING REFERENCES - ECTO QUERY.API It will raise an error.... but it's lies do not matter, IntelliJ Elixir will also resolve binding references in those calls.
148. RESOLVING REFERENCES - ECTO SCHEMA For Ecto.Schema, fi eld can be resolved to either the import or macro de fi nition. You're probably thinking, what's so impressive about that? ` fi eld` is just a macro, well..
149. SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA How fi eld works in schema is use Ecto.Schema calls...
150. SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA ... Ecto.Schema.__using__
151. SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA ... which calls import Ecto.Schema
152. SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA ... BUT that's only importing the outer schema DSL! Where is ` fi eld` coming from?
153. SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA When you do schema do
154. SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA It calls the schema macro
155. SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA ... which calls the schema function
156. SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA ...
157. SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA It contains a large quote block, that is stored in the prelude variable, not directly returned
158. SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA At the end of the prelude is a try block.
159. SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA Ah ha! There's the import Ecto.Schema, ...
160. SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA ... but prelude is just a fl oating quote block. How does it get out?
161. SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA At the end of the function is a quote block that is returned and unquotes the prelude quote block
162. SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA 1.Resolve schema/2 to defmacro schema by walking the use, __using__, quote, and import. 2.Inside the schema/2 (or macros in general if you want to get fancy 💅 and support more DSLs), 3.Go into the body of the macro. If there's a call, resolve it 4.Go into the called function 5.Look for a quote block at the end (the same as the previous __using__ support) 6.If there's a Call inside an unquote see if you can resolve it to a variable in addition to a call de fi nition (which is already supported for Phoenix). 7.If it's a variable, check its value. If it's a quote, use the quote block handling 8.In the quote block handling add support for try 9.Walk the try and see the import, walk the import to fi nd Ecto.Schema.field/2 So to statically analyze an Ecto.Schema module, there are only these 9 simple steps
163. V11.11.0 RESOLVING REFERENCES - BUILT-IN TYPES Built-in types have no source de nition VS De fi nitions need to exist for... Go To De fi nition Find Usage Making type resolution was impeded by built-in types not having a shared de fi nition, so...
164. V11.11.0 RESOLVING REFERENCES - BUILT-IN TYPES I fi xed that by decompiling fake types in erlang.beam
165. V11.11.0 RESOLVING REFERENCES - BUILT-IN TYPES ... and so fi nd usages works on built-in types
166. V11.11.0 RESOLVING REFERENCES - TEACHING THE COMPUTER TO YELL AT ME I was able to fi nd and fi x or improve so many reference resolutions because I made an inspection that tells me any alias or call that does not resolve, except for ones where it looks like a potential key or fi eld access because I'm wasn't prepared to implement chained resolution like that.
167. V11.11.0 RESOLVING REFERENCES - DON'T CARE WHAT THE COMPILER THINK Elixir Metadata Location - location: :keep IntelliJ Elixir Resolving Call ✅ ✅ Uses ✅ Imports ✅ Related 🎉 Definitions ✅ ✅ Why I was able to resolve an assign to maps keys and Ecto references to bindings is unlike the compiler and ElixirLS built on top of it, the resolution of variables, assigns, calls, and aliases is customizable, for the editor, so it doesn't have to be how the code is really compiled and instead whatever leads you to better understanding of complex libraries, including related item like map keys buried deep in other libraries or other modules in your own code is what it should do and does do.
168. INTELLIJ ELIXIR V11.12.0 Version 11.12.0's changes were all powered by testing against the Yacto library that Tomoki Odaka was using. A full, open source, reproduction library unlocks so many inputs when combined with the Elixir References Inspection!
169. INTELLIJ ELIXIR V11.12.0 Yacto is a wrapper around Ecto that automatically generates migrations. To do this it wraps the schema and fi eld DSLs
170. INTELLIJ ELIXIR V11.12.0 Similar to Ecto, when use Yacto.Schema...
171. INTELLIJ ELIXIR V11.12.0 ... is called, ...
172. INTELLIJ ELIXIR V11.12.0 ... it only imports the outer schema call.
173. INTELLIJ ELIXIR V11.12.0 When schema is
174. INTELLIJ ELIXIR V11.12.0 ...called...
175. INTELLIJ ELIXIR V11.12.0 It captures the calling block...
176. INTELLIJ ELIXIR V11.12.0 ... and unquotes it...
177. INTELLIJ ELIXIR V11.12.0 .. after importing Yacto.Schema.
178. INTELLIJ ELIXIR V11.12.0 So, to support this calling convention if a macro has a do keyword argument...
179. INTELLIJ ELIXIR V11.12.0 ... grab the fi rst call in the body if it's a quote.
180. INTELLIJ ELIXIR V11.12.0 ... then if that quote block has an unquote that is the argument to `do:`
181. INTELLIJ ELIXIR V11.12.0 ... then search backwards...
182. INTELLIJ ELIXIR V11.12.0 ... as if those previous siblings were directly in the block.
183. INTELLIJ ELIXIR V12.0.0 •Ecto DSLs •ExUnit DSLs IntelliJ Elixir v12.0.0 answers the question: What if I spent 6 months making everything... I mean everything in Ecto resolve. I chose Ecto after how e ff ective using Yacto was for version 11.11.0
184. V12.0.0 ECTO Using the Elixir References inspection, the number of unresolvable or invalid-only references dropped from 496 errors to only 28 errors in lib
185. V12.0.0 ECTO This was all accomplished with only a few minor changes
186. INTELLIJ ELIXIR V12.1.0 •Elixir 1.12 syntax changes •Per sigil colors •Decompiler Enhancements Version 12.1.0 added Elixir 1.12 syntax changes, sigil colors, and decompiler enhancements
187. V12.1.0 ELIXIR 1.12 SYNTAX CHANGES •base**exponent •start..end//step Elixir 1.12 added new operators to the syntax, including the power operator and stepped ranges, which required updates to IntelliJ Elixir's lexer and parser.
188. V12.1.0 PER SIGIL COLORS With the introduction of more sigils that are common in the ecosystem like E, H, and L, allowing each sigil to be its own color became crucial. Go to Editor > Color Scheme > Elixir to custom the color scheme in general and the Textual > Sigil > Letter to customize a speci fi c sigil from there.
189. PER SIGIL COLORS LIGHT THEME Both the Light theme...
190. PER SIGIL COLORS DARCULA THEME ... and the far superior Darcula theme have the colors of each sigil equally distributed around the hue wheel in the HSB color coding, so they're spaced about 14 degrees apart.
191. V12.1.0 DECOMPILER ENHANCEMENTS •Add guards to decompiled function heads •Improper lists •External Function Captures Version 12.1.0 contains decompiler enhancements
192. DECOMPILER ENHANCEMENTS FUNCTION HEAD GUARDS 🚫 🚫 In previous versions of the decompiler guards weren't being decompiled, which led to function clauses that seemed to con fl ict with each other, but now the guards are decompiled, which makes the decompiled code must easier to understand and more accurate.
193. DECOMPILER ENHANCEMENTS IMPROPER LISTS Lists in Dbgi look like a call to a pipe function and the decompiler used to decompile it as such
194. DECOMPILER ENHANCEMENTS IMPROPER LISTS The fi x is to recognize this as not a real function and decompile it as an improper list.
195. DECOMPILER ENHANCEMENTS EXTERNAL FUNCTION CAPTURES The decompiler is based on a port of the Elixir Macro.to_string to Kotlin, in parts of it, it defers to inspect, which was also a port, which led to inspecting external function captures, which uses the comment tag to indicate it's not valid code, but breaks the decompiled code from parsing because the comment hides the rest of the function call
196. INTELLIJ ELIXIR V12.2.1 •Improve performance by delaying decompilation •Choose by name search •Protocol types •:erlang built-in types •Kernel and Kernel.SpecialForms implicit imports •Modules for quali fi ed calls Version 12.2.1 improved performance by delaying decompilation when resolving references and fi nd usages. Previously, .beam fi les were decompiled and then processed as if they were source, but that led to freezes when having to decompile large modules like Kernel, Kernel.SpecialForms, and the Unicode modules, which when fully decompiled is 160,000 lines long.
197. INTELLIJ ELIXIR V13.0.0 •JetBrains 2022 IDE support Version 13.0.0 added support for 2022.1 JetBrains IDEs
198. INTELLIJ ELIXIR V13.1.0 •mix format external formatter •mix new Project •mix credo with environment variables •Graphical Debugger Enhancements Version 13.1.0 adds support for more integrations with `mix`.
199. V13.1.0 MIX FORMAT EXTERNAL FORMATTER Formattable Internal External New Line ✅ ❌ Selection ✅ ❌ File ❌ ✅ Directory ❌ ✅ JetBrains added an experimental extension point for External Formatters. IntelliJ Elixir already had a con fi gurable formatter that predates `mix format`. That internal formatter is still used for indenting new lines and formatting selections in a fi le, but at the fi le level or above, mix format is used.
200. MIX FORMAT EXTERNAL FORMATTER FORMAT ON SAVE The formatter is not run automatically. If you want it to run automatically on save, you'll need to turn it on by going into Preferences, then in Tools > Actions on Save, you need to check "Reformat code".
201. MIX FORMAT EXTERNAL FORMATTER DISABLING If you're one of the holdouts that doesn't like mix format and wants to give me work to do when I fi x your project to follow community standards, you can disable mix format and continue to use the more con fi gurable internal formatter. Go to Preferences, then Editor > Code Style > Elixir. On the fi rst tab, `mix format`, uncheck "Format fi les with `mix format`".
202. V13.1.0 MIX NEW PROJECT In 2022 IDEs JetBrains changed how New Projects work for custom module types, which had the side-e ff ect of hiding Elixir from the New Project menu. I worked around this by using the newProjectWizard.language extension point to add Elixir to the language list the same as JetBrains supported languages like Java and Kotlin.
203. V13.1.0 MIX NEW PROJECT Since having to work around a JetBrains regression annoyed me, I made this version better by calling `mix new` and allowing all the command line options to be overridden instead of generating a project from a template.
204. V13.1.0 MIX NEW PROJECT It calls `mix new` in the background using the selected SDK.
205. V13.10.0 MIX CREDO BATCH INSPECTION The previous iteration of the credo integration was fragile and non-performant, so it's been rewritten as a batch inspection only. In order to run it you need credo installed in your project...
206. V13.10.0 MIX CREDO BATCH INSPECTION Then go to Code > Analyze Code > Run Inspection by Name...
207. V13.10.0 MIX CREDO BATCH INSPECTION ... and type Credo to fi nd the inspection. Click through to run the inspectiion...
208. V13.10.0 MIX CREDO BATCH INSPECTION The Problems Tool window will open where you can click through to the problems identi fi ed by credo.
209. INTELLIJ ELIXIR GRAPHICAL DEBUGGER ENHANCEMENTS •Environment variables for ESpec and ExUnit debugging •Debugging server supports Elixir 1.13.0 Environment variables for ExUnit and ESpec run con fi guration are used when debugging them; the debugging server now supports Elixir 1.13.0. The changes are due to di ff erence in the variable name for Erlang code generated from Elixir code and how variables are tracked in Elixir Env struct.
210. Contact Personal Email Kronic.Deth@gmail.com Twitter/ Slack @KronicDeth Sponsor https://github.com/sponsors/KronicDeth Professional Email Elle.Imho ff @DockYard.com Twitter @DockYard Hire https://dockyard.com/contact/hire-us IntelliJ Elixir IDE https://www.jetbrains.com/idea/download Plugin https://plugins.jetbrains.com/plugin/7522 GitHub https://github.com/KronicDeth/intellij-elixir IntelliJ Community Edition is Links If you have questions you can contact me by email or on Twitter or Elixir Lang slack @KronicDeth. If you'd like to support my work on the plugin you can support me directly through GitHub sponsors.
 If you're interested in working with me, you can contact DockYard with the Hire link or email me directly and I'll get you in touch with our sale department. Working with me through DockYard means I can get IntelliJ Elixir working on your exact project and work on it during business hours too!🌅 If you're interested in IntelliJ Elixir, you can download IntelliJ Community Edition for Free. Plugin install happens inside the IDE, but you can also view plugins in the JetBrains Plugin repository and the source for the plugin is available on GitHub.