IntelliJ Elixir Works Even if your Code Doesn't

Slide 1

Slide 1 text

IntelliJ Elixir Works Even if your Code Doesn't Elle Imho ff Principal Software Engineer at DockYard, Inc. Hi, I'm Elle Imho ff .   I'm known across the internet as KronicDeth because I'm full of diseases! I got new ones now and they're not all auto-immune anymore! I'm a member of the Lumen Core team, a Principal Software Engineer at DockYard, and most importantly for this talk, the creator of the IntelliJ Elixir plugin.

Slide 2

Slide 2 text

INTELLIJ ELIXIR WHAT IS INTELLIJ ELIXIR? IDE Project Language Add-on IntelliJ Elixir Any AppCode Objective-C Elixir AppCode Swift Elixir CLion C Elixir CLion C++ Elixir DataGrip SQL Elixir GoLand Go Elixir PHPStorm PHP Elixir PyCharm Python Elixir Rider .NET Elixir RubyMine Ruby Elixir WebStorm JS Elixir IntelliJ Elixir is a plugin for JetBrains IDEs: IntelliJ (Community and Ultimate Edition), of course, which supports any language, but it also works for those coming from an Objective-C background using AppCode, C/C++ using CLion, SQL using DataGrip, Go using GoLand, PHP using PHPStorm, Python using PyCharm, .NET using Rider, Ruby using Rubymine, or JS using WebStorm.

Slide 3

Slide 3 text

INTELLIJ ELIXIR WHEN IS INTELLIJ ELIXIR? •Started 2014-07-27 •87 releases •Updates every 33 days •Last update 2022-06-01 In the almost 8 years since I started the project, I've released 87 updates, which averages out to 33 days between updates with the last update only 9 days ago on June 1st.

Slide 4

Slide 4 text

INTELLIJ ELIXIR WHY IS INTELLIJ ELIXIR? •Used RubyMine for Ruby development •Wanted vim key bindings •Wanted Cmd+Click Go To De fi nition for Elixir •Wanted Search Everywhere for Elixir I originally considered starting plugin development after seeing Dave Thomas talk about Elixir at Lone Star Ruby and it was 2 days after the fi rst Elixir Conf that I did the fi rst commit. I thought: "Elixir's syntax is similar to Ruby's and RubyMine exists, so it can't be that hard to make an Elixir plugin".

Slide 5

Slide 5 text

INTELLIJ ELIXIR HOW IS INTELLIJ ELIXIR? •Custom lexer using JFlex •Custom parser using GrammarKit •Built on top of JetBrains' Open API Unlike the other editors that use the Elixir Language Server, the IntelliJ Elixir plugin actually predates ElixirLS by 3 years. IntelliJ Elixir has a custom lexer using JFlex, a custom parser using GrammarKit, and uses the JetBrains OpenAPI to deliver features you'd expect in JetBrain IDEs. This means that I'm writing Java and Kotlin to analyze Elixir. There turns out to be almost a 3rd of the project in Elixir due to the extensive testing. Early in the project my grammar and lexer were actually revealing bugs and inconsistencies in the lexer and parser in Elixir itself. The independent lexer and parser also means that code analysis works even if your code won't compile!

Slide 6

Slide 6 text

JETBRAINS OPEN API CODE IN MARKDOWN •Start blocks with language tag:  ```elixir Why is building on JetBrains OpenAPI useful? You get cool features for free like integration with the Markdown support when you use language tagged code blocks

Slide 7

Slide 7 text

CODE IN MARKDOWN SYNTAX HIGHLIGHTING So, when writing your guides syntax highlighting still works in the code blocks

Slide 8

Slide 8 text

CODE IN MARKDOWN PARAMETER AND VARIABLE MATCHING Parameter highlighting still works

Slide 9

Slide 9 text

CODE IN MARKDOWN COMPLETION Completion still works

Slide 10

Slide 10 text

CODE IN MARKDOWN GO TO DEFINITION Go To De fi nition even works.

Slide 11

Slide 11 text

GO TO DEFINITION WAIT... WHY DOES THAT WORK? As you can see the code block, `Repo` isn't aliased. You as a human reading the code know that Repo means your app's MyApp.Repo, but the code itself isn't valid as Repo isn't a top-level module, Ecto.Repo is, so why does the plugin fi nd a result? It's again because of JetBrains Open API. Part of the resolving system in the API is that a result can be marked as valid or invalid. Invalid matches are a crucial concept in editors as you want completion and Go To Declaration to work while you're typing.

Slide 12

Slide 12 text

GO TO DEFINITION VALID AND INVALID MATCHES Module Name Typing? Arity Interval Match Exact Unknown Yes Out of Interval Invalid Exact Unknown Don't Care In Interval Invalid Exact No Pre fi x Don't Care Don't Care No Exact Pre fi x Don't Care Don't Care Invalid Exact Exact Don't Care Out of Interval Invalid Exact Exact Don't Care In Interval Valid Any No Pre fi x Don't Care Don't Care No Any Pre fi x Yes Don't Care Invalid Any Pre fi x No Don't Care No Any Exact Don't Care Out of Interval No Any Exact Don't Care In Interval Invalid The matching logic needs to handle whether the module matches, how well the name matches, which includes unknown names for derived function names, whether the user is typing, and if the call is in the arity interval of the de fi nition. I say arity interval and not range because the special forms like `with` take an unlimited number of arguments, which for `with` is an interval from 1 to in fi nity. If there are no valid Exact module matches, then IntelliJ Elixir uses the stub index to fi nd any matches that match the name or arity regardless of module.

Slide 13

Slide 13 text

GO TO DEFINITION GO TO INVALID MATCHES IN ANY MODULE So we can navigate from Markdown even without the `alias` because the indexes built of the project let us fi nd invalid matches with the same name and arity even if in the wrong module.

Slide 14

Slide 14 text

GO TO DEFINITION GO TO CALLBACK If we go to the second entry you'll see that what IntelliJ Elixir counts as a de fi nition is more than just that which uses def or defmacro: it can resolves callbacks too!

Slide 15

Slide 15 text

INTELLIJ ELIXIR FEATURES •BEAM Files •Builder •Code Folding •Commenter •Completion •Credo Integration •Debugger •Dialyzer Integraton •EEx Templates •Find Usages •Formatting •Go To Declaration/Related/Symbol •Inspections •Live Template •Quick Fixes •Refactor •Run/Debug Con fi guration •Show Parameters •SDK •Structure •Syntax Highlighting IntelliJ Elixir has tons of features that I've implemented in the past 8 years, but that's too much to talk about in the time we have here today, so...

Slide 16

Slide 16 text

INTELLIJ ELIXIR ☣ CHANGES Releases 19 Commits 1015 Days 859 ... let's cover what's new since the last time I spoke in February 2020. It should be no problem, it's only 19 releases over 2 years and we have less than 40 minutes!

Slide 17

Slide 17 text

INTELLIJ ELIXIR V11.5.0 •More completions for invalid matches •Debugger expression evaluator in Elixir 1.9.2 As hinted at in the explanation of Go To De fi nition, IntelliJ Elixir fi nds as many de fi nitions as it can that could be related to the current call. Version 11.5.0 extended that...

Slide 18

Slide 18 text

V11.5.0 MORE COMPLETIONS FOR INVALID MATCHES •Quali fi ed calls (Process.flag(...)) •Type Specs (@spec foo(...)) •Aliases (alias Foo...) •Module Attributes (@attribute) •Variables (foo) •Protocol reserved module attributes (@protocol and @for) ... with quali fi ed calls, specs, aliases, module attributes, and variables with special support for the @protocol and @for module attributes de fi ned implicitly in Protocols. Expanding support in the plugin reminds me a lot of how exploit development works, you fi nd one bug in a system and then you keep exploring around the area to fi nd similar problem.

Slide 19

Slide 19 text

V11.5.0 DEBUGGER EXPRESSION EVALUATOR IN ELIXIR 1.9.2 •elixir_erl record changed José Valim has said that the debugger and IEx API are unsupported and not public, but that I can use it, so I do, but that does mean chasing after changes made to the internals. In Elixir 1.9.2, the elixir_erl record changed. Because records are just tuples and not maps, any change makes the code I wrote in the debugger break, so..

Slide 20

Slide 20 text

V11.5.0 DEBUGGER EXPRESSION EVALUATOR IN ELIXIR 1.9.2 ... I needed to change my version of it in the debug server. It's not guaranteed that the SDK shipped with the Erlang header fi les and not just the `.beam` fi les, so I can't depend on the same headers as Elixir itself.

Slide 21

Slide 21 text

INTELLIJ ELIXIR V11.6.0 •LEEx highlighting like EEx Version 11.6.0 was released in the long-long ago of April 2020, so before HEEx was part of Phoenix LiveView, so LEEx highlighting was added.

Slide 22

Slide 22 text

V11.6.0 LEEX HIGHLIGHTING LIKE EEX How LEEx and EEx works is once again based on JetBrains OpenAPI. In this case, support for template languages.

Slide 23

Slide 23 text

LEEX HIGHLIGHTING LIKE EEX A SMALL CHANGE FOR EEX The original EEx support was a simple and small change...

Slide 24

Slide 24 text

A SMALL CHANGE FOR EEX EEX GRAMMAR ... a lot of that change was regenerating the Elixir parser to support interleaving EEx template tags, but EEx as a language itself is really simple. It's either DATA or a tag. LEEx is the same since the actual syntax is not di ff erent, only the semantics and how it is compiled.

Slide 25

Slide 25 text

A SMALL CHANGE FOR EEX DATA IN TEMPLATEDATA That DATA tag is then hooked up as a TemplateDataElementType, TemplateData, which...

Slide 26

Slide 26 text

A SMALL CHANGE FOR EEX TEMPLATEDATA ...is used in...

Slide 27

Slide 27 text

A SMALL CHANGE FOR EEX TEMPLATEDATA IN ELEMENTTYPE ... getting the ViewProvider elementType

Slide 28

Slide 28 text

A SMALL CHANGE FOR EEX VIEWPROVIDER ELEMENTTYPE The ViewProvider elementType is then...

Slide 29

Slide 29 text

A SMALL CHANGE FOR EEX VIEWPROVIDER CREATEFILE ... called to create the fi le, but why does it know that the `.html.leex` fi le is HTML in addition to LEEx?!

Slide 30

Slide 30 text

A SMALL CHANGE FOR EEX VIEWPROVIDER Well, that happens when the ViewProvider is created. It calls templateDataLanguage

Slide 31

Slide 31 text

A SMALL CHANGE FOR EEX VIEWPROVIDER ...

Slide 32

Slide 32 text

A SMALL CHANGE FOR EEX VIEWPROVIDER There a lot of stu ff about caching here

Slide 33

Slide 33 text

A SMALL CHANGE FOR EEX VIEWPROVIDER ... and fallbacks if the template language isn't known, but ...

Slide 34

Slide 34 text

A SMALL CHANGE FOR EEX VIEWPROVIDER The actual logic of fi nding the template language from the fi le

Slide 35

Slide 35 text

A SMALL CHANGE FOR EEX VIEWPROVIDER Let's focus on onlyTemplateDateFileType

Slide 36

Slide 36 text

A SMALL CHANGE FOR EEX EEX FILE TYPE .. which is in the EEx File Type

Slide 37

Slide 37 text

A SMALL CHANGE FOR EEX EEX FILE TYPE This function is making sure there is only a single known fi le type, so let's see if how the set of fi le types are found...

Slide 38

Slide 38 text

A SMALL CHANGE FOR EEX EEX FILE TYPE In templateDataFileTypeSet

Slide 39

Slide 39 text

A SMALL CHANGE FOR EEX EEX FILE TYPE TemplateDataFileTypeSet fi nds the fi le types associated with the virtualFIle.

Slide 40

Slide 40 text

A SMALL CHANGE FOR EEX EEX FILE TYPE ... then gets those fi le types that have a fi xed extension...

Slide 41

Slide 41 text

A SMALL CHANGE FOR EEX EEX FILE TYPE ... that are then removed from the path...

Slide 42

Slide 42 text

A SMALL CHANGE FOR EEX EEX FILE TYPE ... and fi nally checking if that new fi le name has a known fi le type

Slide 43

Slide 43 text

A SMALL CHANGE FOR EEX EEX FILE TYPE 1.Start with full fi le name index.html.eex 2..eex is the EEx fi le type 3.Strip .eex: index.html 4..html is the HTML fi le type 5.HTML is the template data language How would this work for a real fi le? Say we have an `index.html.eex`, it's an EEx fi le type, which has `.eex` as its extension. The code strips that extension o ff , leaving `index.html`, which matches the HTML language, so...

Slide 44

Slide 44 text

A SMALL CHANGE FOR EEX HTML FROM JETBRAINS ❤ EEX FROM INTELLIJ ELIXIR ... we get HTML support developed by JetBrains mixed with the EEx I developed as you can see from the CSS selector breadcrumbs

Slide 45

Slide 45 text

A SMALL CHANGE FOR EEX SVG FROM JETBRAINS ❤ EEX FROM INTELLIJ ELIXIR Because I don't hard code the supported template languages you can template stu ff besides HTML, like SVG

Slide 46

Slide 46 text

A SMALL CHANGE FOR EEX ERL FROM INTELLIJ ERLANG ❤ EEX FROM INTELLIJ ELIXIR It also means it works with language support from other plugins, say templating Erlang source from the IntelliJ Erlang plugin using EEx.

Slide 47

Slide 47 text

INTELLIJ ELIXIR V11.8.0 •Quick Documentation •Gutter Icons for ExUnit Version 11.8.0 was a great release because all the features were implemented by community contributors.

Slide 48

Slide 48 text

V11.8.0 QUICK DOCUMENTATION •Implemented by Marcin Dawidziuk (@marcindawidziuk) Quick Documentation is the docs that pop up as hovers over the call being documented.

Slide 49

Slide 49 text

QUICK DOCUMENTATION ACTIVATING •F1 on keyboard You can trigger it with F1, but since not all of you are lucky enough to need a Kinesis Advantage 2, which has function keys, for RSI like I do, you can also activate it from the menu: go to View > Quick Documentation and you'll get the docs you're used to seeing in Hex docs.

Slide 50

Slide 50 text

V11.8.0 GUTTER ICONS FOR EXUNIT •Implemented by Niko Strijbol The gutter icons are a single play button for individual tests and fast forward buttons for groups like modules and describe. If you click on one of the icons it will run the module, describe, or test.

Slide 51

Slide 51 text

GUTTER ICONS FOR EXUNIT RESULTS The icons for groups or tests that pass will remain the same, but for tests that have errors, such as this test that incorrectly is checking for the "Sign out" message when it should be "Sign in", then the error is marked with an alert exclamation point behind the play button to re-run the test.

Slide 52

Slide 52 text

INTELLIJ ELIXIR V11.8.1 •JInterface 1.11 •Graphical Debugger on Windows Version 11.8.1 had graphical debugger fi xes.

Slide 53

Slide 53 text

V11.8.1 GRAPHICAL DEBUGGER COMMUNICATION PROTOCOL •No Pipe •Uses Distributed Erlang •IDE is a Java Node •Debugged is normal BEAM Node The graphical debugger in IntelliJ Elixir doesn't communicate with the BEAM using a pipe, but instead uses Distributed Erlang with the IDE acting as one node.

Slide 54

Slide 54 text

V11.8.1 JINTERFACE •Can't depend on SDK •Can't depend on build •Use Java Package from Maven To make a Java Node, the IDE needs to load the JInterface library. However, most package managers don't install JInterface and by default the erlang build doesn't build JInterface. It's one of those thing like SSL, XML, and doc support that it will silently skip building if it can't fi nd the dependency. So, to be a good Java application I had Jinterface pulling from the public Maven repository. Maven is like Hex, but for Java.

Slide 55

Slide 55 text

V11.8.1 JINTERFACE 1.6.1 💔 OTP 19.0 •BIG_CREATION •NEW_PID_EXT •NEW_PORT_EXT •NEWER_REFERENCE_EXT Jinterface doesn't get updated as often as Erlang itself because it only needs to change when the Distributed Erlang protocol changes. In OTP 19.0, the protocol changed to support something called BIG_CREATION. BIG_CREATION was a future proo fi ng to allow changes to how PIDs, Ports, and References were transmitted.

Slide 56

Slide 56 text

V11.8.1 JINTERFACE 1.6.1 💔 OTP 19.0 •BIG_CREATION •NEW_PID_EXT •NEW_PORT_EXT •NEWER_REFERENCE_EXT 🚫 The important point was that it was unused in OTP 19.0.

Slide 57

Slide 57 text

V11.8.1 MAVEN CENTRAL JINTERFACE •OTP Team didn't maintain it! •Basho maintained it! So even though this incompatibility existed since June of 2016, it didn't actually break my usage of JInterface 1.6.1 until May 2020 when OTP 24 was out. It turns out Jinterface wasn't being updated on Maven Centeral because the OTP team wasn't the publisher: Basho was and they were just putting the OTP team data in it the package information since Bash only did the build and publish

Slide 58

Slide 58 text

V11.8.1 BASHO •Creators of Riak •Fell over in 2017 ... oh, but Basho fell over in 2017, which explains why their publishing of Jinterface stopped. So, IntelliJ Elixir happened to start using the Maven Central repository just a few months before Riak fell over when Jake Becker converted the build from Ant to Gradle. I'd like to thank Jake because he's also the contributor that fi rst added the Graphical Debugger to IntelliJ Elixir before moving on to all his work on ElixirLS.

Slide 59

Slide 59 text

V11.8.1 JINTERFACE 1.11 ❤ OTP 24 🚫 OTP Team 🚫 Basho   🤦 Me So, OTP team doesn't want it and Basho can't do it, so I'm back to maintaining JInterface myself, I built it against OTP 24.0.3 and that's what's still being used to this day.

Slide 60

Slide 60 text

INTELLIJ ELIXIR V11.9.0 •Documentation Provider Performance Improvements In version 11.9.0, after only adding the documentaton provider in 11.8.0 Marcin came back and improved the performance

Slide 61

Slide 61 text

V11.9.0 DOCUMENTATION PROVIDER PERFORMANCE IMPROVEMENTS •Fast enough for mouse over •Read .beam or source directly (no more mix) •EEP-48 The performance was improved by not calling mix in another process, but instead reading the `.beam` fi les directly using the BEAM handling originally implemented for the decompiler.

Slide 62

Slide 62 text

DOCUMENTATION PROVIDER PERFORMANCE IMPROVEMENTS EEP-48 EEP-48 is the standard that introduced the Docs chunk to the BEAM format.

Slide 63

Slide 63 text

DOCUMENTATION PROVIDER PERFORMANCE IMPROVEMENTS EEP-48 The data is stored in a tuple. Remember this is an _Erlang_ Enhancement Proposal so lowercase words are atoms and titlecase words are variables.

Slide 64

Slide 64 text

EEP-48 "DOCS" FORMAT The Metadata portion gives us some extra tags to apply to docs, like deprecated and since.

Slide 65

Slide 65 text

DOCUMENTATION PROVIDER PERFORMANCE IMPROVEMENTS RENDERED METADATA That deprecated is rendered as a separate section

Slide 66

Slide 66 text

DOCUMENTATION PROVIDER PERFORMANCE IMPROVEMENTS USE DOCS FOR BETTER DECOMPILATION Since the faster documentation provider had to read the Docs chunks anyway, Marcin also added those docs to the decompiled code!

Slide 67

Slide 67 text

INTELLIJ ELIXIR V11.9.2 •Decompiler Enhancements •Fix for a 6 year old 🤦 bug In 11.9.2 there were even more decompiler enhancements

Slide 68

Slide 68 text

V11.9.2 DECOMPILER ENHANCEMENTS = HIGHLIGHTING BUGS 1.Decompile documentation with """ in it 2.Kill parsing of that decompiled module 3.Kill highlighting that references that module The addition of new features often lead to new bugs in old features due to new interactions, in the case of the Quick Documentation adding docs to the decompiled code, it meant that errors in the docs could break parsing those decompiled fi les, which meant it couldn't resolve references to those modules.

Slide 69

Slide 69 text

V11.9.2 MODULE PARSING FAILURE = HIGHLIGHTING BUGS This a ff ected Module, which is referenced by a lot of stu ff . So the example @doc in the @moduledoc for Module made all of Module's functions and macros unparseable

Slide 70

Slide 70 text

V11.9.2 MODULE PARSING FAILURE = HIGHLIGHTING BUGS So why wasn't this a problem in the original source for Module?

Slide 71

Slide 71 text

V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK Well it uses the capital S sigil and triple single quotes to avoid needing to escape the inner double quotes. The Docs chunk doesn't preserve that information since it's just a binary string that all BEAM languages can understand on purpose.

Slide 72

Slide 72 text

V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK So, what's the fi x? We need to calculate a safe promoter and terminator. The promoter is the symbol that start a quote while the terminator is the symbol that ends a quote. Sometimes they are the same, but sometimes it's matched curly braces. I got those terms from genetics because I couldn't think of the opposite of a terminator in computer science when writing the plugin.

Slide 73

Slide 73 text

V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK The safePromoterTerminator calculates the safe promoter...

Slide 74

Slide 74 text

V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK By checking for either triple single or double quotes

Slide 75

Slide 75 text

V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK If it has both nothing is safe!

Slide 76

Slide 76 text

V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK But if it has only one, then the other is safe

Slide 77

Slide 77 text

V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK If it has neither, it just defaults to triple double quotes

Slide 78

Slide 78 text

V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK With the safe promoter found or no safe promoter terminator at all, we use that to choose the outer promoter terminator.

Slide 79

Slide 79 text

V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK Then we use the the safePromoterTerminator to reformat the original text when appending

Slide 80

Slide 80 text

V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK ...

Slide 81

Slide 81 text

V11.9.2 QUOTE TYPE NOT PRESERVED IN DOCS CHUNK Only when we don't have a safe promoter terminator do we actually need to reformat the documentation text to escape the existing triple double quotes, which fi xes the docs and the parsing and the references.

Slide 82

Slide 82 text

V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG Tomoki Odaka, long time excellent bug reporter and sponsor of the project noticed that sometimes the highlighting would just give up while typing invalid code that was then fi xed. You want to know how good of a bug reporter Tomoki is? Tomoki links to similar bugs, gives reproduction steps and screenshots, but wait...

Slide 83

Slide 83 text

V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG .... That's not just a picture, the screenshot is a movie!

Slide 84

Slide 84 text

V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG So even though the map on line 3 gets highlighted correctly, line two never recovers.

Slide 85

Slide 85 text

V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG 1.Lexer parses code: Map.put(%{}, :) 2.Parser fi nds no matching element for :) 3.Parser tells lexer to advance to past : to ) Part of the reason I originally picked using JetBrain's lexing and parsing libraries instead of Elixir is that it has automatic error recovery: the parser will automatically shift the lexing around an error, but there's a hesitation and users can see the lexer and parser take time to recovery. You can see the editor still thinks the code is messed up with the red squiggle even though the code is valid and it triggers an error report

Slide 86

Slide 86 text

V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG 1.Lexer parses code: Map.put(%{}, :) 2.Parser fi nds no matching element for :) 3.Parser tells lexer to advance to past : to ) 4.If the lexer is in YYINITIAL state... It's ok to restore from that point! So, why that delay where it think there's still an error in the code? It turns out there are some unwritten rules about how the parser uses the lexer: if the lexer is in it's initial state then it is OK to only rewind the lexer to that point in the fi le.

Slide 87

Slide 87 text

V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG State Stack Token Text Token Text Remaining YYINITIAL Empty " STRING_PROMOTER #{{}}" GROUP 1.YYINITIAL #{ INTERPOLATION_START {}}" INTERPOLATION 1.GROUP 2.YYINITIAL { OPENING_CURLY }}" MULTILINE_WHITE_SPACE_MAYBE 1.YYINITIAL 2.INTERPOLATION 3.GROUP 4.YYINITIAL None None }}" YYINITIAL 1.INTERPOLATION 2.GROUP 3.YYINITIAL } CLOSING_CURLY }" ADDITION_OR_SUBTRACTION_MAYBE 1.INTERPOLATION 2.GROUP 3.YYINITIAL None None }" INTERPOLATION 1.GROUP 2.YYINITIAL } INTERPOLATION_END " GROUP 1.YYINITIAL " STRING_TERMINATOR YYINITIAL Empty Why does that cause a problem? Well, whenever a `{` is encountered, this might be in interpolation, but we don't want a closing curly brace to close an interpolation if it's actually closing a tuple

Slide 88

Slide 88 text

V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG State Stack Token Text Token Text Remaining YYINITIAL Empty " STRING_PROMOTER #{{}}" GROUP 1.YYINITIAL #{ INTERPOLATION_START {}}" INTERPOLATION 1.GROUP 2.YYINITIAL { OPENING_CURLY }}" MULTILINE_WHITE_SPACE_MAYBE 1.YYINITIAL 2.INTERPOLATION 3.GROUP 4.YYINITIAL None None }}" YYINITIAL 1.INTERPOLATION 2.GROUP 3.YYINITIAL } CLOSING_CURLY }" ADDITION_OR_SUBTRACTION_MAYBE 1.INTERPOLATION 2.GROUP 3.YYINITIAL None None }" INTERPOLATION 1.GROUP 2.YYINITIAL } INTERPOLATION_END " GROUP 1.YYINITIAL " STRING_TERMINATOR YYINITIAL Empty When in the initial state, YYINITIAL the closing curly brace needs to count as just a closing curly brace for a tuple or map, but...

Slide 89

Slide 89 text

V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG State Stack Token Text Token Text Remaining YYINITIAL Empty " STRING_PROMOTER #{{}}" GROUP 1.YYINITIAL #{ INTERPOLATION_START {}}" INTERPOLATION 1.GROUP 2.YYINITIAL { OPENING_CURLY }}" MULTILINE_WHITE_SPACE_MAYBE 1.YYINITIAL 2.INTERPOLATION 3.GROUP 4.YYINITIAL None None }}" YYINITIAL 1.INTERPOLATION 2.GROUP 3.YYINITIAL } CLOSING_CURLY }" ADDITION_OR_SUBTRACTION_MAYBE 1.INTERPOLATION 2.GROUP 3.YYINITIAL None None }" INTERPOLATION 1.GROUP 2.YYINITIAL } INTERPOLATION_END " GROUP 1.YYINITIAL " STRING_TERMINATOR YYINITIAL Empty ... when in the INTERPOLATION state it needs to count as the end of the interpolation

Slide 90

Slide 90 text

V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG State Stack Token Text Token Text Remaining YYINITIAL Empty " STRING_PROMOTER #{{}}" GROUP 1.YYINITIAL #{ INTERPOLATION_START {}}" INTERPOLATION 1.GROUP 2.YYINITIAL { OPENING_CURLY }}" MULTILINE_WHITE_SPACE_MAYBE 1.YYINITIAL 2.INTERPOLATION 3.GROUP 4.YYINITIAL None None }}" YYINITIAL 1.INTERPOLATION 2.GROUP 3.YYINITIAL } CLOSING_CURLY }" ADDITION_OR_SUBTRACTION_MAYBE 1.INTERPOLATION 2.GROUP 3.YYINITIAL None None }" INTERPOLATION 1.GROUP 2.YYINITIAL } INTERPOLATION_END " GROUP 1.YYINITIAL " STRING_TERMINATOR YYINITIAL Empty If the lexer is in YYINITIAL state, it's ok to restore from that point! So if we look, when we're in the middle of interpolation, we're in YYINITIAL again, so when we combine that with assumption that Open API makes that YYINITIAL is OK to resume from, we can resume from the middle of interpolation when there is an error.

Slide 91

Slide 91 text

V11.9.2 FINDING A 6 YEAR OLD 🤦 BUG What's the problem with resuming from there? The reset function doesn't restore the stack!

Slide 92

Slide 92 text

Slide 93

Slide 93 text

FINDING A 6 YEAR OLD 🤦 BUG ERLANG VS FLEX Erlang Flex Language Class Decidable Regular Computational Model Turing machine Finite Automaton Supports In fi nite Nesting (for interpolation)? YES NO Why is the lexer not restoring its own stack?! Well, I kinda added the stack to make it work for interpolation. Since the Elixir lexer is written as Erlang functions, it can be a full Turing machine, but Flex can't. Flex is able to generate a very e ffi cient grammar by translating regular expressions, not extended regular expressions, to a fi nite automaton. A fi nite automaton lacks one important feature needed for Elixir: it cannot track in fi nite recursion of the language.

Slide 94

Slide 94 text

FINDING A 6 YEAR OLD 🤦 BUG INFINITE INTERPOLATION "#{"#{"#{"#{"#{"#{..}"}"}"}"}"}" Elixir requires this as you can nest any expression inside of interpolation, including a string or charlist, which supports interpolation. Although this is unlikely to happen in actual code for readability reasons, I didn't want to make the editor's grammar more strict than the real grammar as I believe Editor grammars need to keep working while you're typing and writing bad code.

Slide 95

Slide 95 text

FINDING A 6 YEAR OLD 🤦 BUG PUSHDOWN AUTOMATON Finite Automaton + Stack = Pushdown Automaton Flex, thankfully, has an escape hatch. Because arbitrary Java code can be run on each regular expression match, the Java code can keep track of a stack, which elevates the computational model from a Finite Automaton to a Pushdown Automaton. A Pushdown Automaton is the minimal complexity needed to track anything that requires balanced nesting like parentheses or interpolation braces.

Slide 96

Slide 96 text

FINDING A 6 YEAR OLD 🤦 BUG ERLANG VS FLEX Erlang Flex + Stack Language Class Decidable Decidable Computational Model Turing machine Pushdown Automaton Supports In fi nite Nesting (for interpolation)? YES YES Now, the lexers were equivalent for handling interpolation

Slide 97

Slide 97 text

V11.9.2 FIXING 🎉 A 6 YEAR OLD BUG •Don't match curly braces in YYINITIAL •Don't re-enter YYINITIAL with a stack •Clear stack on reset So the lexer needed a fi x. There were 3 ways the bug was fi xed and prevented from happening again

Slide 98

Slide 98 text

FIXING 🎉 A 6 YEAR OLD BUG DON'T MATCH CURLY BRACES IN YYINITIAL State Stack Token Text Token Next State Next Stack YYINTIAL Empty { OPENING_CURLY YYINITIAL Empty YYINITIAL Empty } CLOSING_CURLY YYINITIAL Empty INTERPOLATION 1.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL INTERPOLATION 1.YYINITIAL } INTERPOLATION_END YYINITIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL } CLOSING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION 2. YYINTIIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL } CLOSING_CURLY INTERPOLATION 1. YYINTIIAL The problem was ALWAYS counting nested curly braces even if it didn't a ff ect interpolation, so for YYINITIAL the lexer no longer adds to the stack

Slide 99

Slide 99 text

Slide 100

Slide 100 text

Slide 101

Slide 101 text

Slide 102

Slide 102 text

FIXING 🎉 A 6 YEAR OLD BUG DON'T MATCH CURLY BRACES IN YYINITIAL State Stack Token Text Token Next State Next Stack YYINTIAL Empty { OPENING_CURLY YYINITIAL Empty YYINITIAL Empty } CLOSING_CURLY YYINITIAL Empty INTERPOLATION 1.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL INTERPOLATION 1.YYINITIAL } INTERPOLATION_END YYINITIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL } CLOSING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION 2. YYINTIIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL } CLOSING_CURLY INTERPOLATION 1. YYINTIIAL ... it recursively adds to the stack for each opening curly brace, but...

Slide 103

Slide 103 text

FIXING 🎉 A 6 YEAR OLD BUG DON'T MATCH CURLY BRACES IN YYINITIAL State Stack Token Text Token Next State Next Stack YYINTIAL Empty { OPENING_CURLY YYINITIAL Empty YYINITIAL Empty } CLOSING_CURLY YYINITIAL Empty INTERPOLATION 1.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL INTERPOLATION 1.YYINITIAL } INTERPOLATION_END YYINITIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL } CLOSING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION 2. YYINTIIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL } CLOSING_CURLY INTERPOLATION 1. YYINTIIAL ... we pop each from the stack on the matching closing curly...

Slide 104

Slide 104 text

FIXING 🎉 A 6 YEAR OLD BUG DON'T MATCH CURLY BRACES IN YYINITIAL State Stack Token Text Token Next State Next Stack YYINTIAL Empty { OPENING_CURLY YYINITIAL Empty YYINITIAL Empty } CLOSING_CURLY YYINITIAL Empty INTERPOLATION 1.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL INTERPOLATION 1.YYINITIAL } INTERPOLATION_END YYINITIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL { OPENING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL INTERPOLATION_CURLY 1. INTERPOLATION_CURLY 2. INTERPOLATION 3. YYINTIIAL } CLOSING_CURLY INTERPOLATION_CURLY 1. INTERPOLATION 2. YYINTIIAL INTERPOLATION_CURLY 1.INTERPOLATION 2.YYINITIAL } CLOSING_CURLY INTERPOLATION 1. YYINTIIAL ... until the fi nal one get us back into INTERPOLATION

Slide 105

Slide 105 text

FIXING 🎉 A 6 YEAR OLD BUG DON'T RE-ENTER YYINITIAL WITH A STACK The next part of the fi x was to not re-enter YYINITIAL with a stack, so if `pushAndBegin` is ever called with YYINITIAL throw an exception.

Slide 106

Slide 106 text

FIXING 🎉 A 6 YEAR OLD BUG DON'T RE-ENTER YYINITIAL WITH A STACK And also if the Stack itself is ever pushed another YYINITIAL when it isn't empty

Slide 107

Slide 107 text

FIXING 🎉 A 6 YEAR OLD BUG CLEAR STACK ON RESET The 3rd part of the fi x is to clear the stack on reset

Slide 108

Slide 108 text

FIXING 🎉 A 6 YEAR OLD BUG CLEAR STACK ON RESET ... I could easily lose this crucial line when making lexer updates in the future, so protect that with a test that reminds what to do if I mess up.

Slide 109

Slide 109 text

INTELLIJ ELIXIR V11.10.0 •OTP 23 and 24 opcodes •ASDF SDKs •Dialyzer Inspection After fi nding a 6 year old bug, version 11.10.0 was much more chill. It only involved expanding support for newer versions of OTP and SDKs and porting the credo inspection to work with dialyzer too.

Slide 110

Slide 110 text

V11.10.0 OTP 23 AND 24 OPCODES •BEAM Code chunk is not ETF •Opcodes don't include argument count •Every Code chunk includes the max opcode used IntelliJ Elixir doesn't just have a decompiler, but also a dissassembler, when OTP adds new opcodes I need to explicitly add support for them to the disassembler because the binary format does not include the expected argument count (that would take more space), but the disassembler can protect from incorrect disassembly because the Code chunk includes the max opcode used in its metadata.

Slide 111

Slide 111 text

OTP 23 AND 24 OPCODES ERTS/EMULATOR/BEAM/OPS.TAB Once I know beam opcodes are missing, it's easy to fi nd them because no opcode is ever removed and all the opcodes are in the ops.tab fi le. In BEAM itself this .tab fi le is transformed with Perl to produce the C code for the VM.

Slide 112

Slide 112 text

OTP 23 AND 24 OPCODES SRC/ORG/ELIXIR_LANG/BEAM/CHUNK/CODE/OPERATION/CODE.KT When adding the opcodes to the disassembler the important part is the number of arguments

Slide 113

Slide 113 text

OTP 23 AND 24 OPCODES SRC/ORG/ELIXIR_LANG/BEAM/CHUNK/CODE/OPERATION/CODE.KT ... but since in the Code chunk the arguments are just numbers, it can be useful to specialize the disassemlbly to make the dissassembler show more human-readable results like FAIL_LABELs

Slide 114

Slide 114 text

OTP 23 AND 24 OPCODES SRC/ORG/ELIXIR_LANG/BEAM/CHUNK/CODE/OPERATION/CODE.KT FAIL_LABEL is an Argument...

Slide 115

Slide 115 text

OTP 23 AND 24 OPCODES SRC/ORG/ELIXIR_LANG/BEAM/CHUNK/CODE/OPERATION/CODE.KT ... that should inline labels.

Slide 116

Slide 116 text

OTP 23 AND 24 OPCODES SRC/ORG/ELIXIR_LANG/BEAM/CHUNK/CODE/OPERATION/CODE.KT Options supports treating arguments as labels, but also atoms, functions, integers, lines, literals, local calls, and strings, which all reference other places in the beam fi le. When we combine all these options the disassembler can toggle the inlining on and o ff .

Slide 117

Slide 117 text

OTP 23 AND 24 OPCODES DISASSEMBLER INLINING TOGGLES Here's how that looks in the actual disassembler. I have a .beam fi le open in the editor and I went to the Chunks tab before going to the Code chunk. If I turn o ff all the inlining you get the bare binary format syntax without any semantic.

Slide 118

Slide 118 text

V11.10.0 ASDF SDK Adding a new package manager's SDK has two parts in IntelliJ Elixir: knowing where to look for it and the priority it should have compared to other installed package managers. This order mostly comes down to what is convenient on client projects, but feel free to open PRs placing your favorite package manager in the prime location.

Slide 119

Slide 119 text

V11.10.0 DETECTED SDK If you don't like the suggested directory, you can pick your own or use one of the other suggestions when you go to Preferences > Project Structure > Platform Settings > SDKs and click +

Slide 120

Slide 120 text

V11.10.0 DIALYZER INSPECTION •Implemented by Lutosław (@jacekgajek) Dialyzer inspection support was a community submission

Slide 121

Slide 121 text

V11.10.0 MIX DIALYZER BATCH INSPECTION In order to run dialyzer you need dialyxir or another project that supplies the dialyzer mix task in your project...

Slide 122

Slide 122 text

V11.10.0 MIX DIALYZER BATCH INSPECTION Then go to Code > Analyze Code > Run Inspection by Name...

Slide 123

Slide 123 text

V11.10.0 MIX DIALYZER BATCH INSPECTION ... and type Dialyzer to fi nd the inspection. Importantly, make sure to select the one for Elixir if you also have the IntelliJ Erlang plugin installed.

Slide 124

Slide 124 text

V11.10.0 MIX DIALYZER BATCH INSPECTION Since Elixir has more than one package for running Dialyzer, if your package doesn't call the mix tasks "dialyzer", you can customize the arguments in Perferences > Editor > Inspections > Dialyzer.

Slide 125

Slide 125 text

INTELLIJ ELIXIR V11.11.0 •JetBrains 2020.2 changed an API that broke dynamic languages plugins. Version 11.11.0 started users reporting weird changes in to Go To De fi nition in JetBrains 2020.2 IDEs. It broke multiple dynamic language plugins, including Bash, Clojure, and my own Elixir, and I took that personally...

Slide 126

Slide 126 text

V11.11.0 RESOLVING REFERENCES So, not only did I work-around the break, I also vastly improved the reference resolution because there's no engineering like over-engineering. So, now each Alias in a quali fi ed alias can resolve separately.

Slide 127

Slide 127 text

V11.11.0 RESOLVING REFERENCES - PERFORMANCE Indexed AllName ModularName defmodule ✅ ✅ defimpl ✅ ✅ defprotocol ✅ ✅ def ✅ ❌ defp ✅ ❌ defmacro ✅ ❌ defmacrop ✅ ❌ defdelegate ✅ ❌ @spec ✅ ❌ @callback ✅ ❌ @macrocallback ✅ ❌ Previously, completions for aliases used the larger, AllName index, which is slower...

Slide 128

Slide 128 text

V11.11.0 RESOLVING REFERENCES - PERFORMANCE Indexed AllName ModularName defmodule ✅ ✅ defimpl ✅ ✅ defprotocol ✅ ✅ def ✅ ❌ defp ✅ ❌ defmacro ✅ ❌ defmacrop ✅ ❌ defdelegate ✅ ❌ @spec ✅ ❌ @callback ✅ ❌ @macrocallback ✅ ❌ ... than using the smaller ModularName index it does now.

Slide 129

Slide 129 text

V11.11.0 RESOLVING REFERENCES - PONCHOS AND MONOREPOS Module Component allProject moduleWithDependenciesAndLibraryScope Web Source ✅ ✅ Web Libraries ✅ ✅ Web Dependencies ✅ ✅ Hardware Source ✅ ❌ Hardware Libraries ✅ ❌ Hardware Dependencies ✅ ❌ In monorepos and poncho projects, there can be multiple independent libraries and sources for each IntelliJ Module

Slide 130

Slide 130 text

Slide 131

Slide 131 text

V11.11.0 RESOLVING REFERENCES - PONCHOS AND MONOREPOS Module Component allProject moduleWithDependenciesAndLibraryScope Web Source ✅ ✅ Web Libraries ✅ ✅ Web Dependencies ✅ ✅ Hardware Source ✅ ❌ Hardware Libraries ✅ ❌ Hardware Dependencies ✅ ❌ We don't want hardware modules being given as options

Slide 132

Slide 132 text

V11.11.0 RESOLVING REFERENCES - PONCHOS AND MONOREPOS Module Component allProject moduleWithDependenciesAndLibraryScope Web Source ✅ ✅ Web Libraries ✅ ✅ Web Dependencies ✅ ✅ Hardware Source ✅ ❌ Hardware Libraries ✅ ❌ Hardware Dependencies ✅ ❌ .. but before 11.11.0, allProject was used as the search scope, which would pull in everything

Slide 133

Slide 133 text

V11.11.0 RESOLVING REFERENCES - PONCHOS AND MONOREPOS Module Component allProject moduleWithDependenciesAndLibraryScope Web Source ✅ ✅ Web Libraries ✅ ✅ Web Dependencies ✅ ✅ Hardware Source ✅ ❌ Hardware Libraries ✅ ❌ Hardware Dependencies ✅ ❌ ... instead in 11.11.0, the completions are scoped to just Web and actual declared deps with moduleWithDependenciesAndLibraryScope

Slide 134

Slide 134 text

V11.11.0 RESOLVING REFERENCES - PREFERRED RESULTS 1.Valid 2.Same module 3.Source In the past, users have said they'd prefer not to see both source and decompiled results, so in v11.11.0, preference is given to better results unless no results would be shown

Slide 135

Slide 135 text

V11.11.0 RESOLVING REFERENCES - LEEX TEMPLATES •Assigns to assign/2,3, assign_new/3 calls •@myself •Assigns in live_component calls •Assigns in live_modal calls •@socket •@ fl ash •@inner_content Version 11.11.0 was released in June 2021 and Phoenix was still using LEEx instead of HEEx, so projects I had access to had LEEx I could use to vastly improve the resolving

Slide 136

Slide 136 text

RESOLVING REFERENCES - LEEX TEMPLATES ASSIGN/2,3 CALLS So @changeset is used in this form template, it is found in 3 places

Slide 137

Slide 137 text

RESOLVING REFERENCES - LEEX TEMPLATES ASSIGN/2 CALLS In the second case you can see the it is resolving to a key...

Slide 138

Slide 138 text

RESOLVING REFERENCES - LEEX TEMPLATES ASSIGN/2 CALLS But the variable name is fi nal_changeset, so it really is checking the key and not the variable name

Slide 139

Slide 139 text

RESOLVING REFERENCES - LEEX TEMPLATES ASSIGN/3 CALLS Then 3rd chageset is resolved as as argument to assign/3

Slide 140

Slide 140 text

RESOLVING REFERENCES - LEEX TEMPLATES @MYSELF @myself isn't de fi ned in a macro and instead buried in the libraries, so IntelliJ Elixir fi nd it there.

Slide 141

Slide 141 text

RESOLVING REFERENCES - LEEX TEMPLATES @SOCKET @socket in the template is resolved to the socket in handle_params

Slide 142

Slide 142 text

V11.11.0 RESOLVING REFERENCES - CODE_RELOADING? In the generated code for a Phoenix Endpoint, the `code_reloading?` is used to con fi g the code reloading socket. It's resolved with a the var! Declaration in the library code.

Slide 143

Slide 143 text

V11.11.0 RESOLVING REFERENCES - ECTO •Query •Query.API •Schema After LEEx, the next big library for reference improvements was Ecto. Ecto's macros are super weird, most of the words and variables aren't real functions or variables.

Slide 144

Slide 144 text

RESOLVING REFERENCES - ECTO QUERY For Ecto.Query, the p declared in a from in can be resolved in a select; the binding in functional join, where, group_by, ...

Slide 145

Slide 145 text

RESOLVING REFERENCES - ECTO QUERY ... order_by, having, select_merge, or distinct.

Slide 146

Slide 146 text

RESOLVING REFERENCES - ECTO QUERY.API Ecto.Query.API is a house of lies! The de fi nitions in the actual module are just there to allow documentation. If you import and try to use it directly.

Slide 147

Slide 147 text

RESOLVING REFERENCES - ECTO QUERY.API It will raise an error.... but it's lies do not matter, IntelliJ Elixir will also resolve binding references in those calls.

Slide 148

Slide 148 text

RESOLVING REFERENCES - ECTO SCHEMA For Ecto.Schema, fi eld can be resolved to either the import or macro de fi nition. You're probably thinking, what's so impressive about that? ` fi eld` is just a macro, well..

Slide 149

Slide 149 text

SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA How fi eld works in schema is use Ecto.Schema calls...

Slide 150

Slide 150 text

SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA ... Ecto.Schema.__using__

Slide 151

Slide 151 text

SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA ... which calls import Ecto.Schema

Slide 152

Slide 152 text

SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA ... BUT that's only importing the outer schema DSL! Where is ` fi eld` coming from?

Slide 153

Slide 153 text

SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA When you do schema do

Slide 154

Slide 154 text

SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA It calls the schema macro

Slide 155

Slide 155 text

SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA ... which calls the schema function

Slide 156

Slide 156 text

SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA ...

Slide 157

Slide 157 text

SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA It contains a large quote block, that is stored in the prelude variable, not directly returned

Slide 158

Slide 158 text

SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA At the end of the prelude is a try block.

Slide 159

Slide 159 text

SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA Ah ha! There's the import Ecto.Schema, ...

Slide 160

Slide 160 text

SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA ... but prelude is just a fl oating quote block. How does it get out?

Slide 161

Slide 161 text

SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA At the end of the function is a quote block that is returned and unquotes the prelude quote block

Slide 162

Slide 162 text

SCHEMA HOW FIELD WORKS IN SCHEMA FOR ECTO.SCHEMA 1.Resolve schema/2 to defmacro schema by walking the use, __using__, quote, and import. 2.Inside the schema/2 (or macros in general if you want to get fancy 💅 and support more DSLs), 3.Go into the body of the macro. If there's a call, resolve it 4.Go into the called function 5.Look for a quote block at the end (the same as the previous __using__ support) 6.If there's a Call inside an unquote see if you can resolve it to a variable in addition to a call de fi nition (which is already supported for Phoenix). 7.If it's a variable, check its value. If it's a quote, use the quote block handling 8.In the quote block handling add support for try 9.Walk the try and see the import, walk the import to fi nd Ecto.Schema.field/2 So to statically analyze an Ecto.Schema module, there are only these 9 simple steps

Slide 163

Slide 163 text

V11.11.0 RESOLVING REFERENCES - BUILT-IN TYPES Built-in types have no source de nition VS De fi nitions need to exist for... Go To De fi nition Find Usage Making type resolution was impeded by built-in types not having a shared de fi nition, so...

Slide 164

Slide 164 text

V11.11.0 RESOLVING REFERENCES - BUILT-IN TYPES I fi xed that by decompiling fake types in erlang.beam

Slide 165

Slide 165 text

V11.11.0 RESOLVING REFERENCES - BUILT-IN TYPES ... and so fi nd usages works on built-in types

Slide 166

Slide 166 text

V11.11.0 RESOLVING REFERENCES - TEACHING THE COMPUTER TO YELL AT ME I was able to fi nd and fi x or improve so many reference resolutions because I made an inspection that tells me any alias or call that does not resolve, except for ones where it looks like a potential key or fi eld access because I'm wasn't prepared to implement chained resolution like that.

Slide 167

Slide 167 text

V11.11.0 RESOLVING REFERENCES - DON'T CARE WHAT THE COMPILER THINK Elixir Metadata Location - location: :keep IntelliJ Elixir Resolving Call ✅ ✅ Uses ✅ Imports ✅ Related 🎉 Definitions ✅ ✅ Why I was able to resolve an assign to maps keys and Ecto references to bindings is unlike the compiler and ElixirLS built on top of it, the resolution of variables, assigns, calls, and aliases is customizable, for the editor, so it doesn't have to be how the code is really compiled and instead whatever leads you to better understanding of complex libraries, including related item like map keys buried deep in other libraries or other modules in your own code is what it should do and does do.

Slide 168

Slide 168 text

INTELLIJ ELIXIR V11.12.0 Version 11.12.0's changes were all powered by testing against the Yacto library that Tomoki Odaka was using. A full, open source, reproduction library unlocks so many inputs when combined with the Elixir References Inspection!

Slide 169

Slide 169 text

INTELLIJ ELIXIR V11.12.0 Yacto is a wrapper around Ecto that automatically generates migrations. To do this it wraps the schema and fi eld DSLs

Slide 170

Slide 170 text

INTELLIJ ELIXIR V11.12.0 Similar to Ecto, when use Yacto.Schema...

Slide 171

Slide 171 text

INTELLIJ ELIXIR V11.12.0 ... is called, ...

Slide 172

Slide 172 text

INTELLIJ ELIXIR V11.12.0 ... it only imports the outer schema call.

Slide 173

Slide 173 text

INTELLIJ ELIXIR V11.12.0 When schema is

Slide 174

Slide 174 text

INTELLIJ ELIXIR V11.12.0 ...called...

Slide 175

Slide 175 text

INTELLIJ ELIXIR V11.12.0 It captures the calling block...

Slide 176

Slide 176 text

INTELLIJ ELIXIR V11.12.0 ... and unquotes it...

Slide 177

Slide 177 text

INTELLIJ ELIXIR V11.12.0 .. after importing Yacto.Schema.

Slide 178

Slide 178 text

INTELLIJ ELIXIR V11.12.0 So, to support this calling convention if a macro has a do keyword argument...

Slide 179

Slide 179 text

INTELLIJ ELIXIR V11.12.0 ... grab the fi rst call in the body if it's a quote.

Slide 180

Slide 180 text

INTELLIJ ELIXIR V11.12.0 ... then if that quote block has an unquote that is the argument to `do:`

Slide 181

Slide 181 text

INTELLIJ ELIXIR V11.12.0 ... then search backwards...

Slide 182

Slide 182 text

INTELLIJ ELIXIR V11.12.0 ... as if those previous siblings were directly in the block.

Slide 183

Slide 183 text

INTELLIJ ELIXIR V12.0.0 •Ecto DSLs •ExUnit DSLs IntelliJ Elixir v12.0.0 answers the question: What if I spent 6 months making everything... I mean everything in Ecto resolve. I chose Ecto after how e ff ective using Yacto was for version 11.11.0

Slide 184

Slide 184 text

V12.0.0 ECTO Using the Elixir References inspection, the number of unresolvable or invalid-only references dropped from 496 errors to only 28 errors in lib

Slide 185

Slide 185 text

V12.0.0 ECTO This was all accomplished with only a few minor changes

Slide 186

Slide 186 text

INTELLIJ ELIXIR V12.1.0 •Elixir 1.12 syntax changes •Per sigil colors •Decompiler Enhancements Version 12.1.0 added Elixir 1.12 syntax changes, sigil colors, and decompiler enhancements

Slide 187

Slide 187 text

V12.1.0 ELIXIR 1.12 SYNTAX CHANGES •base**exponent •start..end//step Elixir 1.12 added new operators to the syntax, including the power operator and stepped ranges, which required updates to IntelliJ Elixir's lexer and parser.

Slide 188

Slide 188 text

V12.1.0 PER SIGIL COLORS With the introduction of more sigils that are common in the ecosystem like E, H, and L, allowing each sigil to be its own color became crucial. Go to Editor > Color Scheme > Elixir to custom the color scheme in general and the Textual > Sigil > Letter to customize a speci fi c sigil from there.

Slide 189

Slide 189 text

PER SIGIL COLORS LIGHT THEME Both the Light theme...

Slide 190

Slide 190 text

PER SIGIL COLORS DARCULA THEME ... and the far superior Darcula theme have the colors of each sigil equally distributed around the hue wheel in the HSB color coding, so they're spaced about 14 degrees apart.

Slide 191

Slide 191 text

V12.1.0 DECOMPILER ENHANCEMENTS •Add guards to decompiled function heads •Improper lists •External Function Captures Version 12.1.0 contains decompiler enhancements

Slide 192

Slide 192 text

DECOMPILER ENHANCEMENTS FUNCTION HEAD GUARDS 🚫 🚫 In previous versions of the decompiler guards weren't being decompiled, which led to function clauses that seemed to con fl ict with each other, but now the guards are decompiled, which makes the decompiled code must easier to understand and more accurate.

Slide 193

Slide 193 text

DECOMPILER ENHANCEMENTS IMPROPER LISTS Lists in Dbgi look like a call to a pipe function and the decompiler used to decompile it as such

Slide 194

Slide 194 text

DECOMPILER ENHANCEMENTS IMPROPER LISTS The fi x is to recognize this as not a real function and decompile it as an improper list.

Slide 195

Slide 195 text

DECOMPILER ENHANCEMENTS EXTERNAL FUNCTION CAPTURES The decompiler is based on a port of the Elixir Macro.to_string to Kotlin, in parts of it, it defers to inspect, which was also a port, which led to inspecting external function captures, which uses the comment tag to indicate it's not valid code, but breaks the decompiled code from parsing because the comment hides the rest of the function call

Slide 196

Slide 196 text

INTELLIJ ELIXIR V12.2.1 •Improve performance by delaying decompilation •Choose by name search •Protocol types •:erlang built-in types •Kernel and Kernel.SpecialForms implicit imports •Modules for quali fi ed calls Version 12.2.1 improved performance by delaying decompilation when resolving references and fi nd usages. Previously, .beam fi les were decompiled and then processed as if they were source, but that led to freezes when having to decompile large modules like Kernel, Kernel.SpecialForms, and the Unicode modules, which when fully decompiled is 160,000 lines long.

Slide 197

Slide 197 text

INTELLIJ ELIXIR V13.0.0 •JetBrains 2022 IDE support Version 13.0.0 added support for 2022.1 JetBrains IDEs

Slide 198

Slide 198 text

INTELLIJ ELIXIR V13.1.0 •mix format external formatter •mix new Project •mix credo with environment variables •Graphical Debugger Enhancements Version 13.1.0 adds support for more integrations with `mix`.

Slide 199

Slide 199 text

V13.1.0 MIX FORMAT EXTERNAL FORMATTER Formattable Internal External New Line ✅ ❌ Selection ✅ ❌ File ❌ ✅ Directory ❌ ✅ JetBrains added an experimental extension point for External Formatters. IntelliJ Elixir already had a con fi gurable formatter that predates `mix format`. That internal formatter is still used for indenting new lines and formatting selections in a fi le, but at the fi le level or above, mix format is used.

Slide 200

Slide 200 text

MIX FORMAT EXTERNAL FORMATTER FORMAT ON SAVE The formatter is not run automatically. If you want it to run automatically on save, you'll need to turn it on by going into Preferences, then in Tools > Actions on Save, you need to check "Reformat code".

Slide 201

Slide 201 text

MIX FORMAT EXTERNAL FORMATTER DISABLING If you're one of the holdouts that doesn't like mix format and wants to give me work to do when I fi x your project to follow community standards, you can disable mix format and continue to use the more con fi gurable internal formatter. Go to Preferences, then Editor > Code Style > Elixir. On the fi rst tab, `mix format`, uncheck "Format fi les with `mix format`".

Slide 202

Slide 202 text

V13.1.0 MIX NEW PROJECT In 2022 IDEs JetBrains changed how New Projects work for custom module types, which had the side-e ff ect of hiding Elixir from the New Project menu. I worked around this by using the newProjectWizard.language extension point to add Elixir to the language list the same as JetBrains supported languages like Java and Kotlin.

Slide 203

Slide 203 text

V13.1.0 MIX NEW PROJECT Since having to work around a JetBrains regression annoyed me, I made this version better by calling `mix new` and allowing all the command line options to be overridden instead of generating a project from a template.

Slide 204

Slide 204 text

V13.1.0 MIX NEW PROJECT It calls `mix new` in the background using the selected SDK.

Slide 205

Slide 205 text

V13.10.0 MIX CREDO BATCH INSPECTION The previous iteration of the credo integration was fragile and non-performant, so it's been rewritten as a batch inspection only. In order to run it you need credo installed in your project...

Slide 206

Slide 206 text

V13.10.0 MIX CREDO BATCH INSPECTION Then go to Code > Analyze Code > Run Inspection by Name...

Slide 207

Slide 207 text

V13.10.0 MIX CREDO BATCH INSPECTION ... and type Credo to fi nd the inspection. Click through to run the inspectiion...

Slide 208

Slide 208 text

V13.10.0 MIX CREDO BATCH INSPECTION The Problems Tool window will open where you can click through to the problems identi fi ed by credo.

Slide 209

Slide 209 text

INTELLIJ ELIXIR GRAPHICAL DEBUGGER ENHANCEMENTS •Environment variables for ESpec and ExUnit debugging •Debugging server supports Elixir 1.13.0 Environment variables for ExUnit and ESpec run con fi guration are used when debugging them; the debugging server now supports Elixir 1.13.0. The changes are due to di ff erence in the variable name for Erlang code generated from Elixir code and how variables are tracked in Elixir Env struct.

Slide 210

Slide 210 text

Contact Personal Email [email protected] Twitter/ Slack @KronicDeth Sponsor https://github.com/sponsors/KronicDeth Professional Email Elle.Imho ff @DockYard.com Twitter @DockYard Hire https://dockyard.com/contact/hire-us IntelliJ Elixir IDE https://www.jetbrains.com/idea/download Plugin https://plugins.jetbrains.com/plugin/7522 GitHub https://github.com/KronicDeth/intellij-elixir IntelliJ Community Edition is Links If you have questions you can contact me by email or on Twitter or Elixir Lang slack @KronicDeth. If you'd like to support my work on the plugin you can support me directly through GitHub sponsors.  If you're interested in working with me, you can contact DockYard with the Hire link or email me directly and I'll get you in touch with our sale department. Working with me through DockYard means I can get IntelliJ Elixir working on your exact project and work on it during business hours too!🌅 If you're interested in IntelliJ Elixir, you can download IntelliJ Community Edition for Free. Plugin install happens inside the IDE, but you can also view plugins in the JetBrains Plugin repository and the source for the plugin is available on GitHub.