Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Anatomy of a translating compiler
Search
Serge Smertin
June 30, 2022
Programming
0
420
Anatomy of a translating compiler
How to build a programming language in Scala?
Serge Smertin
June 30, 2022
Tweet
Share
More Decks by Serge Smertin
See All by Serge Smertin
Building Databricks integrations with Go
nfx
0
320
Building robust Python applications on top of Databricks: UCX case study
nfx
0
310
Introducing the Power of Databricks SDK
nfx
0
430
Reaching 3M downloads and 90%+ unit test coverage for a Terraform provider: lessons learned
nfx
0
500
Distributed Data Mesh, Delta Lake, and Terraform
nfx
0
500
Data Quality with or without Apache Spark and its ecosystem
nfx
0
590
Data Privacy with Apache Spark: Defensive and Offensive Approaches
nfx
0
590
Other Decks in Programming
See All in Programming
1から理解するWeb Push
dora1998
7
1.9k
開発チーム・開発組織の設計改善スキルの向上
masuda220
PRO
20
11k
print("Hello, World")
eddie
2
530
Compose Multiplatform × AI で作る、次世代アプリ開発支援ツールの設計と実装
thagikura
0
140
「待たせ上手」なスケルトンスクリーン、 そのUXの裏側
teamlab
PRO
0
500
AI時代のUIはどこへ行く?
yusukebe
18
8.8k
testingを眺める
matumoto
1
140
モバイルアプリからWebへの横展開を加速した話_Claude_Code_実践術.pdf
kazuyasakamoto
0
320
テストコードはもう書かない:JetBrains AI Assistantに委ねる非同期処理のテスト自動設計・生成
makun
0
250
もうちょっといいRubyプロファイラを作りたい (2025)
osyoyu
1
430
250830 IaCの選定~AWS SAMのLambdaをECSに乗り換えたときの備忘録~
east_takumi
0
390
Android端末で実現するオンデバイスLLM 2025
masayukisuda
1
140
Featured
See All Featured
[RailsConf 2023] Rails as a piece of cake
palkan
57
5.8k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
49
3k
Java REST API Framework Comparison - PWX 2021
mraible
33
8.8k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
36
2.5k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
18
1.1k
Code Review Best Practice
trishagee
70
19k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
The Power of CSS Pseudo Elements
geoffreycrofte
77
6k
Site-Speed That Sticks
csswizardry
10
810
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
131
19k
Writing Fast Ruby
sferik
628
62k
Transcript
Anatomy of a translating compiler … or how to use
debugger all the time. 1 Serge Smertin Senior Resident Solutions Architect Databricks
None
Apache Spark query execution recap
4 code IN(4*, 5*)
5 code IN(4*, 5*)
6 code IN(4*, 5*)
7 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") )))
8 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\'))
9 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\')) private def expression(ctx: LogicalContext , expr: ast.Expr): Expression = expr match { case ast.FieldIn(field, exprs) if exprs.exists(_.isInstanceOf[ast.Wildcard]) => val expand = (expr: ast.Expr) => ast.Binary(ast.Field(field), ast.Equals, expr) expression(ctx, exprs.tail.foldLeft(expand(exprs.head)) { (left, right) => ast.Binary(left, ast.Or, expand(right)) }) case ast.FieldIn(field, exprs) => In(UnresolvedAttribute(field), exprs.map(expression(ctx, _))) case ast.Binary(left, ast.Equals, ast.Wildcard(pattern)) => like(ctx, left, pattern) case ast.Binary(left, symbol, right) => symbol match { case ast.Or => Or(attrOrExpr(ctx, left), attrOrExpr(ctx, right)) // ... } // ... }
10 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\')) display(spark.table('main') .where((F.col('code').like('4%') | F.col('code').like('5%'))))
11 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\')) display(spark.table('main') .where((F.col('code').like('4%') | F.col('code').like('5%')))) case relation: UnresolvedRelation => s"spark.table(${q(relation.name)})" private def unfoldWheres(expr: Expression): String = expr match { case And(left, right) => s"${unfoldWheres(left)}\n${unfoldWheres(right)}" case _ => s".where(${expressionCode(expr)})" } private def expressionCode(expr: Expression): String = expr match { case b: BinaryOperator => val symbol = jvmToPythonOverrides.getOrElse(b.symbol, b.symbol) s"(${expressionCode(b.left)} $symbol ${expressionCode(b.right)})" case attr: UnresolvedAttribute => s"F.col(${q(attr.name)})" case Like(col, Literal(value, _ @ StringType), _) => s"${expressionCode(col)}.like('$value')" case _ => s"F.expr(${q(expr.sql)})" }