Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Anatomy of a translating compiler
Search
Serge Smertin
June 30, 2022
Programming
0
420
Anatomy of a translating compiler
How to build a programming language in Scala?
Serge Smertin
June 30, 2022
Tweet
Share
More Decks by Serge Smertin
See All by Serge Smertin
Building Databricks integrations with Go
nfx
0
320
Building robust Python applications on top of Databricks: UCX case study
nfx
0
310
Introducing the Power of Databricks SDK
nfx
0
430
Reaching 3M downloads and 90%+ unit test coverage for a Terraform provider: lessons learned
nfx
0
500
Distributed Data Mesh, Delta Lake, and Terraform
nfx
0
500
Data Quality with or without Apache Spark and its ecosystem
nfx
0
590
Data Privacy with Apache Spark: Defensive and Offensive Approaches
nfx
0
590
Other Decks in Programming
See All in Programming
Updates on MLS on Ruby (and maybe more)
sylph01
1
180
Processing Gem ベースの、2D レトロゲームエンジンの開発
tokujiros
2
120
「待たせ上手」なスケルトンスクリーン、 そのUXの裏側
teamlab
PRO
0
480
請來的 AI Agent 同事們在寫程式時,怎麼用 pytest 去除各種幻想與盲點
keitheis
0
110
Tool Catalog Agent for Bedrock AgentCore Gateway
licux
6
2.3k
Navigation 2 を 3 に移行する(予定)ためにやったこと
yokomii
0
120
プロパティベーステストによるUIテスト: LLMによるプロパティ定義生成でエッジケースを捉える
tetta_pdnt
0
300
Ruby×iOSアプリ開発 ~共に歩んだエコシステムの物語~
temoki
0
270
AI時代のUIはどこへ行く?
yusukebe
17
8.7k
ユーザーも開発者も悩ませない TV アプリ開発 ~Compose の内部実装から学ぶフォーカス制御~
taked137
0
140
アプリの "かわいい" を支えるアニメーションツールRiveについて
uetyo
0
220
Flutter with Dart MCP: All You Need - 박제창 2025 I/O Extended Busan
itsmedreamwalker
0
150
Featured
See All Featured
Gamification - CAS2011
davidbonilla
81
5.4k
Java REST API Framework Comparison - PWX 2021
mraible
33
8.8k
It's Worth the Effort
3n
187
28k
Documentation Writing (for coders)
carmenintech
74
5k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.9k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.6k
Agile that works and the tools we love
rasmusluckow
330
21k
Side Projects
sachag
455
43k
Reflections from 52 weeks, 52 projects
jeffersonlam
352
21k
Building Applications with DynamoDB
mza
96
6.6k
Typedesign – Prime Four
hannesfritz
42
2.8k
Transcript
Anatomy of a translating compiler … or how to use
debugger all the time. 1 Serge Smertin Senior Resident Solutions Architect Databricks
None
Apache Spark query execution recap
4 code IN(4*, 5*)
5 code IN(4*, 5*)
6 code IN(4*, 5*)
7 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") )))
8 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\'))
9 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\')) private def expression(ctx: LogicalContext , expr: ast.Expr): Expression = expr match { case ast.FieldIn(field, exprs) if exprs.exists(_.isInstanceOf[ast.Wildcard]) => val expand = (expr: ast.Expr) => ast.Binary(ast.Field(field), ast.Equals, expr) expression(ctx, exprs.tail.foldLeft(expand(exprs.head)) { (left, right) => ast.Binary(left, ast.Or, expand(right)) }) case ast.FieldIn(field, exprs) => In(UnresolvedAttribute(field), exprs.map(expression(ctx, _))) case ast.Binary(left, ast.Equals, ast.Wildcard(pattern)) => like(ctx, left, pattern) case ast.Binary(left, symbol, right) => symbol match { case ast.Or => Or(attrOrExpr(ctx, left), attrOrExpr(ctx, right)) // ... } // ... }
10 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\')) display(spark.table('main') .where((F.col('code').like('4%') | F.col('code').like('5%'))))
11 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\')) display(spark.table('main') .where((F.col('code').like('4%') | F.col('code').like('5%')))) case relation: UnresolvedRelation => s"spark.table(${q(relation.name)})" private def unfoldWheres(expr: Expression): String = expr match { case And(left, right) => s"${unfoldWheres(left)}\n${unfoldWheres(right)}" case _ => s".where(${expressionCode(expr)})" } private def expressionCode(expr: Expression): String = expr match { case b: BinaryOperator => val symbol = jvmToPythonOverrides.getOrElse(b.symbol, b.symbol) s"(${expressionCode(b.left)} $symbol ${expressionCode(b.right)})" case attr: UnresolvedAttribute => s"F.col(${q(attr.name)})" case Like(col, Literal(value, _ @ StringType), _) => s"${expressionCode(col)}.like('$value')" case _ => s"F.expr(${q(expr.sql)})" }