Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Anatomy of a translating compiler
Search
Serge Smertin
June 30, 2022
Programming
0
280
Anatomy of a translating compiler
How to build a programming language in Scala?
Serge Smertin
June 30, 2022
Tweet
Share
More Decks by Serge Smertin
See All by Serge Smertin
Introducing the Power of Databricks SDK
nfx
0
130
Reaching 3M downloads and 90%+ unit test coverage for a Terraform provider: lessons learned
nfx
0
230
Distributed Data Mesh, Delta Lake, and Terraform
nfx
0
300
Data Quality with or without Apache Spark and its ecosystem
nfx
0
370
Data Privacy with Apache Spark: Defensive and Offensive Approaches
nfx
0
310
Other Decks in Programming
See All in Programming
Deep Dive 大規模システムアーキテクチャ/開発組織エンジニアリング / Deep Dive Large-Scale System Architecture, Development Organization Engineering
nrslib
15
2.9k
両面どころかインフラもTSでできるよ ~ 全方位TypeScriptによるプロダクト開発 ~
myfinder
7
2.1k
DocC Tutorial と TCA におけるテスト機能の紹介
kalupas226
1
330
Material 3で Material 2ぽい見た目にする
numeroanddev
2
250
Prepare for Jakarta EE 11 - Performance and Developer Productivity
ivargrimstad
0
480
WebComponentsで フレームワークを1ページに共存させる
webuilder240
0
150
htmx is fun!
codehex
2
190
App Router への移行は「改善」となり得るのか?/ Can migration to App Router be an improvement
takefumiyoshii
1
110
PHPerKaigi 2024〜10年以上動いているレガシーなバッチシステムを Kubernetes(Amazon EKS) に移行する取り組み〜
tshinowpub
1
220
OpenAPI を守るのは難しい
ohmori_yusuke
1
140
WinUI 3デモ - "CommunityToolkit.Mvvm"NuGetパッケージ編
andrewkeepcoding
0
130
イベントストーミングによるオブジェクトモデリング・オブジェクト指向プログラミングの適用・開発プロセスの変遷・アーキテクチャの変革 / Object modeling with Event Storming.
nrslib
12
2.3k
Featured
See All Featured
No one is an island. Learnings from fostering a developers community.
thoeni
14
2k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
153
14k
Building Better People: How to give real-time feedback that sticks.
wjessup
350
18k
What's new in Ruby 2.0
geeforr
335
31k
Designing with Data
zakiwarfel
94
4.8k
Web Components: a chance to create the future
zenorocha
304
41k
Navigating Team Friction
lara
177
13k
Build your cross-platform service in a week with App Engine
jlugia
223
17k
Gamification - CAS2011
davidbonilla
76
4.5k
BBQ
matthewcrist
78
8.7k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
101
6.6k
How GitHub (no longer) Works
holman
301
140k
Transcript
Anatomy of a translating compiler … or how to use
debugger all the time. 1 Serge Smertin Senior Resident Solutions Architect Databricks
None
Apache Spark query execution recap
4 code IN(4*, 5*)
5 code IN(4*, 5*)
6 code IN(4*, 5*)
7 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") )))
8 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\'))
9 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\')) private def expression(ctx: LogicalContext , expr: ast.Expr): Expression = expr match { case ast.FieldIn(field, exprs) if exprs.exists(_.isInstanceOf[ast.Wildcard]) => val expand = (expr: ast.Expr) => ast.Binary(ast.Field(field), ast.Equals, expr) expression(ctx, exprs.tail.foldLeft(expand(exprs.head)) { (left, right) => ast.Binary(left, ast.Or, expand(right)) }) case ast.FieldIn(field, exprs) => In(UnresolvedAttribute(field), exprs.map(expression(ctx, _))) case ast.Binary(left, ast.Equals, ast.Wildcard(pattern)) => like(ctx, left, pattern) case ast.Binary(left, symbol, right) => symbol match { case ast.Or => Or(attrOrExpr(ctx, left), attrOrExpr(ctx, right)) // ... } // ... }
10 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\')) display(spark.table('main') .where((F.col('code').like('4%') | F.col('code').like('5%'))))
11 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\')) display(spark.table('main') .where((F.col('code').like('4%') | F.col('code').like('5%')))) case relation: UnresolvedRelation => s"spark.table(${q(relation.name)})" private def unfoldWheres(expr: Expression): String = expr match { case And(left, right) => s"${unfoldWheres(left)}\n${unfoldWheres(right)}" case _ => s".where(${expressionCode(expr)})" } private def expressionCode(expr: Expression): String = expr match { case b: BinaryOperator => val symbol = jvmToPythonOverrides.getOrElse(b.symbol, b.symbol) s"(${expressionCode(b.left)} $symbol ${expressionCode(b.right)})" case attr: UnresolvedAttribute => s"F.col(${q(attr.name)})" case Like(col, Literal(value, _ @ StringType), _) => s"${expressionCode(col)}.like('$value')" case _ => s"F.expr(${q(expr.sql)})" }