Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Anatomy of a translating compiler
Search
Serge Smertin
June 30, 2022
Programming
0
360
Anatomy of a translating compiler
How to build a programming language in Scala?
Serge Smertin
June 30, 2022
Tweet
Share
More Decks by Serge Smertin
See All by Serge Smertin
Building Databricks integrations with Go
nfx
0
110
Building robust Python applications on top of Databricks: UCX case study
nfx
0
98
Introducing the Power of Databricks SDK
nfx
0
240
Reaching 3M downloads and 90%+ unit test coverage for a Terraform provider: lessons learned
nfx
0
290
Distributed Data Mesh, Delta Lake, and Terraform
nfx
0
410
Data Quality with or without Apache Spark and its ecosystem
nfx
0
480
Data Privacy with Apache Spark: Defensive and Offensive Approaches
nfx
0
450
Other Decks in Programming
See All in Programming
go.mod、DockerfileやCI設定に分散しがちなGoのバージョンをまとめて管理する / Go Connect #3
arthur1
10
2.3k
What’s New in Compose Multiplatform - A Live Tour (droidcon London 2024)
zsmb
1
330
組織に自動テストを書く文化を根付かせる戦略(2024秋版) / Building Automated Test Culture 2024 Autumn Edition
twada
PRO
10
4.5k
Nuxtベースの「WXT」でChrome拡張を作成する | Vue Fes 2024 ランチセッション
moshi1121
1
490
Macとオーディオ再生 2024/11/02
yusukeito
0
150
hotwire_or_react
harunatsujita
8
4k
macOS でできる リアルタイム動画像処理
biacco42
4
1.6k
Android 15 でアクションバー表示時にステータスバーが白くなってしまう問題
tonionagauzzi
0
140
2万ページのSSG運用における工夫と注意点 / Vue Fes Japan 2024
chinen
3
1.3k
Streams APIとTCPフロー制御 / Web Streams API and TCP flow control
tasshi
1
290
PHP でアセンブリ言語のように書く技術
memory1994
PRO
1
150
約9000個の自動テストの 時間を50分->10分に短縮 Flakyテストを1%以下に抑えた話
hatsu38
23
11k
Featured
See All Featured
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.2k
Unsuck your backbone
ammeep
668
57k
Thoughts on Productivity
jonyablonski
67
4.3k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
41
2.1k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
26
2k
Become a Pro
speakerdeck
PRO
24
5k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
9
680
How STYLIGHT went responsive
nonsquared
95
5.2k
Happy Clients
brianwarren
97
6.7k
How to Think Like a Performance Engineer
csswizardry
19
1.1k
Building Flexible Design Systems
yeseniaperezcruz
327
38k
Transcript
Anatomy of a translating compiler … or how to use
debugger all the time. 1 Serge Smertin Senior Resident Solutions Architect Databricks
None
Apache Spark query execution recap
4 code IN(4*, 5*)
5 code IN(4*, 5*)
6 code IN(4*, 5*)
7 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") )))
8 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\'))
9 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\')) private def expression(ctx: LogicalContext , expr: ast.Expr): Expression = expr match { case ast.FieldIn(field, exprs) if exprs.exists(_.isInstanceOf[ast.Wildcard]) => val expand = (expr: ast.Expr) => ast.Binary(ast.Field(field), ast.Equals, expr) expression(ctx, exprs.tail.foldLeft(expand(exprs.head)) { (left, right) => ast.Binary(left, ast.Or, expand(right)) }) case ast.FieldIn(field, exprs) => In(UnresolvedAttribute(field), exprs.map(expression(ctx, _))) case ast.Binary(left, ast.Equals, ast.Wildcard(pattern)) => like(ctx, left, pattern) case ast.Binary(left, symbol, right) => symbol match { case ast.Or => Or(attrOrExpr(ctx, left), attrOrExpr(ctx, right)) // ... } // ... }
10 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\')) display(spark.table('main') .where((F.col('code').like('4%') | F.col('code').like('5%'))))
11 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\')) display(spark.table('main') .where((F.col('code').like('4%') | F.col('code').like('5%')))) case relation: UnresolvedRelation => s"spark.table(${q(relation.name)})" private def unfoldWheres(expr: Expression): String = expr match { case And(left, right) => s"${unfoldWheres(left)}\n${unfoldWheres(right)}" case _ => s".where(${expressionCode(expr)})" } private def expressionCode(expr: Expression): String = expr match { case b: BinaryOperator => val symbol = jvmToPythonOverrides.getOrElse(b.symbol, b.symbol) s"(${expressionCode(b.left)} $symbol ${expressionCode(b.right)})" case attr: UnresolvedAttribute => s"F.col(${q(attr.name)})" case Like(col, Literal(value, _ @ StringType), _) => s"${expressionCode(col)}.like('$value')" case _ => s"F.expr(${q(expr.sql)})" }