Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Anatomy of a translating compiler
Search
Serge Smertin
June 30, 2022
Programming
0
420
Anatomy of a translating compiler
How to build a programming language in Scala?
Serge Smertin
June 30, 2022
Tweet
Share
More Decks by Serge Smertin
See All by Serge Smertin
Building Databricks integrations with Go
nfx
0
320
Building robust Python applications on top of Databricks: UCX case study
nfx
0
300
Introducing the Power of Databricks SDK
nfx
0
430
Reaching 3M downloads and 90%+ unit test coverage for a Terraform provider: lessons learned
nfx
0
490
Distributed Data Mesh, Delta Lake, and Terraform
nfx
0
490
Data Quality with or without Apache Spark and its ecosystem
nfx
0
580
Data Privacy with Apache Spark: Defensive and Offensive Approaches
nfx
0
580
Other Decks in Programming
See All in Programming
エンジニアのための”最低限いい感じ”デザイン入門
shunshobon
0
110
Claude Code と OpenAI o3 で メタデータ情報を作る
laket
0
130
書き捨てではなく継続開発可能なコードをAIコーディングエージェントで書くために意識していること
shuyakinjo
1
280
プロダクトという一杯を作る - プロダクトチームが味の責任を持つまでの煮込み奮闘記
hiliteeternal
0
460
自作OSでDOOMを動かしてみた
zakki0925224
1
1.3k
0から始めるモジュラーモノリス-クリーンなモノリスを目指して
sushi0120
1
280
新しいモバイルアプリ勉強会(仮)について
uetyo
1
260
一人でAIプロダクトを作るための工夫 〜技術選定・開発プロセス編〜 / I want AI to work harder
rkaga
12
2.6k
パスタの技術
yusukebe
1
380
#QiitaBash TDDで(自分の)開発がどう変わったか
ryosukedtomita
1
370
Webinar: AI-Powered Development: Transformiere deinen Workflow mit Coding Tools und MCP Servern
danielsogl
0
130
あのころの iPod を どうにか再生させたい
orumin
2
2.5k
Featured
See All Featured
Music & Morning Musume
bryan
46
6.7k
Code Reviewing Like a Champion
maltzj
525
40k
Raft: Consensus for Rubyists
vanstee
140
7.1k
Optimizing for Happiness
mojombo
379
70k
Navigating Team Friction
lara
188
15k
Typedesign – Prime Four
hannesfritz
42
2.8k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
183
54k
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
Optimising Largest Contentful Paint
csswizardry
37
3.4k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.4k
Transcript
Anatomy of a translating compiler … or how to use
debugger all the time. 1 Serge Smertin Senior Resident Solutions Architect Databricks
None
Apache Spark query execution recap
4 code IN(4*, 5*)
5 code IN(4*, 5*)
6 code IN(4*, 5*)
7 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") )))
8 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\'))
9 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\')) private def expression(ctx: LogicalContext , expr: ast.Expr): Expression = expr match { case ast.FieldIn(field, exprs) if exprs.exists(_.isInstanceOf[ast.Wildcard]) => val expand = (expr: ast.Expr) => ast.Binary(ast.Field(field), ast.Equals, expr) expression(ctx, exprs.tail.foldLeft(expand(exprs.head)) { (left, right) => ast.Binary(left, ast.Or, expand(right)) }) case ast.FieldIn(field, exprs) => In(UnresolvedAttribute(field), exprs.map(expression(ctx, _))) case ast.Binary(left, ast.Equals, ast.Wildcard(pattern)) => like(ctx, left, pattern) case ast.Binary(left, symbol, right) => symbol match { case ast.Or => Or(attrOrExpr(ctx, left), attrOrExpr(ctx, right)) // ... } // ... }
10 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\')) display(spark.table('main') .where((F.col('code').like('4%') | F.col('code').like('5%'))))
11 SearchCommand( FieldIn("code", Seq( Wildcard("4*"), Wildcard("5*") ))) Or( Like(UnresolvedAttribute("code"), Literal.create("4%"),
'\\'), Like(UnresolvedAttribute("code"), Literal.create("5%"), '\\')) display(spark.table('main') .where((F.col('code').like('4%') | F.col('code').like('5%')))) case relation: UnresolvedRelation => s"spark.table(${q(relation.name)})" private def unfoldWheres(expr: Expression): String = expr match { case And(left, right) => s"${unfoldWheres(left)}\n${unfoldWheres(right)}" case _ => s".where(${expressionCode(expr)})" } private def expressionCode(expr: Expression): String = expr match { case b: BinaryOperator => val symbol = jvmToPythonOverrides.getOrElse(b.symbol, b.symbol) s"(${expressionCode(b.left)} $symbol ${expressionCode(b.right)})" case attr: UnresolvedAttribute => s"F.col(${q(attr.name)})" case Like(col, Literal(value, _ @ StringType), _) => s"${expressionCode(col)}.like('$value')" case _ => s"F.expr(${q(expr.sql)})" }