Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ndb
Search
spicyj
May 28, 2014
Technology
0
120
ndb
spicyj
May 28, 2014
Tweet
Share
More Decks by spicyj
See All by spicyj
React: What Lies Ahead
spicyj
6
360
Creating interactive learning interfaces at Khan Academy
spicyj
0
110
Understanding state in React
spicyj
1
100
css
spicyj
2
840
Other Decks in Technology
See All in Technology
ビズリーチが挑む メトリクスを活用した技術的負債の解消 / dev-productivity-con2025
visional_engineering_and_design
3
7.9k
対話型音声AIアプリケーションの信頼性向上の取り組み
ivry_presentationmaterials
1
270
SmartNewsにおける 1000+ノード規模 K8s基盤 でのコスト最適化 – Spot・Gravitonの大規模導入への挑戦
vsanna2
0
140
データグループにおけるフロントエンド開発
lycorptech_jp
PRO
1
110
さくらのIaaS基盤のモニタリングとOpenTelemetry/OSC Hokkaido 2025
fujiwara3
3
460
データ基盤からデータベースまで?広がるユースケースのDatabricksについて教えるよ!
akuwano
3
130
CDKTFについてざっくり理解する!!~CloudFormationからCDKTFへ変換するツールも作ってみた~
masakiokuda
1
170
開発生産性を組織全体の「生産性」へ! 部門間連携の壁を越える実践的ステップ
sudo5in5k
3
7.4k
OSSのSNSツール「Misskey」をさわってみよう(右下ワイプで私のOSCの20年を振り返ります) / 20250705-osc2025-do
akkiesoft
0
170
Tokyo_reInforce_2025_recap_iam_access_analyzer
hiashisan
0
190
Getting to Know Your Legacy (System) with AI-Driven Software Archeology (WeAreDevelopers World Congress 2025)
feststelltaste
1
160
united airlines ™®️ USA Contact Numbers: Complete 2025 Support Guide
flyunitedhelp
1
420
Featured
See All Featured
Fireside Chat
paigeccino
37
3.5k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
44
2.4k
A Modern Web Designer's Workflow
chriscoyier
695
190k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.4k
Music & Morning Musume
bryan
46
6.6k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Practical Orchestrator
shlominoach
189
11k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.7k
Writing Fast Ruby
sferik
628
62k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
Transcript
ndb “NDB is a better datastore API for the Google
App Engine Python runtime.”
Part 1 of 2
Why ndb? 1. Less stupid by default 2. More flexible
queries 3. Tasklets with autobatching
Less stupid by default With db: class UserVideo(db.Model): user_id =
db.StringProperty() video = db.ReferenceProperty(Video) user_video = UserVideo.get_for_video_and_user_data( video, user_data) return jsonify(user_video) # slow
Less stupid by default With ndb: class UserVideo(ndb.Model): user_id =
ndb.StringProperty() video = ndb.KeyProperty(kind=Video) user_video = UserVideo.get_for_video_and_user_data( video, user_data) return jsonify(user_video) # not slow!
More flexible queries ndb lets you build filters using ndb.AND
and ndb.OR: questions = Feedback.query() .filter(Feedback.type == 'question') .filter(Feedback.target == video_key) .filter(ndb.OR( Feedback.is_visible_to_public == True, Feedback.author_user_id == current_id)) .fetch(1000) Magic happens.
Performance The datastore is slow. How can we speed things
up? 4 Batch operations together 4 Do things in parallel 4 Avoid the datastore
Tasklets and autobatching def get_user_exercise_cache(user_data): uec = UEC.get_for_user_data(user_data) if not
uec: user_exercises = UE.get_all(user_data) uec = UEC.build(user_exercises) return uec def get_all_uecs(user_datas): return map(get_user_exercise_cache, user_datas)
Tasklets and autobatching @ndb.tasklet def get_user_exercise_cache_async(user_data): uec = yield UEC.get_for_user_data_async(user_data)
if not uec: user_exercises = yield UE.get_all(user_data) uec = UEC.build(user_exercises) raise ndb.Return(uec) @ndb.synctasklet def get_all_uecs(user_datas): uecs = yield map(get_user_exercise_cache_async, user_datas) raise ndb.Return(uecs)
Moral ndb is awesome. Use it.
Part 2 of 2
The sad truth ndb isn't perfect.
Mysterious errors You heard from Marcia about this gem back
in March: TypeError: '_BaseValue' object is not subscriptable
Q: What's worse than code that doesn't work at all?
A: Code that mostly works but breaks in subtle ways.
Secret slowness #1 Multi-queries, with IN and OR: answers =
Feedback.query() .filter(Feedback.type == 'answer') .filter(Feedback.in_reply_to.IN(question_keys)) .fetch(1000) Doesn't run in parallel!
Secret slowness #1 A not-horribly-slow multi-query: answers = Feedback.query() .filter(Feedback.type
== 'answer') .filter(Feedback.in_reply_to.IN(question_keys)) .order(Feedback.__key__) .fetch(1000)
Secret slowness #2 Query iterators: query = Feedback.query().filter( Feedback.topic_ids ==
'algebra') questions = [] for q in query.iter(batch_size=20): if q.is_visible_to(user_data): questions.append(q) if len(questions) >= 10: break
Secret slowness #2 Solution? Sometimes you have to do it
by hand.
Moral ndb isn't perfect. Pay attention. Profile your code.
The End