Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Refactoring a Solr based api application
Search
Torsten Bøgh Köster
April 13, 2012
Programming
3
110
Refactoring a Solr based api application
Held on Apache Lucene Eurocon 2011 in Barcelona
Torsten Bøgh Köster
April 13, 2012
Tweet
Share
More Decks by Torsten Bøgh Köster
See All by Torsten Bøgh Köster
Taking an abandoned Solr search from zero to GenAI hero
tboeghk
0
18
Oder mache ich es lieber selbst? Wie sich Kosten und Geopolitik auf Cloud-Betrieb auswirken
tboeghk
0
35
🔪 How we cut our AWS costs in half
tboeghk
0
270
Shared Nothing Logging Infrastructure
tboeghk
0
120
Beyond Cloud: A road trip into AWS and back to bare metal
tboeghk
1
100
Shared Nothing Logging Infrastructure
tboeghk
0
1.3k
Kubernetes the ❤️ way
tboeghk
0
1.1k
Beyond Cloud: A road trip into AWS and back to bare metal
tboeghk
0
100
Open-Source-Logging und -Monitoring (W-JAX 2017)
tboeghk
0
99
Other Decks in Programming
See All in Programming
UbieのAIパートナーを支えるコンテキストエンジニアリング実践
syucream
2
830
FindyにおけるTakumi活用と脆弱性管理のこれから
rvirus0817
0
420
旅行プランAIエージェント開発の裏側
ippo012
2
850
Ruby Parser progress report 2025
yui_knk
1
300
CloudflareのChat Agent Starter Kitで簡単!AIチャットボット構築
syumai
2
430
Vue・React マルチプロダクト開発を支える Vite
andpad
0
110
「手軽で便利」に潜む罠。 Popover API を WCAG 2.2の視点で安全に使うには
taitotnk
0
740
Honoアップデート 2025年夏
yusukebe
1
920
The Past, Present, and Future of Enterprise Java
ivargrimstad
0
230
HTMLの品質ってなんだっけ? “HTMLクライテリア”の設計と実践
unachang113
4
2.2k
AWS発のAIエディタKiroを使ってみた
iriikeita
1
160
レガシープロジェクトで最大限AIの恩恵を受けられるようClaude Codeを利用する
tk1351
4
1.6k
Featured
See All Featured
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.4k
GitHub's CSS Performance
jonrohan
1032
460k
Why You Should Never Use an ORM
jnunemaker
PRO
59
9.5k
GraphQLとの向き合い方2022年版
quramy
49
14k
Reflections from 52 weeks, 52 projects
jeffersonlam
352
21k
RailsConf 2023
tenderlove
30
1.2k
Writing Fast Ruby
sferik
628
62k
What's in a price? How to price your products and services
michaelherold
246
12k
Thoughts on Productivity
jonyablonski
70
4.8k
Raft: Consensus for Rubyists
vanstee
140
7.1k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Visualization
eitanlees
148
16k
Transcript
Architectural lessons learned from refactoring a Solr based API application.
Torsten Bøgh Köster (Shopping24) Apache Lucene Eurocon, 19.10.2011
Contents Shopping24 and it‘s API Technical scaling solutions Sharding Caching
Solr Cores „Elastic“ infrastructure business requirements as key factor
@tboeghk Software- and systems- architect 2 years experience with Solr
3 years experience with Lucene Team of 7 Java developers currently at Shopping24
shopping24 internet group
1 portal became n portals
30 partner shops became 700
500k to 7m documents
index fact time •16 Gig Data •Single-Core-Layout •Up to 17s
response time •Machine size limited •Stalled at solr version 1.4 •API designed for small tools
scaling goal: 15-50m documents
ask the nerds „Shard!“ That‘ll be fun! „Use spare compute
cores at Amazon?“ breathe load into the cloud „Reduce that index size“ „Get rid of those long running queries!“
data sharding ...
... is highly effective. 125ms 250ms 375ms 500ms 1 4
8 12 16 20 1shard 2shard 3shard 4shard 6shard 8shard concurrent requests
Sharding: size matters the bigger your index gets, the more
complex your queries are, the more concurrent requests, the more sharding you need
but wait ...
Why do we have such a big index?
7m documents vs. 2m active poducts
fashion product lifecycle meets SEO Bastografie / photocase.com
Separation of duties! Remove unsearchable data from your index.
Why do we have complex queries?
A Solr index designed for 1 portal
Grown into a multi-portal index
Let “sharding“ follow your data ...
... and build separate cores for every client.
Duplicate data as long as access is fast. andybahn /
photocase.com
Streamline your index provisioning process.
A thousand splendid cores at your fingertips.
Throwing hardware at problems. Automated.
evil traps: latency, $$
mirror your complete system – solve load balancer problems froodmat
/ photocase.com
I said faster!
use a cache layer like Varnish.
What about those complex queries? Why do we have them?
And how do we get rid of them?
Lost in encapsulation: Solr API exposed to world.
What‘s the key factor?
look at your business requirements
decrease complexity
Questions? Comments? Ideas? Twitter: @tboeghk Github: @tboeghk Email:
[email protected]
Web:
http://www.s24.com Images: sxc.hu (unless noted otherwise)