Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Importing Wikipedia in Plone
Search
Makina Corpus
October 02, 2013
Technology
1
76
Importing Wikipedia in Plone
Eric BREHAULT – Plone Conference 2013
Makina Corpus
October 02, 2013
Tweet
Share
More Decks by Makina Corpus
See All by Makina Corpus
Publier vos données sur le Web - Forum TIC de l'ATEN 2014
makinacorpus
0
740
Créez votre propre fond de plan à partir de données OSM en utilisant TileMill
makinacorpus
0
130
Team up Django and Web mapping - DjangoCon Europe 2014
makinacorpus
3
870
Petit déjeuner "Les bases de la cartographie sur le Web"
makinacorpus
0
420
Petit déjeuner "Développer sur le cloud, ou comment tout construire à partir de rien" le 11 février - Toulouse
makinacorpus
0
270
CoDe, le programme de développement d'applications mobiles de Makina Corpus
makinacorpus
0
110
Petit déjeuner "Alternatives libres à GoogleMaps" du 11 février 2014 - Nantes - Sylvain Beorchia
makinacorpus
0
660
Petit déjeuner "Les nouveautés de la cartographie en ligne" du 12 décembre
makinacorpus
1
390
Tests carto avec Mocha
makinacorpus
0
810
Other Decks in Technology
See All in Technology
Introduction to Sansan, inc / Sansan Global Development Center, Inc.
sansan33
PRO
0
2.7k
低レイヤソフトウェア技術者が YouTuberとして食っていこうとした話
sat
PRO
7
5.8k
なぜAI時代に 「イベント」を中心に考えるのか? / Why focus on "events" in the age of AI?
ytake
2
380
怖くない!GritQLでBiomeプラグインを作ろうよ
pal4de
1
120
Snowflake のアーキテクチャは本当に筋がよかったのか / Data Engineering Study #30
indigo13love
0
250
スプリントゴール未達症候群に送る処方箋
kakehashi
PRO
1
190
Digitization部 紹介資料
sansan33
PRO
1
4.6k
OpenTelemetry の Log を使いこなそう
biwashi
4
970
機械学習を「社会実装」するということ 2025年夏版 / Social Implementation of Machine Learning July 2025 Version
moepy_stats
1
500
組織内、組織間の資産保護に必要なアイデンティティ基盤と関連技術の最新動向
fujie
0
500
ObsidianをLLM時代のナレッジベースに! クリッピング→Markdown→CLI連携の実践
srvhat09
7
8.9k
Talk to Someone At Delta Airlines™️ USA Contact Numbers
travelcarecenter
0
170
Featured
See All Featured
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.4k
Side Projects
sachag
455
43k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Intergalactic Javascript Robots from Outer Space
tanoku
271
27k
Measuring & Analyzing Core Web Vitals
bluesmoon
7
530
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
108
19k
Building an army of robots
kneath
306
45k
Making the Leap to Tech Lead
cromwellryan
134
9.4k
The Language of Interfaces
destraynor
158
25k
Mobile First: as difficult as doing things right
swwweet
223
9.7k
Building Adaptive Systems
keathley
43
2.7k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Transcript
Importing Wikipedia in Plone Eric BREHAULT – Plone Conference 2013
ZODB is good at storing objects • Plone contents are
objects, • we store them in the ZODB, • everything is fine, end of the story.
But what if ... ... we want to store non-contentish
records? Like polls, statistics, mail-list subscribers, etc., or any business-specific structured data.
Store them as contents anyway That is a powerfull solution.
But there are 2 major problems...
Problem 1: You need to manage a secondary system •
you need to deploy it, • you need to backup it, • you need to secure it, • etc.
Problem 2: I hate SQL No explanation here.
I think I just cannot digest it...
How to store many records in the ZODB? • Is
the ZODB strong enough? • Is the ZCatalog strong enough?
My grandmother often told me "If you want to become
stronger, you have to eat your soup."
Where do we find a good soup for Plone? In
a super souper!!!
souper.plone and souper • It provides both storage and indexing.
• Record can store any persistent pickable data. • Created by BlueDynamics. • Based on ZODB BTrees, node.ext.zodb, and repoze.catalog.
Add a record >>> soup = get_soup('mysoup', context) >>> record
= Record() >>> record.attrs['user'] = 'user1' >>> record.attrs['text'] = u'foo bar baz' >>> record.attrs['keywords'] = [u'1', u'2', u'ü'] >>> record_id = soup.add(record)
Record in record >>> record['homeaddress'] = Record() >>> record['homeaddress'].attrs['zip'] =
'6020' >>> record['homeaddress'].attrs['town'] = 'Innsbruck' >>> record['homeaddress'].attrs['country'] = 'Austria'
Access record >>> from souper.soup import get_soup >>> soup =
get_soup('mysoup', context) >>> record = soup.get(record_id)
Query >>> from repoze.catalog.query import Eq, Contains >>> [r for
r in soup.query(Eq('user', 'user1') & Contains('text', 'foo'))] [<Record object 'None' at ...>] or using CQE format >>> [r for r in soup.query("user == 'user1' and 'foo' in text")] [<Record object 'None' at ...>]
souper • a Soup-container can be moved to a specific
ZODB mount- point, • it can be shared across multiple independent Plone instances, • souper works on Plone and Pyramid.
Plomino & souper • we use Plomino to build non-content
oriented apps easily, • we use souper to store huge amount of application data.
Plomino data storage Originally, documents (=record) were ATFolder. Capacity about
30 000.
Plomino data storage Since 1.14, documents are pure CMF. Capacity
about 100 000. Usally the Plomino ZCatalog contains a lot of indexes.
Plomino & souper With souper, documents are just soup records.
Capacity: several millions.
Typical use case • Store 500 000 addresses, • Be
able to query them in full text and display the result on a map. Demo
What is the limit? Can we import Wikipedia in souper?
Demo with 400 000 records Demo with 5,5 millions of records
Conclusion • Usage performances are good, • Plone performances are
not impacted. Use it!
Thoughts • What about a REST API on top of
it? • Massive import is long and difficult, could it be improved?
Makina Corpus For all questions related to this talk, please
contact Éric Bréhault
[email protected]
Tel : +33 534 566 958 www.makina-corpus.com