Christopher Lozinski - ZODB, The Graph Database for Python Developers

Slide 1

Slide 1 text

ZODB The Object-Graph Database for Python Developers By Christopher Lozinski asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #1

Slide 2

Slide 2 text

The Graph Database Market asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #2

Slide 3

Slide 3 text

Market Competition Product Language Funding Model Neo4j Java $80.1 Milion Graph OrientDB Java Graph, Document, KeyValue, Object ArrangpDB Java $7.52 Million Graph, Document, Key Value MarkLogic Java $173 Million XML JSON RDF Allegro Graph Lisp RDF ZODB Python $14 Million Persistetn Python Persistent Python Includes Graph, Document, Json, Key Value and Object. Not possible to write Zopache on top of statically bound Java. asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #3

Slide 4

Slide 4 text

Plone ZODB Users US Plone Sites • Federal Bureau of Investigation (FBI) • Central Intelligence Agency (CIA) • Intellectual Property Rights Center • US Department of Energy • USDA Forest Service • Fermi National Accelerator Lab (Fermilab) • NASA Science • Continental Airlines • UCLA • Yale University • Harvard • The Pennsylvania State University • University of Notre Dame • University of Virginia • University of California - Davis • University of North Carolina • University of Louisville • Novell • Akamai • Disney • eBay • Google • Walmart • Marriott • ...and many more. Worldwide Plone Sites • Brazilian Government • 2016 Olympics Brazil • The British Postal Museum and Archive • The New Zealand Treasury • Konica Minolta Printers - Australia • National Sports Council - Spain • National Library of South Africa • University of Oxford • University of Toronto • Academy of Performing Arts - Prague • Open Society Foundation • Amnesty International • OXFAM • Lufthansa • Nokia • Clean Clothes Campaign • RIPE • Cambridge University • Royal College of Surgeons • Oxford University Clinical Academic Graduate School • ... and many more asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #4

Slide 5

Slide 5 text

Why Use a Graph Database? Social Network Computer Network asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #5

Slide 6

Slide 6 text

Knowledge Graphs asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #6

Slide 7

Slide 7 text

Natural Language Processing Graphagus Sara follows Joe. Sara follows Ben. Sara likes bikes. Sara likes cars. Sara likes cats. Aria follows Joe. Maria loves Joe. Maria likes cars. Joe follows Sara. Joe follows Maria. Joe loves Maria. Joe likes bikes. Joe likes nature. asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #7

Slide 8

Slide 8 text

http://PythonLinks.info/home asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #8

Slide 9

Slide 9 text

So Easy to use! import persistent class TreeLeaf(persistent.Persistent): def __init__(self,title=’’): self.title=title def render(self): return self.title asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #9

Slide 10

Slide 10 text

ZODB Tutorial: Add Leaf Objects Single Leaf Object Multiple LEAF Objects #CREATE A SINGLE LEAF OBJECT leaf = TreeLeaf(‘Leaf’) root.leaf=leaf #CREATE MULTIPLE LEAF OBJECTS Leaf1 = TreeLeaf(‘Green Leaf’) leaf2 = TreeLeaf(‘Red Leaf’) #ADD THEM TO THE ROOT root['leaf1'] = leaf1 root['leaf2'] = leaf2 asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #10

Slide 11

Slide 11 text

ZODB Tutorial: Create a Database http://www.zodb.org/en/latest/tutorial.html import ZODB, ZODB.FileStorage db = ZODB.db(‘Data.fs’) connection = db.open() root = connection.root #DO SOMETHING transaction.commit() asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #11

Slide 12

Slide 12 text

It is just Python (No SQL SELECT) #UPDATE THE Leaf root[‘leaf1’].title=”Yellow Leaf” transaction.commit() #STUPID QUERY For key, item in root.items(): print (key, item) # DELETE AN OBJECT del root[‘leaf1’] transaction.commit() asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #12

Slide 13

Slide 13 text

ZODB is Magical Creates the illusion that your Python Objects are Persistent asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #13

Slide 14

Slide 14 text

ZODB is a graph databases #CREATE THE OBJECTS Leaf1 = TreeLeaf(‘Green Leaf’) leaf2 = TreeLeaf(‘Red Leaf’) #ADD THEM TO THE ROOT root['leaf1'] = leaf1 root['leaf2'] = leaf2 #IT IS A GRAPH DATABASE leaf1.sibling = leaf2 leaf2.sibling = leaf1 transaction.commit() asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #14

Slide 15

Slide 15 text

Persistence by Reachability #BOTH OBJECTS STILL ACCESSIBLE del root['leaf1'] #BOTH OBJECTS GET GARBAGE COLLECTED del root['leaf2'] asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #15

Slide 16

Slide 16 text

ZODB ACID Properties Atomicity: each transaction be "all or nothing". Consistency: any transaction will bring the database from one valid state to another. Isolation: the concurrent execution of transactions results in a system state that would be obtained if transactions were executed sequentially. Durability: once a transaction has been committed, it will remain so. asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #16

Slide 17

Slide 17 text

ZODB Advantages By Jim FUlton No Database Schema No ORM No Referential Integity Problems Automatic Garbage Collection No manual reads and writes asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #17

Slide 18

Slide 18 text

Traversal asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #18

Slide 19

Slide 19 text

Canonical URLs asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #19

Slide 20

Slide 20 text

asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #20 Dolmen Cromlech

Slide 21

Slide 21 text

Multiple Views on Every Object asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #21

Slide 22

Slide 22 text

ZODB Demo CRUD Views http://demo.pythonlinks.info/ Traditional Crud ZODB Extended Crud Create Read Update Create Read Update Delete Delete Rename Cut / Paste Copy / Paste History /Restore asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #22

Slide 23

Slide 23 text

ZODB Crud Demo asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #23

Slide 24

Slide 24 text

History And Undo PythonLinks.info/manage asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #24

Slide 25

Slide 25 text

Create Read Update http://demo.pythonlinks.info/ TreeLeaf ITreeLeaf @implementer(ITreeLeaf) class TreeLeaf(persistent.Persistent): def __init__(self,title=’’): self.title=title def render(self): return self.title class ITreeLeaf(ILeaf): title = TextLine( title='Title', required=True) body = Text( title='Body', required=True) asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #25

Slide 26

Slide 26 text

Different Views on Different Ojects Plone, Pyramid, Grok asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #26

Slide 27

Slide 27 text

Which CRUD is Allowed CRUD Interface Create IAddLeaf IAddBranch Read IDisplayable Update IEditable Delete IDeleteable Rename IRenameable Cut ICuttable Copy ICopyable Paste IPasteable History View IHistory asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #27

Slide 28

Slide 28 text

Which CRUD Is Allowed? TreeLeaf Class ITreeLeaf Class @implementer(ITreeLeaf) class TreeLeaf(persistent.Persistent): def __init__(self,title=’’): self.title=title def render(self): return self.title class ITreeLeaf(IDisplayable, IEditable): title = TextLine( title='Title', required=True) body = Text( title='Body', required=True) asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #28

Slide 29

Slide 29 text

ZEO Scaling asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #29

Slide 30

Slide 30 text

ZODB Storage Options asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #30

Slide 31

Slide 31 text

NewtDB Persistetn Python + PostgeSQL Indexes asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #31

Slide 32

Slide 32 text

File Storage Objects are Written to the end of a File asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #32 Object 3 Object 2 Object 1 Object 4

Slide 33

Slide 33 text

asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #33

Slide 34

Slide 34 text

MQTT Pub/Sub For Real-Time Chat MQTT.org asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #34 Subscribers

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Chat Logs After Batching asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #36 Object 1 Object 2 Object 3 Object 3 Version 2 Object 2 Version 2 Object 1 Version 2

Slide 37

Slide 37 text

Speed by Jim Fulton 1000’s of Transactions per second For simple transactions relational databases are slightly faster For complex transactions ZODB is faster. Try to write 10 different clases, in ZODB just append to a file, an RDB requires 10 different writes. asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #37

Slide 38

Slide 38 text

Scalability Several hundred newspaper content-management systems and web sites were hosted using a multi-database configuration with most data in a main database and a catalog database. The databases had several hundred gigabytes of ordinary database records plus multiple terabytes of blob data. For larger systems move to NEO (neo.nexedi.com). Up to 80TB in production. 150TB (and growing) in test. Of course it takes time to move that much data around. asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #38

Slide 39

Slide 39 text

Number of Objects 18,446,744,073,709,551,616 Objects asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #39

Slide 40

Slide 40 text

Slide 41

Slide 41 text

s it a Property Graph Database? ZODB vs Relational Databases asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #41

Slide 42

Slide 42 text

ZODB vs Relational Databases ZODB Relational Databases for item in node: print (item) You have to do a database join across every single table. 10 Tables: App, Category, City, Company, Country, iFrame Link, Job, Link, Region, Product, and Video. Zcatalog, Hypatia, repoze.catalog Select statement asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #42

Slide 43

Slide 43 text

Slide 44

Slide 44 text

PythonLinks is a Content Aggregation System asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #44 PyCon UK PyData EuroPython PyCon US PyCon CZ YouTube SciPi PythonLinks. info

Slide 45

Slide 45 text

Content Aggregation System Plone is a Content Management System. Blogory is also a Content Aggregation System asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #45

Slide 46

Slide 46 text

Contact Information Follow @PythonLinks on Twitter Christopher Lozinski Http://PythonLinks.info EMail: [email protected] Twitter: @PythonLinks Skype: clozinski US Phone: +1 (650) 614 1836 EU Phone: +48 12 361 3136 asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #46

Slide 47

Slide 47 text

Release Form Name:__________________ Email:__________________ Phone:__________________ I give permission to publish my name , ☐ or to process my professional information to help me get a better job , or to help ☐ me hire a smart Python developer . ☐ asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #47

Slide 48

Slide 48 text

CREATING INDEXES Using Repoze.catalog 1) catalog = Catalog() 2) def get_area(object, default): return getattr(object, ‘area’, default) 3) catalog[‘area’] = CatalogFieldIndex(get_area) 4) leaf-1 = Leaf(area=20) 5) catalog.index_doc(1, leaf-1) 6) catalog.reIndex (1,leaf-1) 7) numdocs, results = catalog.query(Range(20,40)) asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #48