Christopher Lozinski - ZODB, The Graph Database for Python Developers

ZODB is a mature graph database written in Python and optimized in C. Just subclass off of class Persistent Object, and Persistent Container, and your objects, graphs and applications become persistent. This talk teaches you what you need to know to start using a pythonic graph database.

Transcript

1. ZODB The Object-Graph Database for Python Developers By Christopher Lozinski asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #1
2. The Graph Database Market asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #2
3. Market Competition Product Language Funding Model Neo4j Java $80.1 Milion Graph OrientDB Java Graph, Document, KeyValue, Object ArrangpDB Java $7.52 Million Graph, Document, Key Value MarkLogic Java $173 Million XML JSON RDF Allegro Graph Lisp RDF ZODB Python $14 Million Persistetn Python Persistent Python Includes Graph, Document, Json, Key Value and Object. Not possible to write Zopache on top of statically bound Java. asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #3
4. Plone ZODB Users US Plone Sites • Federal Bureau of Investigation (FBI) • Central Intelligence Agency (CIA) • Intellectual Property Rights Center • US Department of Energy • USDA Forest Service • Fermi National Accelerator Lab (Fermilab) • NASA Science • Continental Airlines • UCLA • Yale University • Harvard • The Pennsylvania State University • University of Notre Dame • University of Virginia • University of California - Davis • University of North Carolina • University of Louisville • Novell • Akamai • Disney • eBay • Google • Walmart • Marriott • ...and many more. Worldwide Plone Sites • Brazilian Government • 2016 Olympics Brazil • The British Postal Museum and Archive • The New Zealand Treasury • Konica Minolta Printers - Australia • National Sports Council - Spain • National Library of South Africa • University of Oxford • University of Toronto • Academy of Performing Arts - Prague • Open Society Foundation • Amnesty International • OXFAM • Lufthansa • Nokia • Clean Clothes Campaign • RIPE • Cambridge University • Royal College of Surgeons • Oxford University Clinical Academic Graduate School • ... and many more asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #4
5. Why Use a Graph Database? Social Network Computer Network asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #5
6. Knowledge Graphs asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #6
7. Natural Language Processing Graphagus Sara follows Joe. Sara follows Ben. Sara likes bikes. Sara likes cars. Sara likes cats. Aria follows Joe. Maria loves Joe. Maria likes cars. Joe follows Sara. Joe follows Maria. Joe loves Maria. Joe likes bikes. Joe likes nature. asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #7
8. http://PythonLinks.info/home asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #8
9. So Easy to use! import persistent class TreeLeaf(persistent.Persistent): def __init__(self,title=’’): self.title=title def render(self): return self.title asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #9
10. ZODB Tutorial: Add Leaf Objects Single Leaf Object Multiple LEAF Objects #CREATE A SINGLE LEAF OBJECT leaf = TreeLeaf(‘Leaf’) root.leaf=leaf #CREATE MULTIPLE LEAF OBJECTS Leaf1 = TreeLeaf(‘Green Leaf’) leaf2 = TreeLeaf(‘Red Leaf’) #ADD THEM TO THE ROOT root['leaf1'] = leaf1 root['leaf2'] = leaf2 asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #10
11. ZODB Tutorial: Create a Database http://www.zodb.org/en/latest/tutorial.html import ZODB, ZODB.FileStorage db = ZODB.db(‘Data.fs’) connection = db.open() root = connection.root #DO SOMETHING transaction.commit() asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #11
12. It is just Python (No SQL SELECT) #UPDATE THE Leaf root[‘leaf1’].title=”Yellow Leaf” transaction.commit() #STUPID QUERY For key, item in root.items(): print (key, item) # DELETE AN OBJECT del root[‘leaf1’] transaction.commit() asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #12
13. ZODB is Magical Creates the illusion that your Python Objects are Persistent asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #13
14. ZODB is a graph databases #CREATE THE OBJECTS Leaf1 = TreeLeaf(‘Green Leaf’) leaf2 = TreeLeaf(‘Red Leaf’) #ADD THEM TO THE ROOT root['leaf1'] = leaf1 root['leaf2'] = leaf2 #IT IS A GRAPH DATABASE leaf1.sibling = leaf2 leaf2.sibling = leaf1 transaction.commit() asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #14
15. Persistence by Reachability #BOTH OBJECTS STILL ACCESSIBLE del root['leaf1'] #BOTH OBJECTS GET GARBAGE COLLECTED del root['leaf2'] asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #15
16. ZODB ACID Properties Atomicity: each transaction be "all or nothing". Consistency: any transaction will bring the database from one valid state to another. Isolation: the concurrent execution of transactions results in a system state that would be obtained if transactions were executed sequentially. Durability: once a transaction has been committed, it will remain so. asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #16
17. ZODB Advantages By Jim FUlton No Database Schema No ORM No Referential Integity Problems Automatic Garbage Collection No manual reads and writes asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #17
18. Traversal asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #18
19. Canonical URLs asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #19
20. asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #20 Dolmen Cromlech
21. Multiple Views on Every Object asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #21
22. ZODB Demo CRUD Views http://demo.pythonlinks.info/ Traditional Crud ZODB Extended Crud Create Read Update Create Read Update Delete Delete Rename Cut / Paste Copy / Paste History /Restore asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #22
23. ZODB Crud Demo asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #23
24. History And Undo PythonLinks.info/manage asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #24
25. Create Read Update http://demo.pythonlinks.info/ TreeLeaf ITreeLeaf @implementer(ITreeLeaf) class TreeLeaf(persistent.Persistent): def __init__(self,title=’’): self.title=title def render(self): return self.title class ITreeLeaf(ILeaf): title = TextLine( title='Title', required=True) body = Text( title='Body', required=True) asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #25
26. Different Views on Different Ojects Plone, Pyramid, Grok asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #26
27. Which CRUD is Allowed CRUD Interface Create IAddLeaf IAddBranch Read IDisplayable Update IEditable Delete IDeleteable Rename IRenameable Cut ICuttable Copy ICopyable Paste IPasteable History View IHistory asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #27
28. Which CRUD Is Allowed? TreeLeaf Class ITreeLeaf Class @implementer(ITreeLeaf) class TreeLeaf(persistent.Persistent): def __init__(self,title=’’): self.title=title def render(self): return self.title class ITreeLeaf(IDisplayable, IEditable): title = TextLine( title='Title', required=True) body = Text( title='Body', required=True) asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #28
29. ZEO Scaling asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #29
30. ZODB Storage Options asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #30
31. NewtDB Persistetn Python + PostgeSQL Indexes asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #31
32. File Storage Objects are Written to the end of a File asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #32 Object 3 Object 2 Object 1 Object 4
33. asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #33
34. MQTT Pub/Sub For Real-Time Chat MQTT.org asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #34 Subscribers
35. Storing Chat Logs asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #35 Object 1 Object 2 Object 3
36. Chat Logs After Batching asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #36 Object 1 Object 2 Object 3 Object 3 Version 2 Object 2 Version 2 Object 1 Version 2
37. Speed by Jim Fulton 1000’s of Transactions per second For simple transactions relational databases are slightly faster For complex transactions ZODB is faster. Try to write 10 different clases, in ZODB just append to a file, an RDB requires 10 different writes. asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #37
38. Scalability Several hundred newspaper content-management systems and web sites were hosted using a multi-database configuration with most data in a main database and a catalog database. The databases had several hundred gigabytes of ordinary database records plus multiple terabytes of blob data. For larger systems move to NEO (neo.nexedi.com). Up to 80TB in production. 150TB (and growing) in test. Of course it takes time to move that much data around. asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #38
39. Number of Objects 18,446,744,073,709,551,616 Objects asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #39
40. Security ZeroDB Decrypts Everything on the Client. asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #40
41. s it a Property Graph Database? ZODB vs Relational Databases asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #41
42. ZODB vs Relational Databases ZODB Relational Databases for item in node: print (item) You have to do a database join across every single table. 10 Tables: App, Category, City, Company, Country, iFrame Link, Job, Link, Region, Product, and Video. Zcatalog, Hypatia, repoze.catalog Select statement asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #42
43. ZODB Cell Phone Interface http://PythonLinks.info/ asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #43
44. PythonLinks is a Content Aggregation System asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #44 PyCon UK PyData EuroPython PyCon US PyCon CZ YouTube SciPi PythonLinks. info
45. Content Aggregation System Plone is a Content Management System. Blogory is also a Content Aggregation System asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #45
46. Contact Information Follow @PythonLinks on Twitter Christopher Lozinski Http://PythonLinks.info EMail: lozinski@PythonLinks.info Twitter: @PythonLinks Skype: clozinski US Phone: +1 (650) 614 1836 EU Phone: +48 12 361 3136 asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #46
47. Release Form Name:__________________ Email:__________________ Phone:__________________ I give permission to publish my name , ☐ or to process my professional information to help me get a better job , or to help ☐ me hire a smart Python developer . ☐ asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #47
48. CREATING INDEXES Using Repoze.catalog 1) catalog = Catalog() 2) def get_area(object, default): return getattr(object, ‘area’, default) 3) catalog[‘area’] = CatalogFieldIndex(get_area) 4) leaf-1 = Leaf(area=20) 5) catalog.index_doc(1, leaf-1) 6) catalog.reIndex (1,leaf-1) 7) numdocs, results = catalog.query(Range(20,40)) asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #48