MongoDB vs PostgreSQL Performance

Benchmarking is hard. Benchmarking databases, harder. Benchmarking databases that follow different approaches (relational vs document) is even harder.

But the market demands these kinds of benchmarks. Despite the different data models that MongoDB and PostgreSQL expose, many organizations face the challenge of picking either technology. And performance is arguably the main deciding factor.

Join this talk to discover the numbers! After $30K spent on public cloud and months of testing, there are many different scenarios to analyze. Benchmarks on three distinct categories have been performed: OLTP, OLAP and comparing MongoDB 4.0 transaction performance with PostgreSQL's.

Transcript

1. MONGODB vs POSTGRESQL BENCHMARKS
   MONGODB vs POSTGRESQL
   BENCHMARKS
   Álvaro Hernández
   

   View Slide

2. MONGODB vs POSTGRESQL BENCHMARKS
   ` whoami`
   Álvaro Hernández
   @ahachete
   ● Founder & CEO, OnGres
   ● 20+ years PostgreSQL user and DBA
   ● Mostly doing R&D to create new, innovative
   software on Postgres
   ● Frequent speaker at PostgreSQL, database
   conferences
   ● Principal Architect of ToroDB
   ● Founder and President of the NPO
   Fundación PostgreSQL
   

   View Slide

3. MONGODB vs POSTGRESQL BENCHMARKS
   Introduction
   

   View Slide

4. MONGODB vs POSTGRESQL BENCHMARKS
   OnGres Ethics Policy
   This work was sponsored by EnterpriseDB, and performed by OnGres.
   It was conducted according to OnGres Ethics Policy, that observes that:
   ● All the work is conducted with the maximum degree of professionalism
   and independence.
   ● No technology is favored over another.
   ● No results are edited or omitted.
   ● The sponsor of the work does not intervene in the strategy,
   implementation or execution of the work.
   ● Results are verifiable by external, third parties.
   

   View Slide

5. MONGODB vs POSTGRESQL BENCHMARKS
   Benchmarking is hard
   ● Bench-marketing is easy --but not trustable.
   ● Benchmarking is hard.
   ● Benchmarking databases is harder.
   ● Benchmarking databases that follow different design models, is
   even harder.
   ● Are MongoDB and PostgreSQL comparable?
   ● The market demands this: informed decision, performance is key.
   

   View Slide

6. MONGODB vs POSTGRESQL BENCHMARKS
   Pursuing benchmarking fairness
   How to present a fair arena in which the technologies compete in an
   apples-to-apples scenario?
   ● Transparency and reproducibility. Infrastructure-as-Code.
   https://gitlab.com/ongresinc/benchplatform/
   https://gitlab.com/ongresinc/txbenchmark/
   http://benchplatform.ongres.com.s3.amazonaws.com/
   ● Multiple benchmarks
   ● Close-to-real workloads
   ● Production-grade setups
   

   View Slide

7. MONGODB vs POSTGRESQL BENCHMARKS
   Types of benchmarks
   Three main benchmark categories:
   ● Transactions benchmark
   ● OLTP
   ○ In-memory dataset (4GB)
   ○ Larger, on-disk dataset (2TB)
   ● OLAP
   

   View Slide

8. MONGODB vs POSTGRESQL BENCHMARKS
   The contenders
   MongoDB 4.0
   ● Community version used
   ● Journaling active
   ● Replication active
   (single node)
   ● No further tuning required
   PostgreSQL 11
   ● Self-managed instance
   ● Basic production tuning
   ● Tested w/ and w/o
   PgBouncer
   

   View Slide

9. MONGODB vs POSTGRESQL BENCHMARKS
   Architecture: client-server, running on AWS
   Data volume: io1, with number of reserved IOPS depending on the test.
   

   View Slide

10. MONGODB vs POSTGRESQL BENCHMARKS
    Benchmarks:
    Transactions
    

    View Slide

11. MONGODB vs POSTGRESQL BENCHMARKS
    Previous discussion: isolation levels
    

    View Slide

12. MONGODB vs POSTGRESQL BENCHMARKS
    Benchmark description
    ● Custom-developed benchmark.
    ● Inspired by MongoDB post about MongoDB transactions.
    ● Simulates air reservation system: check flight schedule, plane
    availability, purchase, audit log.
    ● Tx with query, insertions and upsert.
    ● Programmed in Java. Official JDBC and MongoDB drivers.
    ● Open source: https://gitlab.com/ongresinc/benchplatform
    

    View Slide

13. MONGODB vs POSTGRESQL BENCHMARKS
    Transaction performance
    

    View Slide

14. MONGODB vs POSTGRESQL BENCHMARKS
    Transaction performance
    

    View Slide

15. MONGODB vs POSTGRESQL BENCHMARKS
    Transaction retries
    

    View Slide

16. MONGODB vs POSTGRESQL BENCHMARKS
    PostgreSQL latency (@ SERIALIZABLE)
    

    View Slide

17. MONGODB vs POSTGRESQL BENCHMARKS
    MongoDB latency
    

    View Slide

18. MONGODB vs POSTGRESQL BENCHMARKS
    Benchmarks:
    OLTP
    

    View Slide

19. MONGODB vs POSTGRESQL BENCHMARKS
    Benchmark description
    ● Industry standard Sysbench was used.
    ● Supports PostgreSQL and MongoDB (via sysbench-mongodb-lua).
    ● Resembles real-world OLTP workload.
    ● Different dimensions benchmarked:
    ○ Dataset size: fit in memory (4GB), on-disk (2TB)
    ○ Read/write work split: 50/50, 95/5.
    ○ Filesystems: XFS or ZFS.
    ○ Different levels of concurrency.
    

    View Slide

20. MONGODB vs POSTGRESQL BENCHMARKS
    Discussion about PostgreSQL Connection Pooling
    ● PostgreSQL best practice is to run always behind a connection pool.
    ● PostgreSQL proved to be highly sensitive to the number of connections,
    with degraded performance when overwhelmed.
    ● A connection pool offers close-to-ideal performance for almost any
    workload.
    ● MongoDB does not require connection pooling:
    is included in drivers.
    

    View Slide

21. MONGODB vs POSTGRESQL BENCHMARKS
    Performance: dataset in memory
    

    View Slide

22. MONGODB vs POSTGRESQL BENCHMARKS
    PG connection pooling effect (dataset in memory)
    

    View Slide

23. MONGODB vs POSTGRESQL BENCHMARKS
    Performance: dataset on disk
    

    View Slide

24. MONGODB vs POSTGRESQL BENCHMARKS
    Benchmarks:
    OLAP
    

    View Slide

25. MONGODB vs POSTGRESQL BENCHMARKS
    Benchmark description
    ● JSON dataset (GitHub Archive). Native to MongoDB, jsonb in PG.
    ● 2015 dataset: 212M records (340GB on PG, 206GB on MongoDB).
    ● Timing of OLAP-style queries that resemble BI natural questions:
    ○ Repositories ordered by most opened issues.
    ○ Most frequent git event types, ordered.
    ○ Top 10 most active actors.
    ○ Repositories that have more than two comments and a specific
    event type, ordered.
    

    View Slide

26. MONGODB vs POSTGRESQL BENCHMARKS
    Query A in PostgreSQL and MongoDB
    

    View Slide

27. MONGODB vs POSTGRESQL BENCHMARKS
    Results
    

    View Slide

28. MONGODB vs POSTGRESQL BENCHMARKS
    QUESTIONS?
    Álvaro Hernández
    
    @ahachete / www.ongres.com
    Thank you
    

    View Slide