Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Apache Ignite

Apache Ignite

This presentation gives an overview of the Apache Ignite project. It explains Ignite in relation to its architecture, scaleability, caching, datagrid and machine learning abilities.

Links for further information and connecting

http://www.amazon.com/Michael-Frampton/e/B00NIQDOOM/

https://nz.linkedin.com/pub/mike-frampton/20/630/385

https://open-source-systems.blogspot.com/

Mike Frampton

May 27, 2020
Tweet

More Decks by Mike Frampton

Other Decks in Technology

Transcript

  1. What Is Apache Ignite ? • An in memory distributed

    database • A cluster based processing platform • Open source / Apache 2.0 license • A key / value store • Written in Java, C#, C++, SQL • Multiple API's available • Multiple deployment options • Developed by GridGain Systems
  2. What Is Apache Ignite ? • Persistence is turned off

    – A distributed in-memory database – In memory data grid – Use SQL or key-value APIs • Persistence is turned on – A distributed, horizontally scalable database – Guarantees full data consistency – Is resilient to full cluster failures
  3. How Does Ignite Work ? • Persistence can be toggled

    • Small data sets can be stored in memory • Large datasets can use disk • Memory can be used as a caching mechanism • RDBMS or NoSQL integration • Partitioning or replication of data across cluster • Horizontally scaleable
  4. How Does Ignite Work ? • Use Ignite as a

    distributed cache • Implements JCache specification (JSR 107) • Supports ACID transactions • Supports SQL ( with limitations ) • Supports Java, C++, and .NET • Has a feature rich key-value API • Offers two user interface ( UI ) options • Supports collocated processing
  5. How Does Ignite Work ? • Ignite has client and

    server nodes in it's cluster • Server nodes are for storage and computation • Client nodes are connection points to the database • They may be embedded in client applications • Application code written in Java, C# or C++ • Ignite supports ODBC, JDBC and REST
  6. Ignite DataGrid • An in-memory distributed key-value store • Horizontal

    scaleable • Add nodes on demand • Scale to hundreds of nodes • View as a distributed partitioned hash map • Every cluster node owns a portion of the data • As cluster (server) nodes are added more data is cached
  7. Ignite Machine Learning • Multiple development API's • A set

    of simple / efficient ML functions • Massively scaleable • Co-location of data and processing • Minimised the need for costly data transfers • Minimised ETL as ML functions act on distributed data • Algorithms support – Classification, Regression, Clustering – Recommendation, Preprocessing
  8. Ignite Deployment Options • Multiple deployment options – Docker –

    Amazon AWS – Google Cloud – Mesos – YARN – VMWare – Microsoft Azure – Kubernetes
  9. Available Books • See “Big Data Made Easy” – Apress

    Jan 2015 • See “Mastering Apache Spark” – Packt Oct 2015 • See “Complete Guide to Open Source Big Data Stack – “Apress Jan 2018” • Find the author on Amazon – www.amazon.com/Michael-Frampton/e/B00NIQDOOM/ • Connect on LinkedIn – www.linkedin.com/in/mike-frampton-38563020
  10. Connect • Feel free to connect on LinkedIn – www.linkedin.com/in/mike-frampton-38563020

    • See my open source blog at – open-source-systems.blogspot.com/ • I am always interested in – New technology – Opportunities – Technology based issues – Big data integration