Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Technologies and Platform Services

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
C Deepa
October 21, 2026
25

Cloud Technologies and Platform Services

Cloud is a collection of IT capabilities and services — software (SaaS), platform (PaaS), IT infrastructure (IaaS), virtualized security services, virtualized storage, and so on — whereas cloud technologies are the various runtime frameworks such as Apache Hadoop, Amazon S3, MapReduce, and so on. High-Performance Computing (HPC) with cloud technologies:**
- HPC libraries/frameworks used as runtime technologies in the cloud: Apache Hadoop, Amazon S3, MapReduce, MPI, Dryad/DryadLINQ.
- Message Passing Interface (MPI): an HPC library used to establish information flow between various nodes and clusters. It is the most commonly used communication protocol/standard for parallel programming on distributed-memory architectures. Open MPI is the most popular open-source implementation. MPI optimizes CPU utilization for tightly coupled problems — canonical examples: climate modelling, car crash simulation, computational fluid dynamics.
- MapReduce: a linear programming model consisting of two main tasks — Map and Reduce. The Mapper task executes one record at a time; the Reducer aggregates the results. Highly scalable for very large data sets; the foundational batch-processing model of Hadoop.
- Dryad: a distributed execution engine for coarse-grained data-parallel applications. It combines the MapReduce programming style with data-flow graphs (Directed Acyclic Graphs, DAGs) to solve computation tasks. In the DAG, vertices are computation tasks and edges are communication channels; it is optimized for data locality. A Dryad job is a graph of sequential programs connected by one-way channels.
- DryadLINQ: a language extension that allows developers to write data-parallel code using high-level .NET languages such as C# and F#, using LINQ; DryadLINQ compiles LINQ programs into Dryad jobs (distributed DAG computations). It combines the declarative power of LINQ with the scalability of Dryad on Windows HPC clusters.

**8.2 Eucalyptus cloud platform:**
- Definition: Eucalyptus is an open-source, Amazon Web Services (AWS)-compatible private and hybrid cloud computing environment. The acronym stands for "Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems." It is Linux-based, provides an EC2-compatible compute service and an S3-compatible storage service, and is a common choice for building private/hybrid clouds that must interoperate with AWS.
- Components (five) — this must be its own diagram slide:
1. Cloud Controller (CLC): the main controller; manages the entire platform; performs scheduling, resource allocation and accounting. Interfaces with AWS-compatible APIs (EC2, S3).
2. Walrus: similar to AWS S3; provides persistent storage to all the instances (object storage / bucket-based storage).
3. Cluster Controller (CC): manages VM (instance) execution and service level agreements (SLAs); communicates with the Storage Controller and network controllers; controls one or more Node Controllers.
4. Storage Controller (SC): similar to AWS EBS; provides block-level storage to instances (elastic block storage volumes).
5. Node Controller (NC): hosts all the instances and manages their end points; runs on the physical host that runs the hypervisor (Xen, KVM).
- Organise the components into the three tiers: Cloud layer (CLC + Walrus), Cluster layer (CC + SC), Node layer (NC).

**8.3 OpenNebula cloud platform:**
- Definition: OpenNebula is an open-source cloud service framework used to manage heterogeneous, distributed data center infrastructures. It is a complete solution for building private clouds and for managing data-center virtualization. It is often described as a "toolkit" for building and managing virtualized data centres / private and hybrid clouds.
- Why should we use OpenNebula? — it provides a turnkey, open-source solution for managing virtualized data centres with no vendor lock-in; it is scalable, interoperable and enterprise-ready.
- Layers of OpenNebula (three-layer internal architecture):
1. Tools layer — management tools built on the interfaces provided by the OpenNebula Core: the Command Line Interface (one) and the Sunstone GUI/Web interface.
2. Drivers layer — responsible for connecting to and communicating with the underlying hypervisors (Xen, KVM, VMware), storage and network subsystems, and for transferring images and monitoring the VMs (including the information manager and virtual machine manager drivers).
3. Core layer — the central component; contains the components used to carry out user requests and control the whole system (request manager, VM manager, host manager, virtual network manager, database).
- OpenNebula operates at two main management layers: (a) Data-center virtualization management — integration directly with hypervisors for capacity management and resource optimization; (b) Cloud management — a multi-tenant, cloud-like provisioning layer.
- Features (verbatim list from the book): Adaptable — assimilation abilities to integrate into any infrastructure; Enterprise-ready — upgrade process and commercial support; No lock-in — expansive infrastructure and platform independent; Light — efficient and intuitive; Proven — thoroughly tested, established and commonly used; Powerful — advanced features for virtualization; Scalable — single instance and multi-tier architectures; Be interoperable — extensive set of APIs and interfaces; Open source — Apache License v2.

**8.4 OpenStack cloud platform:**
- Definition: OpenStack is a free and open-source software platform for Infrastructure-as-a-Service (IaaS) clouds. It provides modular cloud infrastructure and acts as a cloud operating system controlling large pools of compute, storage and networking resources across a data center, all managed through APIs and a dashboard.
- OpenStack components (own slide, ideally a labelled diagram):
- Nova — implements services and associated libraries that provide massively scalable, on-demand, self-service access to compute resources (the compute engine; manages the lifecycle of virtual machine instances).
- Glance — a service that discovers, registers and retrieves virtual machine (VM) images (the image service).
- Keystone — the authentication and authorization component (the identity service; provides a central directory of users mapped to the OpenStack services they can access).
- Neutron — responsible for creating the virtual networks (networking-as-a-service; manages IP addresses, ports, subnets, routers, firewalls).
- Cinder — provides virtualized block storage as a service (persistent block storage volumes for instances).
- Zun — provides an API to launch and manage containers (container service).
- (Swift — object storage service; Horizon — the web-based dashboard for managing OpenStack services. Note in the deck that the book's main text emphasises the components above, and that Swift/Horizon are the object-storage and dashboard services of the same platform.)
- Benefits of OpenStack (from the book): Private clouds — ease of installation and service management; Public clouds — OpenStack is the leading open-source option; Network virtualization — OpenStack supports Network Function Virtualization (NFV); Containers — OpenStack containers speed up application delivery.

**8.5 Nimbus cloud computing platform:**
- Definition: Nimbus — formerly known as Virtual Workspace Service (VWS) — is an Infrastructure-as-a-Service (IaaS) cloud (an open-source toolkit) used to deploy and configure virtual machines on remote resources; it turns a computer cluster into an IaaS cloud, aimed particularly at scientific communities. Implemented largely in Python (and Java).
- Two basic products of Nimbus: (1) Nimbus Platform — a versatile, integrated set of tools that provides infrastructure services such as deployment, scaling and cloud resource management (supports launching a cloud from a single node); (2) Nimbus Infrastructure — an open-source, EC2/S3-compatible IaaS solution (the core toolkit / cloudkit set of tools including the workspace service, workspace control and workspace pilot).
- Features of Nimbus (from the book): open-source Infrastructure-as-a-Service solution; remote deployment and lifecycle management of VMs; compatibility with Amazon's network protocols (EC2/S3); support for multiple credential protocols (e.g. X.509 certificates and SSH keys); local resource management; XEN and KVM hypervisor plugins.

**8.6 The Apache Hadoop ecosystem:**
- Definition: Hadoop is an Apache-based distributed computing ecosystem used for analysing, storing and processing voluminous unstructured data. The architecture is based on the traditional master-slave style.
- Architecture of Hadoop: master-slave; master nodes run the NameNode (HDFS) and the Resource Manager (YARN), slave nodes run the DataNodes and Node Managers; HDFS provides distributed storage and MapReduce/YARN provides distributed processing; designed to run on commodity hardware with high fault tolerance through data replication (default replication factor 3).
- Major components of Hadoop (each needs its own slide):
- HDFS (Hadoop Distributed File System): the core component; provides reliable and distributed storage for huge datasets; consists of a single NameNode (master — stores metadata/file-system namespace) and multiple DataNodes (slaves — store the actual data blocks); blocks are replicated across nodes for fault tolerance.
- YARN (Yet Another Resource Negotiator): the brain of Hadoop; responsible for providing computational resources and managing workloads and job scheduling; consists of the Resource Manager (master) and Node Managers (slaves) plus the Application Master per application.
- MapReduce: the data processing layer for parallel processing; it breaks a job into sub-jobs and processes them in parallel using the Map and Reduce phases.
- Hive: an open-source data query system used to query or analyse large unstructured and semi-structured datasets; provides an SQL-like language (HiveQL/HQL) that translates queries into MapReduce jobs.
- Apache Pig: a high-level procedural language used for parallel processing of large datasets; uses the Pig Latin language and runs on top of Hadoop (executing as MapReduce jobs).
- HBase: an open-source, non-relational (NoSQL) column-oriented database that runs on top of Hadoop (on HDFS); provides real-time read/write random access to very large tables.
- Apache Mahout and Spark MLlib: both open-source frameworks used for creating scalable machine-learning algorithms and data-mining libraries (Mahout historically on MapReduce, MLlib on Apache Spark).
- Apache ZooKeeper: a centralized service used for maintaining configuration information, naming, and providing distributed synchronization and group services for distributed applications.
- (Apache Spark may be mentioned as the in-memory processing engine that MLlib sits on.)

**8.7 Hadoop and cloud:**
- Hadoop is the de-facto batch/big-data processing technology of the cloud era; cloud infrastructure provides the elastic compute and storage on which Hadoop clusters run. Hadoop is offered as a managed service by major cloud vendors (e.g. Amazon EMR, Microsoft Azure HDInsight, Google Cloud Dataproc, and Cloudera/Hortonworks-style distributions), letting organisations rent transient clusters instead of owning them. The synergy: cloud gives elasticity, pay-as-you-go economics, and separation of storage and compute; Hadoop gives a proven distributed storage + processing stack for unstructured data.

C Deepa

October 21, 2026

Transcript

40 slides
  1. CLOUD COMPUTING | TEACHING DECK Cloud Computing Technologies, Platforms and

    Services CHAPTER 8 Cloud Computing: Master the Concepts, Architecture and Applications with Real-world Examples and Case Studies Authors: Kamal Kant Hiran · Ruchi Doshi · Temitayo Fagbola · Mehul Mahrishi Publisher: BPB Publications · First Edition, 2019 · ISBN 9789388511407 · 330 pp. Chapter structure verified against the book's published table of contents and Chapter 8 text (dokumen.pub mirror of the BPB edition). Suggested use: 2 lectures × 90 minutes | Deck covers sections 8.1 – 8.7 plus comparison, summary and review. Chapter 8 | Cloud Computing Technologies, Platforms and Services 1
  2. COURSE MAP Where this chapter sits • 1. Foundation of

    Cloud Computing • 2. Cloud Services and Deployment Models 8. Cloud Computing Technologies, Platform and Services ◀ you are here • 3. Cloud Computing Architecture • 4. Virtualization Technology • 5. Service Oriented Architecture (SOA) • 9. Adoption of Cloud Computing • 6. Cloud Security and Privacy – Chapter 7 introduced the application side of the cloud (Google App Engine, Microsoft Windows Azure, Amazon Web Services). • 7. Cloud Computing Applications Chapter 8 | Cloud Computing Technologies, Platforms and Services – Chapter 8 moves to the technologies, platforms and services layer — the platforms on which those applications run. 2
  3. CHAPTER OBJECTIVES Learning objectives • Distinguish cloud services / capabilities

    from cloud technologies • Explain high-performance computing technologies in the cloud — MPI, MapReduce, Dryad and DryadLINQ • Describe the Eucalyptus cloud platform and its five components • Explain the OpenNebula platform: purpose, three layers and features • Identify OpenStack components and the benefits of OpenStack • Describe the Nimbus IaaS platform, its two products and features • Explain the Apache Hadoop ecosystem, its architecture and major components • Discuss how Hadoop is used together with cloud infrastructure Chapter 8 | Cloud Computing Technologies, Platforms and Services 3
  4. PREREQUISITES · RECAP ONLY Recap: what you should already know

    • Chapter 2 — service models: IaaS, PaaS and SaaS; deployment models: public, private, hybrid, community • Chapter 3 — cloud architecture: the layered reference stack and its components • Chapter 4 — virtualization: hypervisor, virtual machine, Type-1 vs Type-2, resource pooling • Chapter 7 — cloud applications: Google App Engine, Microsoft Windows Azure, Amazon Web Services (AWS) • Bridge to today: from models and applications → to platforms and runtime technologies – Every platform in this chapter is installed on top of the virtualization layer you studied in Chapter 4. – Every platform in this chapter offers an IaaS or private-cloud capability that you met conceptually in Chapter 2. Chapter 8 | Cloud Computing Technologies, Platforms and Services 4
  5. INSTRUCTOR ONLY Teaching plan for the instructor • Lecture 1

    (90 min): cloud vs cloud technologies · 8.1 HPC technologies · 8.2 Eucalyptus · 8.3 OpenNebula · 8.4 OpenStack • Lecture 2 (90 min): 8.5 Nimbus · 8.6 Hadoop ecosystem · 8.7 Hadoop and cloud · four-platform comparison · review • Activity (Lecture 1, 10 min): map each Eucalyptus component to the equivalent AWS service • Activity (Lecture 2, 15 min): draw the Hadoop master–slave architecture from memory, then label HDFS, YARN and MapReduce • Assessment: short-answer questions on components; one comparison question drawn from the platform table • Homework: install OpenStack or OpenNebula in a lab VM and submit five annotated screenshots Chapter 8 | Cloud Computing Technologies, Platforms and Services 5
  6. 8.1 High-Performance Computing with Cloud Technologies Message Passing Interface (MPI)

    · MapReduce programming model · Dryad and DryadLINQ Chapter 8 | Cloud Computing Technologies, Platforms and Services 6
  7. KEY DISTINCTION Cloud vs. cloud technologies • Cloud = a

    collection of IT capabilities and services – Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS) – Virtualized security services and virtualized storage • Cloud technologies = the various runtime frameworks that implement and run those services – Apache Hadoop, Amazon S3, MapReduce, MPI, Dryad / DryadLINQ • One-line test: cloud is what you consume; cloud technologies are what you build and run it with • This chapter follows the technology thread and then the platform thread Chapter 8 | Cloud Computing Technologies, Platforms and Services 7
  8. 8.1 · MOTIVATION High-performance computing meets the cloud • HPC:

    solving large, compute-intensive problems on clusters — traditionally owned and expensive • Cloud adds elastic, on-demand capacity: rent a cluster for the duration of a job, then release it • Runtime technologies discussed in this section: – MPI — message-passing library for tightly coupled parallel programs – MapReduce — two-phase batch programming model for very large data sets – Dryad and DryadLINQ — data-flow graphs and declarative data-parallel programming • These three approaches reappear inside the Hadoop ecosystem in section 8.6 Chapter 8 | Cloud Computing Technologies, Platforms and Services 8
  9. 8.1 · HPC TECHNOLOGY Message Passing Interface (MPI) • An

    HPC library used to establish information flow between various nodes and clusters • The standard communication protocol for parallel programming on distributed-memory architectures – Processes exchange explicit messages — point-to-point sends/receives and collective operations • Open MPI is the most popular and commonly used open-source implementation • Optimizes CPU utilization for tightly coupled problems • Canonical workloads: climate modelling, car crash simulation, computational fluid dynamics – Supplementary: MPI is a specification maintained by the MPI Forum, not a single product Chapter 8 | Cloud Computing Technologies, Platforms and Services 9
  10. 8.1 · HPC TECHNOLOGY MapReduce programming model • A linear

    programming model consisting of two main tasks: Map and Reduce – Mapper — executes one record at a time, emitting intermediate key–value pairs – Reducer — aggregates the results for each key • Data flow: input splits → Map → shuffle and sort by key → Reduce → output • Highly scalable for very large data sets; designed to run on commodity hardware • The foundational batch-processing model of Hadoop (developed in section 8.6) Chapter 8 | Cloud Computing Technologies, Platforms and Services 10
  11. 8.1 · HPC TECHNOLOGY Dryad and DryadLINQ • Dryad: a

    distributed execution engine for coarse-grained data-parallel applications • Combines the MapReduce programming style with data-flow graphs to solve computation tasks – In the directed acyclic graph (DAG): vertices = computation tasks, edges = one-way communication channels – A Dryad job is a graph of sequential programs connected by channels; optimized for data locality • DryadLINQ: write data-parallel code in high-level .NET languages such as C# and F# using LINQ • DryadLINQ compiles LINQ programs into Dryad jobs (distributed DAG computations) – Value: declarative LINQ productivity + Dryad scalability on Windows HPC clusters Chapter 8 | Cloud Computing Technologies, Platforms and Services 11
  12. 8.1 · COMPARISON Comparing the HPC technologies Technology Paradigm Data

    model Interface / language Optimized for Typical use case MPI Explicit message passing Messages between processes, distributed memory C / C++ / Fortran library API (Open MPI) Tightly coupled parallel computation Climate modelling, car-crash simulation, CFD MapReduce Two-phase functional batch Key–value records; one record at a time Job of Map and Reduce tasks; Java/other APIs Very large data sets that parallelize easily Log analysis, batch aggregation, word count Dryad Data-flow graph execution Directed acyclic graph of tasks over partitioned data .NET programs combined into a graph Coarse-grained data parallelism, data Multi-stage big-data pipelines locality and joins DryadLINQ Declarative data-parallel query LINQ sequences evaluated over a Dryad DAG C# / F# LINQ expressions Developer productivity plus Dryad scale Chapter 8 | Cloud Computing Technologies, Platforms and Services Analytics jobs on Windows HPC clusters 12
  13. 8.2 The Eucalyptus Cloud Platform An AWS-compatible open-source private and

    hybrid cloud · components of Eucalyptus Chapter 8 | Cloud Computing Technologies, Platforms and Services 13
  14. 8.2 · PLATFORM Eucalyptus: what it is • An open-source,

    Amazon Web Services (AWS)-compatible private and hybrid cloud computing environment • Acronym: Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems • Linux-based, with an EC2-compatible compute service and an S3-compatible storage service • Common choice when a private or hybrid cloud must interoperate with AWS • Organized in three tiers: Cloud layer, Cluster layer, Node layer – Why it matters: workloads and APIs can move between the private Eucalyptus cloud and public AWS Chapter 8 | Cloud Computing Technologies, Platforms and Services 14
  15. 8.2 · ARCHITECTURE Components of Eucalyptus • Cloud Controller (CLC)

    — the main controller; manages the entire platform – Performs scheduling, resource allocation and accounting; exposes the AWS-compatible APIs • Walrus — similar to AWS S3; provides persistent storage to all the instances • Cluster Controller (CC) — manages VM (instance) execution and service level agreements (SLAs) – Controls one or more Node Controllers and communicates with the storage and network controllers • Storage Controller (SC) — similar to AWS EBS; provides block-level storage to instances • Node Controller (NC) — hosts all the instances and manages their end points (runs the hypervisor) Chapter 8 | Cloud Computing Technologies, Platforms and Services 15
  16. 8.2 · LABELLED DIAGRAM Eucalyptus three-tier architecture Cloud layer: CLC

    + Walrus | Cluster layer: CC + SC | Node layer: NC + hypervisor Cloud layer Cluster layer Node layer Cloud Controller (CLC) Walrus scheduling · resource allocation · accounting object storage — equivalent of AWS S3 Cluster Controller (CC) Storage Controller (SC) VM execution · SLAs · manages Node Controllers block-level storage — equivalent of AWS EBS Node Controller (NC) Hypervisor (Xen / KVM) + instances hosts instances · manages instance end points the physical host that actually runs the VMs Each tier talks only to the tier below or above; the CLC presents the AWS-compatible API to the outside world, while the nodes do the actual virtualization work. Chapter 8 | Cloud Computing Technologies, Platforms and Services 16
  17. 8.3 The OpenNebula Cloud Platform Why use OpenNebula · layers

    of OpenNebula · features of OpenNebula Chapter 8 | Cloud Computing Technologies, Platforms and Services 17
  18. 8.3 · PLATFORM OpenNebula: what it is • An open-source

    cloud service framework used to manage heterogeneous, distributed data centre infrastructures • A complete solution for building private clouds and managing data-centre virtualization • Operates at two main management layers: – Data-centre virtualization management — integrates directly with hypervisors for capacity management and resource optimization – Cloud management — a multi-tenant, cloud-like provisioning layer on top of that virtualized infrastructure • Why should we use OpenNebula? – Turnkey open-source management of a virtualized data centre with no vendor lock-in – Scalable, interoperable and enterprise-ready, with a documented upgrade path and commercial support Chapter 8 | Cloud Computing Technologies, Platforms and Services 18
  19. 8.3 · ARCHITECTURE Layers of OpenNebula Three internal layers: Tools,

    Drivers and Core. Tools and Drivers both work through the Core. Management tools built on the interfaces provided by the OpenNebula Core TOOLS LAYER Command Line Interface (one) · Sunstone graphical / web interface DRIVERS LAYER hypervisor, storage and network drivers: Xen, KVM, VMware, image transfer, VM monitoring CORE LAYER carries out user requests and controls the system: request manager, VM manager, host manager, virtual network manager, database Requests flow down from the tools to the core and out to the drivers; status information flows back up. The Core is the only component that talks to the database, which keeps tool upgrades independent of the platform internals. Chapter 8 | Cloud Computing Technologies, Platforms and Services 19
  20. 8.3 · FEATURES Features of OpenNebula • Adaptable — assimilation

    abilities to integrate into any infrastructure • Enterprise-ready — upgrade process and commercial support • No lock-in — expansive infrastructure and platform independent • Light — efficient and intuitive • Proven — thoroughly tested, established and commonly used • Powerful — advanced features for virtualization • Scalable — single instance and multi-tier architectures • Interoperable — extensive set of APIs and interfaces • Open source — released under the Apache License v2 Chapter 8 | Cloud Computing Technologies, Platforms and Services 20
  21. 8.4 The OpenStack Cloud Platform OpenStack components · benefits of

    OpenStack Chapter 8 | Cloud Computing Technologies, Platforms and Services 21
  22. 8.4 · PLATFORM OpenStack: what it is • A free

    and open-source software platform for Infrastructure-as-a-Service (IaaS) clouds • Acts as a cloud operating system: controls large pools of compute, storage and networking resources • Provides modular cloud infrastructure — every capability is a separate service with its own API • A web dashboard plus APIs are used to provision and manage resources • Supports both private cloud deployments and large public cloud deployments – Supplementary: the project is governed by the Open Infrastructure Foundation and releases on a fixed cadence Chapter 8 | Cloud Computing Technologies, Platforms and Services 22
  23. 8.4 · COMPONENTS OpenStack components • Nova — implements services

    and libraries that give massively scalable, on-demand, self-service access to compute resources • Glance — discovers, registers and retrieves virtual machine (VM) images • Keystone — the authentication and authorization component (identity service) • Neutron — responsible for creating the virtual networks • Cinder — provides virtualized block storage as a service • Zun — provides an API to launch and manage containers – Supplementary: Swift (object storage) and Horizon (web dashboard) are further services of the same platform Chapter 8 | Cloud Computing Technologies, Platforms and Services 23
  24. 8.4 · BENEFITS Benefits of OpenStack • Private clouds —

    ease of installation and service management • Public clouds — OpenStack is the leading open-source option • Network virtualization — supports Network Function Virtualization (NFV) • Containers — OpenStack containers speed up application delivery • Modular design — adopt only the services your deployment needs – Consequence for architects: the same platform can serve an internal private cloud and a commercial public cloud Chapter 8 | Cloud Computing Technologies, Platforms and Services 24
  25. 8.5 The Nimbus Cloud Computing Platform Formerly the Virtual Workspace

    Service (VWS) · products of Nimbus · features of Nimbus Chapter 8 | Cloud Computing Technologies, Platforms and Services 25
  26. 8.5 · PLATFORM Nimbus: what it is • Formerly known

    as the Virtual Workspace Service (VWS) • An Infrastructure-as-a-Service (IaaS) cloud used to deploy and configure virtual machines on remote resources • An open-source toolkit that turns a computer cluster into an IaaS cloud • Two basic products: – Nimbus Platform — a versatile integrated set of tools providing infrastructure services such as deployment, scaling and cloud resource management – Nimbus Infrastructure — an open-source EC2/S3-compatible IaaS solution (the core toolkit, including workspace service, control and pilot tools) • Aimed particularly at scientific communities that need on-demand workspaces Chapter 8 | Cloud Computing Technologies, Platforms and Services 26
  27. 8.5 · FEATURES Features of Nimbus • Open-source Infrastructure-as-a-Service solution

    • Remote deployment and lifecycle management of virtual machines • Compatibility with Amazon's network protocols (EC2 and S3) • Support for multiple credential protocols, such as X.509 certificates and SSH keys • Local resource management on the cluster that hosts the cloud • Hypervisor plug-ins for XEN and KVM – Positioning: lightweight, cluster-scale IaaS for research and teaching Chapter 8 | Cloud Computing Technologies, Platforms and Services 27
  28. 8.6 The Apache Hadoop Ecosystem Architecture of Hadoop · major

    components of Hadoop · HDFS, YARN, MapReduce, Hive, Pig, HBase, Mahout, MLlib, ZooKeeper Chapter 8 | Cloud Computing Technologies, Platforms and Services 28
  29. 8.6 · PLATFORM The Apache Hadoop ecosystem: what it is

    and how it is built An Apache-based distributed computing ecosystem used for analysing, storing and processing voluminous unstructured data Master–slave architecture MASTER NODE SLAVE NODE 1 SLAVE NODE 2 SLAVE NODE 3 NameNode (HDFS metadata) + Resource Manager (YARN) DataNode (data blocks) + Node Manager DataNode (data blocks) + Node Manager DataNode (data blocks) + Node Manager HDFS provides distributed storage; YARN and MapReduce provide distributed processing. The design assumes commodity hardware, so fault tolerance is achieved by replicating data blocks across nodes (default replication factor 3). STORAGE SIDE PROCESSING SIDE NameNode holds the file-system namespace and metadata; DataNodes hold the actual blocks and serve reads/writes Resource Manager allocates cluster resources; Node Managers run tasks; Application Master manages one application Chapter 8 | Cloud Computing Technologies, Platforms and Services 29
  30. 8.6 · COMPONENTS Major components of Hadoop (1): HDFS, YARN,

    MapReduce • HDFS (Hadoop Distributed File System) — the core component; provides reliable and distributed storage for huge datasets – A single NameNode (master) stores metadata and the file-system namespace; multiple DataNodes (slaves) store the data blocks • YARN (Yet Another Resource Negotiator) — the brain of Hadoop; provides computational resources and manages workloads and job scheduling – Resource Manager (master) plus Node Managers (slaves) and one Application Master per application • MapReduce — the data processing layer for parallel processing – Breaks a job into sub-jobs and processes them in parallel using the Map and Reduce phases Chapter 8 | Cloud Computing Technologies, Platforms and Services 30
  31. 8.6 · COMPONENTS Major components of Hadoop (2): Hive, Pig,

    HBase • Hive — an open-source data query system used to query or analyse large unstructured and semi-structured datasets – Provides an SQL-like language (HiveQL / HQL) which is translated into MapReduce jobs • Apache Pig — a high-level procedural language used for parallel processing of large datasets – Uses the Pig Latin language and runs on top of Hadoop, executing as MapReduce jobs • HBase — an open-source, non-relational (NoSQL) database that runs on top of Hadoop (on HDFS) – Column-oriented, providing real-time read/write random access to very large tables Chapter 8 | Cloud Computing Technologies, Platforms and Services 31
  32. 8.6 · COMPONENTS Major components of Hadoop (3): machine learning

    and coordination • Apache Mahout and Spark MLlib — both open-source frameworks used for creating scalable machine-learning algorithms and data-mining libraries – Mahout historically executes on MapReduce; MLlib runs on Apache Spark • Apache ZooKeeper — a centralized service used for maintaining configuration information, naming, and providing distributed synchronization – Supports group services that let distributed processes coordinate safely • Supplementary: Apache Spark is the in-memory processing engine on which MLlib depends • Recurring pattern: add capability around a reliable distributed core instead of replacing it Chapter 8 | Cloud Computing Technologies, Platforms and Services 32
  33. 8.6 · DIAGRAMS MapReduce and the Hadoop ecosystem at a

    glance Left: the Map → Shuffle → Reduce data flow. Right: components grouped by role. Input splits Map (key, value) Shuffle & sort by key Reduce (aggregate) Output Mappers work record by record in parallel; the shuffle guarantees that all values for a key reach the same reducer. Hadoop components by role Storage HDFS · HBase Resource & coordination YARN · ZooKeeper Processing MapReduce · Spark Teaching tip: have students reproduce this grouping from memory — storage, resource/coordination, processing, query/language, machine learning. It converts a long component list into five buckets and is far easier to recall under exam pressure. Cross-check question: which bucket does YARN belong to, and why is it not part of MapReduce? (YARN allocates resources for any application, not only MapReduce jobs.) Query & language Hive · Pig Machine learning Mahout · Spark MLlib Chapter 8 | Cloud Computing Technologies, Platforms and Services 33
  34. 8.7 Hadoop and Cloud Why the Hadoop ecosystem is the

    natural big-data workload of the cloud era Chapter 8 | Cloud Computing Technologies, Platforms and Services 34
  35. 8.7 · SYNTHESIS Hadoop and cloud • Hadoop is the

    de-facto batch and big-data processing technology of the cloud era • Cloud infrastructure supplies the elastic compute and storage on which Hadoop clusters run • Cloud contribution: elasticity, pay-as-you-go economics, separation of storage and compute • Hadoop contribution: a proven distributed storage and processing stack for unstructured data • Clusters can be transient — created for a job, then released, instead of being owned permanently • Practical consequence: the same platform skills transfer between on-premises and cloud deployments Chapter 8 | Cloud Computing Technologies, Platforms and Services 35
  36. SUPPLEMENTARY Industry context (beyond the book): managed Hadoop services •

    Amazon EMR — managed Hadoop and Spark clusters on AWS • Microsoft Azure HDInsight — managed Hadoop, Hive, HBase and Spark on Azure • Google Cloud Dataproc — managed Hadoop and Spark on Google Cloud • Vendor distributions (for example Cloudera) package the same open-source components with enterprise tooling • Common pattern: object storage holds the data, the managed cluster is created per job and released – Supplementary material for discussion — not drawn from the textbook text; always cite the vendor's own documentation Chapter 8 | Cloud Computing Technologies, Platforms and Services 36
  37. SYNTHESIS · SECTIONS 8.2–8.5 Comparing the four open-source cloud platforms

    Platform Type / category Model & licence AWS compatible Key components Hypervisor support Distinguishing strength Eucalyptus Private and hybrid cloud environment (IaaS APIs) Open source; acronym Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems Yes — EC2-compatible compute; S3-compatible object storage (Walrus) CLC, Walrus, Cluster Controller, Storage Controller, Node Controller Xen and KVM (Linux-based platform) Private cloud that mirrors AWS APIs, so workloads can move between private an public OpenNebula Cloud service framework to manage heterogeneous distributed Open source under Apache License v2 data centres Not its defining feature; interoperability via APIs Tools layer (CLI, Sunstone), Drivers layer, Core layer Xen, KVM and VMware through drivers No vendor lock-in; manages mixed existi infrastructure OpenStack Free and open-source IaaS cloud platform (cloud operating system) Open source; modular services Not its defining feature; open APIs Nova, Glance, Keystone, Neutron, Cinder, Zun Hypervisor-agnostic; widely deployed with KVM Modular IaaS: leading open-source optio public and private clouds, NFV and conta Nimbus Infrastructure-as-a-Service cloud (formerly Virtual Workspace Service) Open source toolkit Yes — EC2/S3-compatible infrastructure product Nimbus Platform (management tools); Nimbus Infrastructure (core toolkit) XEN and KVM plug-ins Lightweight cluster-to-cloud IaaS aimed scientific communities Chapter 8 | Cloud Computing Technologies, Platforms and Services 37
  38. WRAP-UP Summary and key takeaways • Cloud services are what

    you consume; cloud technologies are the runtime frameworks — Hadoop, S3-style storage, MapReduce, MPI, Dryad • MPI suits tightly coupled parallel jobs; MapReduce suits very large key-value batches; Dryad/DryadLINQ generalizes the model to data-flow graphs • Eucalyptus is an AWS-compatible private/hybrid cloud built from CLC, Walrus, Cluster Controller, Storage Controller and Node Controller • OpenNebula manages heterogeneous data centres through Tools, Drivers and Core layers, with no vendor lock-in • OpenStack is the modular open-source IaaS platform — Nova, Glance, Keystone, Neutron, Cinder, Zun • Nimbus converts a cluster into an IaaS cloud for scientific users; formerly the Virtual Workspace Service • Hadoop is a master–slave ecosystem: HDFS stores, YARN schedules, MapReduce computes, with Hive, Pig, HBase, Mahout/MLlib and ZooKeeper around them • Cloud supplies elasticity and pay-as-you-go economics; Hadoop supplies the proven distributed big-data stack Chapter 8 | Cloud Computing Technologies, Platforms and Services 38
  39. ASSESSMENT Review and self-assessment questions • 1. Distinguish between cloud

    services and cloud technologies, with two examples of each. • 2. What is MPI and for which class of problems is it optimized? • 3. Explain the roles of the Mapper and the Reducer in the MapReduce model. • 4. What is a Dryad job, and what does DryadLINQ add to it? • 5. Name the five components of Eucalyptus and state the AWS service each most closely resembles. • 6. Describe the three layers of OpenNebula and the responsibility of each. • 7. List any five OpenStack components and state the function of each. • 8. What are the two basic products of Nimbus, and how do they differ? • 9. Compare Eucalyptus and OpenStack as choices for a private cloud deployment. • 10. Explain how HDFS, YARN and MapReduce divide the work in a Hadoop cluster. Chapter 8 | Cloud Computing Technologies, Platforms and Services 39
  40. SOURCES Further reading and references • Primary textbook: Hiran K.

    K., Doshi R., Fagbola T., Mahrishi M., Cloud Computing: Master the Concepts, Architecture and Applications with Realworld Examples and Case Studies, BPB Publications, 1st Edition, 2019, ISBN 9789388511407 — Chapter 8, pp. covering sections 8.1–8.7 • Chapter 8 structure and text verified against the published table of contents and chapter content: dokumen.pub/book/cloud-computing-masterthe-concepts-architecture-and-applications-with-real-world-examples-and-case-studies-9388511409-9789388511407.html • Eucalyptus — project and administrator documentation: docs.eucalyptus.cloud • OpenNebula — concept and feature documentation: docs.opennebula.io • OpenStack — component and project navigation: openstack.org/software/project-navigator/openstack-components • Nimbus — toolkit documentation and scientific-cloud papers (Keahey et al., Cloud Computing with Nimbus) • Apache Hadoop — official project documentation: hadoop.apache.org • MPI — standard maintained by the MPI Forum: mpi-forum.org Chapter 8 | Cloud Computing Technologies, Platforms and Services 40