https://speakerdeck.com/drcd 2026-10-05 13:33:37 -0400 Cloud is a collection of IT capabilities and services — software (SaaS), platform (PaaS), IT infrastructure (IaaS), virtualized security services, virtualized storage, and so on — whereas cloud technologies are the various runtime frameworks such as Apache Hadoop, Amazon S3, MapReduce, and so on. High-Performance Computing (HPC) with cloud technologies:** - HPC libraries/frameworks used as runtime technologies in the cloud: Apache Hadoop, Amazon S3, MapReduce, MPI, Dryad/DryadLINQ. - Message Passing Interface (MPI): an HPC library used to establish information flow between various nodes and clusters. It is the most commonly used communication protocol/standard for parallel programming on distributed-memory architectures. Open MPI is the most popular open-source implementation. MPI optimizes CPU utilization for tightly coupled problems — canonical examples: climate modelling, car crash simulation, computational fluid dynamics. - MapReduce: a linear programming model consisting of two main tasks — Map and Reduce. The Mapper task executes one record at a time; the Reducer aggregates the results. Highly scalable for very large data sets; the foundational batch-processing model of Hadoop. - Dryad: a distributed execution engine for coarse-grained data-parallel applications. It combines the MapReduce programming style with data-flow graphs (Directed Acyclic Graphs, DAGs) to solve computation tasks. In the DAG, vertices are computation tasks and edges are communication channels; it is optimized for data locality. A Dryad job is a graph of sequential programs connected by one-way channels. - DryadLINQ: a language extension that allows developers to write data-parallel code using high-level .NET languages such as C# and F#, using LINQ; DryadLINQ compiles LINQ programs into Dryad jobs (distributed DAG computations). It combines the declarative power of LINQ with the scalability of Dryad on Windows HPC clusters. **8.2 Eucalyptus cloud platform:** - Definition: Eucalyptus is an open-source, Amazon Web Services (AWS)-compatible private and hybrid cloud computing environment. The acronym stands for "Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems." It is Linux-based, provides an EC2-compatible compute service and an S3-compatible storage service, and is a common choice for building private/hybrid clouds that must interoperate with AWS. - Components (five) — this must be its own diagram slide: 1. Cloud Controller (CLC): the main controller; manages the entire platform; performs scheduling, resource allocation and accounting. Interfaces with AWS-compatible APIs (EC2, S3). 2. Walrus: similar to AWS S3; provides persistent storage to all the instances (object storage / bucket-based storage). 3. Cluster Controller (CC): manages VM (instance) execution and service level agreements (SLAs); communicates with the Storage Controller and network controllers; controls one or more Node Controllers. 4. Storage Controller (SC): similar to AWS EBS; provides block-level storage to instances (elastic block storage volumes). 5. Node Controller (NC): hosts all the instances and manages their end points; runs on the physical host that runs the hypervisor (Xen, KVM). - Organise the components into the three tiers: Cloud layer (CLC + Walrus), Cluster layer (CC + SC), Node layer (NC). **8.3 OpenNebula cloud platform:** - Definition: OpenNebula is an open-source cloud service framework used to manage heterogeneous, distributed data center infrastructures. It is a complete solution for building private clouds and for managing data-center virtualization. It is often described as a "toolkit" for building and managing virtualized data centres / private and hybrid clouds. - Why should we use OpenNebula? — it provides a turnkey, open-source solution for managing virtualized data centres with no vendor lock-in; it is scalable, interoperable and enterprise-ready. - Layers of OpenNebula (three-layer internal architecture): 1. Tools layer — management tools built on the interfaces provided by the OpenNebula Core: the Command Line Interface (one) and the Sunstone GUI/Web interface. 2. Drivers layer — responsible for connecting to and communicating with the underlying hypervisors (Xen, KVM, VMware), storage and network subsystems, and for transferring images and monitoring the VMs (including the information manager and virtual machine manager drivers). 3. Core layer — the central component; contains the components used to carry out user requests and control the whole system (request manager, VM manager, host manager, virtual network manager, database). - OpenNebula operates at two main management layers: (a) Data-center virtualization management — integration directly with hypervisors for capacity management and resource optimization; (b) Cloud management — a multi-tenant, cloud-like provisioning layer. - Features (verbatim list from the book): Adaptable — assimilation abilities to integrate into any infrastructure; Enterprise-ready — upgrade process and commercial support; No lock-in — expansive infrastructure and platform independent; Light — efficient and intuitive; Proven — thoroughly tested, established and commonly used; Powerful — advanced features for virtualization; Scalable — single instance and multi-tier architectures; Be interoperable — extensive set of APIs and interfaces; Open source — Apache License v2. **8.4 OpenStack cloud platform:** - Definition: OpenStack is a free and open-source software platform for Infrastructure-as-a-Service (IaaS) clouds. It provides modular cloud infrastructure and acts as a cloud operating system controlling large pools of compute, storage and networking resources across a data center, all managed through APIs and a dashboard. - OpenStack components (own slide, ideally a labelled diagram): - Nova — implements services and associated libraries that provide massively scalable, on-demand, self-service access to compute resources (the compute engine; manages the lifecycle of virtual machine instances). - Glance — a service that discovers, registers and retrieves virtual machine (VM) images (the image service). - Keystone — the authentication and authorization component (the identity service; provides a central directory of users mapped to the OpenStack services they can access). - Neutron — responsible for creating the virtual networks (networking-as-a-service; manages IP addresses, ports, subnets, routers, firewalls). - Cinder — provides virtualized block storage as a service (persistent block storage volumes for instances). - Zun — provides an API to launch and manage containers (container service). - (Swift — object storage service; Horizon — the web-based dashboard for managing OpenStack services. Note in the deck that the book's main text emphasises the components above, and that Swift/Horizon are the object-storage and dashboard services of the same platform.) - Benefits of OpenStack (from the book): Private clouds — ease of installation and service management; Public clouds — OpenStack is the leading open-source option; Network virtualization — OpenStack supports Network Function Virtualization (NFV); Containers — OpenStack containers speed up application delivery. **8.5 Nimbus cloud computing platform:** - Definition: Nimbus — formerly known as Virtual Workspace Service (VWS) — is an Infrastructure-as-a-Service (IaaS) cloud (an open-source toolkit) used to deploy and configure virtual machines on remote resources; it turns a computer cluster into an IaaS cloud, aimed particularly at scientific communities. Implemented largely in Python (and Java). - Two basic products of Nimbus: (1) Nimbus Platform — a versatile, integrated set of tools that provides infrastructure services such as deployment, scaling and cloud resource management (supports launching a cloud from a single node); (2) Nimbus Infrastructure — an open-source, EC2/S3-compatible IaaS solution (the core toolkit / cloudkit set of tools including the workspace service, workspace control and workspace pilot). - Features of Nimbus (from the book): open-source Infrastructure-as-a-Service solution; remote deployment and lifecycle management of VMs; compatibility with Amazon's network protocols (EC2/S3); support for multiple credential protocols (e.g. X.509 certificates and SSH keys); local resource management; XEN and KVM hypervisor plugins. **8.6 The Apache Hadoop ecosystem:** - Definition: Hadoop is an Apache-based distributed computing ecosystem used for analysing, storing and processing voluminous unstructured data. The architecture is based on the traditional master-slave style. - Architecture of Hadoop: master-slave; master nodes run the NameNode (HDFS) and the Resource Manager (YARN), slave nodes run the DataNodes and Node Managers; HDFS provides distributed storage and MapReduce/YARN provides distributed processing; designed to run on commodity hardware with high fault tolerance through data replication (default replication factor 3). - Major components of Hadoop (each needs its own slide): - HDFS (Hadoop Distributed File System): the core component; provides reliable and distributed storage for huge datasets; consists of a single NameNode (master — stores metadata/file-system namespace) and multiple DataNodes (slaves — store the actual data blocks); blocks are replicated across nodes for fault tolerance. - YARN (Yet Another Resource Negotiator): the brain of Hadoop; responsible for providing computational resources and managing workloads and job scheduling; consists of the Resource Manager (master) and Node Managers (slaves) plus the Application Master per application. - MapReduce: the data processing layer for parallel processing; it breaks a job into sub-jobs and processes them in parallel using the Map and Reduce phases. - Hive: an open-source data query system used to query or analyse large unstructured and semi-structured datasets; provides an SQL-like language (HiveQL/HQL) that translates queries into MapReduce jobs. - Apache Pig: a high-level procedural language used for parallel processing of large datasets; uses the Pig Latin language and runs on top of Hadoop (executing as MapReduce jobs). - HBase: an open-source, non-relational (NoSQL) column-oriented database that runs on top of Hadoop (on HDFS); provides real-time read/write random access to very large tables. - Apache Mahout and Spark MLlib: both open-source frameworks used for creating scalable machine-learning algorithms and data-mining libraries (Mahout historically on MapReduce, MLlib on Apache Spark). - Apache ZooKeeper: a centralized service used for maintaining configuration information, naming, and providing distributed synchronization and group services for distributed applications. - (Apache Spark may be mentioned as the in-memory processing engine that MLlib sits on.) **8.7 Hadoop and cloud:** - Hadoop is the de-facto batch/big-data processing technology of the cloud era; cloud infrastructure provides the elastic compute and storage on which Hadoop clusters run. Hadoop is offered as a managed service by major cloud vendors (e.g. Amazon EMR, Microsoft Azure HDInsight, Google Cloud Dataproc, and Cloudera/Hortonworks-style distributions), letting organisations rent transient clusters instead of owning them. The synergy: cloud gives elasticity, pay-as-you-go economics, and separation of storage and compute; Hadoop gives a proven distributed storage + processing stack for unstructured data. Cloud is a collection of IT capabilities and services — software (SaaS), platform (PaaS), IT infrastructure (IaaS), virtualized security services, virtualized storage, and so on — whereas cloud technologies are the various runtime frameworks such as Apache Hadoop, Amazon S3, MapReduce, and so on. High-Performance Computing (HPC) with cloud technologies:** - HPC libraries/frameworks used as runtime technologies in the cloud: Apache Hadoop, Amazon S3, MapReduce, MPI, Dryad/DryadLINQ. - Message Passing Interface (MPI): an HPC library used to establish information flow between various nodes and clusters. It is the most commonly used communication protocol/standard for parallel programming on distributed-memory architectures. Open MPI is the most popular open-source implementation. MPI optimizes CPU utilization for tightly coupled problems — canonical examples: climate modelling, car crash simulation, computational fluid dynamics. - MapReduce: a linear programming model consisting of two main tasks — Map and Reduce. The Mapper task executes one record at a time; the Reducer aggregates the results. Highly scalable for very large data sets; the foundational batch-processing model of Hadoop. - Dryad: a distributed execution engine for coarse-grained data-parallel applications. It combines the MapReduce programming style with data-flow graphs (Directed Acyclic Graphs, DAGs) to solve computation tasks. In the DAG, vertices are computation tasks and edges are communication channels; it is optimized for data locality. A Dryad job is a graph of sequential programs connected by one-way channels. - DryadLINQ: a language extension that allows developers to write data-parallel code using high-level .NET languages such as C# and F#, using LINQ; DryadLINQ compiles LINQ programs into Dryad jobs (distributed DAG computations). It combines the declarative power of LINQ with the scalability of Dryad on Windows HPC clusters. **8.2 Eucalyptus cloud platform:** - Definition: Eucalyptus is an open-source, Amazon Web Services (AWS)-compatible private and hybrid cloud computing environment. The acronym stands for "Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems." It is Linux-based, provides an EC2-compatible compute service and an S3-compatible storage service, and is a common choice for building private/hybrid clouds that must interoperate with AWS. - Components (five) — this must be its own diagram slide: 1. Cloud Controller (CLC): the main controller; manages the entire platform; performs scheduling, resource allocation and accounting. Interfaces with AWS-compatible APIs (EC2, S3). 2. Walrus: similar to AWS S3; provides persistent storage to all the instances (object storage / bucket-based storage). 3. Cluster Controller (CC): manages VM (instance) execution and service level agreements (SLAs); communicates with the Storage Controller and network controllers; controls one or more Node Controllers. 4. Storage Controller (SC): similar to AWS EBS; provides block-level storage to instances (elastic block storage volumes). 5. Node Controller (NC): hosts all the instances and manages their end points; runs on the physical host that runs the hypervisor (Xen, KVM). - Organise the components into the three tiers: Cloud layer (CLC + Walrus), Cluster layer (CC + SC), Node layer (NC). **8.3 OpenNebula cloud platform:** - Definition: OpenNebula is an open-source cloud service framework used to manage heterogeneous, distributed data center infrastructures. It is a complete solution for building private clouds and for managing data-center virtualization. It is often described as a "toolkit" for building and managing virtualized data centres / private and hybrid clouds. - Why should we use OpenNebula? — it provides a turnkey, open-source solution for managing virtualized data centres with no vendor lock-in; it is scalable, interoperable and enterprise-ready. - Layers of OpenNebula (three-layer internal architecture): 1. Tools layer — management tools built on the interfaces provided by the OpenNebula Core: the Command Line Interface (one) and the Sunstone GUI/Web interface. 2. Drivers layer — responsible for connecting to and communicating with the underlying hypervisors (Xen, KVM, VMware), storage and network subsystems, and for transferring images and monitoring the VMs (including the information manager and virtual machine manager drivers). 3. Core layer — the central component; contains the components used to carry out user requests and control the whole system (request manager, VM manager, host manager, virtual network manager, database). - OpenNebula operates at two main management layers: (a) Data-center virtualization management — integration directly with hypervisors for capacity management and resource optimization; (b) Cloud management — a multi-tenant, cloud-like provisioning layer. - Features (verbatim list from the book): Adaptable — assimilation abilities to integrate into any infrastructure; Enterprise-ready — upgrade process and commercial support; No lock-in — expansive infrastructure and platform independent; Light — efficient and intuitive; Proven — thoroughly tested, established and commonly used; Powerful — advanced features for virtualization; Scalable — single instance and multi-tier architectures; Be interoperable — extensive set of APIs and interfaces; Open source — Apache License v2. **8.4 OpenStack cloud platform:** - Definition: OpenStack is a free and open-source software platform for Infrastructure-as-a-Service (IaaS) clouds. It provides modular cloud infrastructure and acts as a cloud operating system controlling large pools of compute, storage and networking resources across a data center, all managed through APIs and a dashboard. - OpenStack components (own slide, ideally a labelled diagram): - Nova — implements services and associated libraries that provide massively scalable, on-demand, self-service access to compute resources (the compute engine; manages the lifecycle of virtual machine instances). - Glance — a service that discovers, registers and retrieves virtual machine (VM) images (the image service). - Keystone — the authentication and authorization component (the identity service; provides a central directory of users mapped to the OpenStack services they can access). - Neutron — responsible for creating the virtual networks (networking-as-a-service; manages IP addresses, ports, subnets, routers, firewalls). - Cinder — provides virtualized block storage as a service (persistent block storage volumes for instances). - Zun — provides an API to launch and manage containers (container service). - (Swift — object storage service; Horizon — the web-based dashboard for managing OpenStack services. Note in the deck that the book's main text emphasises the components above, and that Swift/Horizon are the object-storage and dashboard services of the same platform.) - Benefits of OpenStack (from the book): Private clouds — ease of installation and service management; Public clouds — OpenStack is the leading open-source option; Network virtualization — OpenStack supports Network Function Virtualization (NFV); Containers — OpenStack containers speed up application delivery. **8.5 Nimbus cloud computing platform:** - Definition: Nimbus — formerly known as Virtual Workspace Service (VWS) — is an Infrastructure-as-a-Service (IaaS) cloud (an open-source toolkit) used to deploy and configure virtual machines on remote resources; it turns a computer cluster into an IaaS cloud, aimed particularly at scientific communities. Implemented largely in Python (and Java). - Two basic products of Nimbus: (1) Nimbus Platform — a versatile, integrated set of tools that provides infrastructure services such as deployment, scaling and cloud resource management (supports launching a cloud from a single node); (2) Nimbus Infrastructure — an open-source, EC2/S3-compatible IaaS solution (the core toolkit / cloudkit set of tools including the workspace service, workspace control and workspace pilot). - Features of Nimbus (from the book): open-source Infrastructure-as-a-Service solution; remote deployment and lifecycle management of VMs; compatibility with Amazon's network protocols (EC2/S3); support for multiple credential protocols (e.g. X.509 certificates and SSH keys); local resource management; XEN and KVM hypervisor plugins. **8.6 The Apache Hadoop ecosystem:** - Definition: Hadoop is an Apache-based distributed computing ecosystem used for analysing, storing and processing voluminous unstructured data. The architecture is based on the traditional master-slave style. - Architecture of Hadoop: master-slave; master nodes run the NameNode (HDFS) and the Resource Manager (YARN), slave nodes run the DataNodes and Node Managers; HDFS provides distributed storage and MapReduce/YARN provides distributed processing; designed to run on commodity hardware with high fault tolerance through data replication (default replication factor 3). - Major components of Hadoop (each needs its own slide): - HDFS (Hadoop Distributed File System): the core component; provides reliable and distributed storage for huge datasets; consists of a single NameNode (master — stores metadata/file-system namespace) and multiple DataNodes (slaves — store the actual data blocks); blocks are replicated across nodes for fault tolerance. - YARN (Yet Another Resource Negotiator): the brain of Hadoop; responsible for providing computational resources and managing workloads and job scheduling; consists of the Resource Manager (master) and Node Managers (slaves) plus the Application Master per application. - MapReduce: the data processing layer for parallel processing; it breaks a job into sub-jobs and processes them in parallel using the Map and Reduce phases. - Hive: an open-source data query system used to query or analyse large unstructured and semi-structured datasets; provides an SQL-like language (HiveQL/HQL) that translates queries into MapReduce jobs. - Apache Pig: a high-level procedural language used for parallel processing of large datasets; uses the Pig Latin language and runs on top of Hadoop (executing as MapReduce jobs). - HBase: an open-source, non-relational (NoSQL) column-oriented database that runs on top of Hadoop (on HDFS); provides real-time read/write random access to very large tables. - Apache Mahout and Spark MLlib: both open-source frameworks used for creating scalable machine-learning algorithms and data-mining libraries (Mahout historically on MapReduce, MLlib on Apache Spark). - Apache ZooKeeper: a centralized service used for maintaining configuration information, naming, and providing distributed synchronization and group services for distributed applications. - (Apache Spark may be mentioned as the in-memory processing engine that MLlib sits on.) **8.7 Hadoop and cloud:** - Hadoop is the de-facto batch/big-data processing technology of the cloud era; cloud infrastructure provides the elastic compute and storage on which Hadoop clusters run. Hadoop is offered as a managed service by major cloud vendors (e.g. Amazon EMR, Microsoft Azure HDInsight, Google Cloud Dataproc, and Cloudera/Hortonworks-style distributions), letting organisations rent transient clusters instead of owning them. The synergy: cloud gives elasticity, pay-as-you-go economics, and separation of storage and compute; Hadoop gives a proven distributed storage + processing stack for unstructured data. Wed, 21 Oct 2026 00:00:00 -0400 https://speakerdeck.com/drcd/cloud-technologies-and-platform-services https://speakerdeck.com/drcd/cloud-technologies-and-platform-services Cloud service models define the level of abstraction and management responsibility you share with a provider, while deployment models define where the infrastructure lives and who owns or manages it. Cloud service models define the level of abstraction and management responsibility you share with a provider, while deployment models define where the infrastructure lives and who owns or manages it. Tue, 20 Oct 2026 00:00:00 -0400 https://speakerdeck.com/drcd/cloud-services-and-deployment-models https://speakerdeck.com/drcd/cloud-services-and-deployment-models The evolution of deep learning transformed a fringe mathematical concept into the backbone of modern global technology. It began in the 1940s–1960s with the perceptron, an early mathematical model attempting to digitally mimic biological neurons. However, severe computational limitations and strict linear logic constraints triggered the first "AI Winter," freezing vital research funding. The field revived significantly in 1986 when Geoffrey Hinton and his colleagues popularized the backpropagation algorithm, enabling multi-layer networks to learn efficiently. Despite this theoretical breakthrough, slow CPU hardware and tiny dataset sizes limited practical computing applications during the 1990s. The explosive turning point occurred in 2012 with AlexNet. By leveraging high-powered NVIDIA GPUs and the massive ImageNet dataset, this deep network shattered vision benchmarks. This milestone catalyzed a rapid golden era, leading directly to the breakthrough Transformer architecture (2017) and today’s foundational Generative AI platforms that permanently reshape software engineering. The evolution of deep learning transformed a fringe mathematical concept into the backbone of modern global technology. It began in the 1940s–1960s with the perceptron, an early mathematical model attempting to digitally mimic biological neurons. However, severe computational limitations and strict linear logic constraints triggered the first "AI Winter," freezing vital research funding. The field revived significantly in 1986 when Geoffrey Hinton and his colleagues popularized the backpropagation algorithm, enabling multi-layer networks to learn efficiently. Despite this theoretical breakthrough, slow CPU hardware and tiny dataset sizes limited practical computing applications during the 1990s. The explosive turning point occurred in 2012 with AlexNet. By leveraging high-powered NVIDIA GPUs and the massive ImageNet dataset, this deep network shattered vision benchmarks. This milestone catalyzed a rapid golden era, leading directly to the breakthrough Transformer architecture (2017) and today’s foundational Generative AI platforms that permanently reshape software engineering. Tue, 20 Oct 2026 00:00:00 -0400 https://speakerdeck.com/drcd/ai-and-deep-architect-deep-learning https://speakerdeck.com/drcd/ai-and-deep-architect-deep-learning Combining deep reinforcement learning (DRL) with multimodality allows autonomous agents to process and learn from diverse data types—such as visual images, text instructions, structured sensor telemetry, and audio—to make sequential decisions. Combining deep reinforcement learning (DRL) with multimodality allows autonomous agents to process and learn from diverse data types—such as visual images, text instructions, structured sensor telemetry, and audio—to make sequential decisions. Mon, 05 Oct 2026 00:00:00 -0400 https://speakerdeck.com/drcd/deep-reinforcement-learning-and-multimodality https://speakerdeck.com/drcd/deep-reinforcement-learning-and-multimodality