Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Chunk computing

Chunk computing

Chunk computing
This talk is about Chunk computing and exemples in PHP

Article associated to this talk

https://blog.darkwood.com/article/chunk-computing-in-php

Video associated to this talk

https://www.youtube.com/watch?v=JBx4bjB8HIM

Mathieu Ledru

August 28, 2024
Tweet

More Decks by Mathieu Ledru

Other Decks in Technology

Transcript

  1. address the inherent limitations of memory making it easier to

    manage, understand, and use information “ @matyo91 27-08-2024
  2. A "chunk" is a smaller, manageable segment of a larger

    data set used for easier data transfer and processing. “ @matyo91 27-08-2024
  3. File Splitting: File-sharing services like BitTorrent break large files into

    smaller ‘chunks’ for faster, more efficient data transfer. Video Streaming: Platforms like YouTube and Netflix split videos into ‘chunks’ to enable smoother streaming without waiting for the entire video to load. Big Data Processing: Tools like Apache Hadoop process large datasets in parallel by dividing them into smaller ‘chunks,’ speeding up analysis. AI Model Training: In AI, data is split into smaller batches during training, enabling frequent updates, faster learning, and efficient handling of large datasets. Examples @matyo91 @matyo91 27-08-2024
  4. Dynamic Allocation: Memory is allocated from heaps, which can be

    time-consuming. Performance Impact: Frequent heap management calls may slow down systems. Chunking Strategy: Groups related allocations to minimize calls. Example: Allocate/free multiple objects at once (e.g., 8 at a time), reducing overhead. Chunk Memory Management @matyo91 @matyo91 27-08-2024
  5. Http Chunked Response : Transfert-Encoding: chunked Available since 1997 :

    HTTP 1.1 (RFC 2068) @matyo91 @matyo91 27-08-2024 Http Chunk
  6. Chunk: A chunk is a small piece of binary data

    transferred from source to destination via streams. It carries all necessary information for processing. Buffer: A buffer is temporary RAM storage that manages chunk data. When full, it sends data for processing, acting as an intermediary between data reception and processing. @matyo91 @matyo91 27-08-2024 Chunk vs Buffer Data
  7. @matyo91 @matyo91 27-08-2024 Chunk vs Buffer Data Chunks Buffer A

    piece of binary data. A class for managing binary data. Contains binary data. Manages data in binary and other formats. In octet format, it must be handled carefully to avoid data corruption. Handles exceptions when managing chunk data. Independent container, managed by Buffer. Dependent on Chunks or streams for data transfer. Used for retrieving data externally. Used to retrieve and format data.
  8. @matyo91 @matyo91 27-08-2024 Chunk vs Buffer Data Chunks Buffer Uncontrolled

    flow. Controlled flow via Buffer and streams. Cannot be created. Can be created to contain Chunks. Example: <Buffer 48 65 6c 6c 6f>. Underlined part is a chunk. Example: <Buffer 48 65 6c 6c 6f>. Entire contents are a buffer. Stores one byte of data. Size depends on RAM and hardware. Uncontrolled flow. Controlled flow via Buffer and streams.
  9. Thunk is a subroutine used to inject a calculation into

    another subroutine. Thunks are primarily used to delay a calculation until its result is needed, or to insert operations at the beginning or end of the other subroutine. They have many other applications in compiler code generation and modular programming. @matyo91 @matyo91 27-08-2024 Chunk vs Thunk
  10. Hello! I Am Mathieu Ledru You can contact me at

    @matyo91 27-08-2024 @matyo91 Thanks! Any questions?