Flow Tokens — Token Optimization Is Stream Discipline

Présentation canonique Slidewire par Mathieu Ledru (@matyo91), Darkwood. 19 slides, 23 pages PDF avec pagination des contenus denses. Version interactive (mise en ligne à venir) : https://slides.darkwood.com/slides/flow-pipe

Transcript

1. Darkwood · PHP 8.5 · Flow · Symfony Flow Tokens Token optimization is stream discipline Symfony Console demo · flow-pipe Noisy context → Pipeline → Budgeted signal @matyo91
2. The problem Agentic workflows consume too many tokens. Engine logs Tool outputs RAG dumps Verbose, repetitive, ANSI- Stack traces, debug lines, Useful signal buried in colored. boilerplate. noise. Context floods the agent - before any reasoning starts. 01 | problem @matyo91
3. Wrong instinct The issue is not word shortening. ABBREVIATE VOCABUL ARY → → → waterfall wf pipeline pln stream strm Destroys readability. Saves almost nothing. 02 | not-shortening STREAM DISCIPLINE Strip noise Chunk bounded segments Compress duplicates Apply a token budget @matyo91
4. Slide 4 · 1/1 · représentation statique Root cause Uncontrolled context. Engine logs Tool outputs Repeated traces Noisy chunks Agent context window Data enters as one giant string - not as a disciplined stream. 03 | root-cause Version interactive (mise en ligne à venir) : https://slides.darkwood.com/slides/flow-pipe @matyo91
5. Thesis Token optimization is not word shortening. Token optimization is stream discipline. How data moves - not how words are spelled. 04 | thesis @matyo91
6. The Flow angle Flow is not only about running tasks. It controls how data moves: as a pipe as a stream as chunks as measurable context as budgeted context for humans · machines · agents 05 | flow-angle @matyo91
7. The demo flow-pipe A Symfony Console project that runs a token pipeline through Flow. php bin/console app:flow-token-demo \ --input=flow-engine-log --show-chunks Deterministic. Local. No external API. 06 | demo @matyo91
8. Slide 8 · 1/1 · représentation statique Architecture Three layers. One pipeline. FlowTokenDemoCommand app:flow-token-demo TokenPipelineService PipelineExpressionParser TokenPipelineFlowRunner FlowFixtureProvider PipelineStepRegistry FlowFactory + FiberDriver PipelineContext 07 | architecture Version interactive (mise en ligne à venir) : https://slides.darkwood.com/slides/flow-pipe @matyo91
9. Slide 9 · 1/2 · représentation statique Project structure Application · Infrastructure · Domain Command FlowTokenDemoCommand CLI I/O only Application TokenPipelineService Parse + run facade Infrastructure TokenPipelineFlowRunner 08 | structure Version interactive (mise en ligne à venir) : https://slides.darkwood.com/slides/flow-pipe @matyo91
10. Slide 9 · 2/2 · représentation statique Project structure Application · Infrastructure · Domain Full pipeline via FlowFactory TokenFlow domain PipelineExpressionParser PipelineStepRegistry PipelineContext FlowFixtureProvider Step classes 08 | structure Version interactive (mise en ligne à venir) : https://slides.darkwood.com/slides/flow-pipe @matyo91
11. Slide 10 · 1/2 · représentation statique Pipeline steps Registered operations - not hardcoded branches. SOURCELoad fixture into stream STRIP_ANSIRemove escape sequences REMOVE_NOISEDrop debug / trace lines NORMALIZE_WHITESPACECollapse blank runs 09 | steps Version interactive (mise en ligne à venir) : https://slides.darkwood.com/slides/flow-pipe @matyo91
12. Slide 10 · 2/2 · représentation statique Pipeline steps Registered operations - not hardcoded branches. CHUNK:300Split into ~300-char segments COMPRESSDeduplicate & collapse repeats BUDGET:1000Keep chunks until token limit SINKFlush output & metrics 09 | steps Version interactive (mise en ligne à venir) : https://slides.darkwood.com/slides/flow-pipe @matyo91
13. Pipeline expression Declarative. Shareable. Readable. source |> strip_ansi |> remove_noise |> normalize_whitespace |> chunk:300 |> compress |> budget:1000 |> sink PipelineExpressionParser splits on |> and resolves names through PipelineStepRegistry. 10 | expression @matyo91
14. Three |> semantics Same symbol. Three layers. Expression DSL PHP 8.5 |> Flow fn() String pipeline language Step ClosureJob mapping Runtime composition "source |> sink" → $flow->fn($job) $step |> (fn ($s) => ...) Visual rhythm of the DSL - at Flow composition time. 11 | three-pipes @matyo91
15. Slide 13 · 1/2 · représentation statique Flow runtime Full pipeline. One graph. $toJob = fn ($s) => new ClosureJob(fn ($ctx) => $s->apply($ctx)); $flow = (new FlowFactory())->createFlow( $firstStep |> $toJob, ['driver' => $driver], ); foreach ($remaining as $step) { $flow = $flow->fn($step |> $toJob); } $flow(new Ip($context)); 12 | runtime Version interactive (mise en ligne à venir) : https://slides.darkwood.com/slides/flow-pipe @matyo91
16. Slide 13 · 2/2 · représentation statique Flow runtime Full pipeline. One graph. $flow->await(); TokenPipelineFlowRunner - each step becomes a ClosureJob on PipelineContext. 12 | runtime Version interactive (mise en ligne à venir) : https://slides.darkwood.com/slides/flow-pipe @matyo91
17. Slide 14 · 1/1 · représentation statique The result From flood to signal. Noisy input 17k chars Cleaned stream strip + normalize Chunked data ~300 char segments Compressed context deduped lines Budgeted output ≤1000 tokens Fixture: flow-engine-log · heuristic ~1 token / 4 chars 13 | stages Version interactive (mise en ligne à venir) : https://slides.darkwood.com/slides/flow-pipe @matyo91
18. Before / after flow-engine-log BEFORE AFTER 4,488 est. tokens 88 est. tokens 17,398 characters 334 characters · 98% reduction Domain vocabulary intact. Noise and repetition removed. 14 | numbers @matyo91
19. Signal vs noise flow-lexicon resists compression. BEFORE AFTER 371 est. tokens 354 est. tokens · 4.6% Semantic content carries signal - abbreviating words is not the strategy. 15 | contrast @matyo91
20. Tomorrow From CLI demo to async streams. Non-blocking streams stream_select Fibers Process pipes PTY / TTY modes Event loop drivers Async workers Flow orchestration layer Read process stdout as it arrives - not after wait(). 16 | future @matyo91
21. Takeaway Control the stream. Not the spelling. Expression |> Flow |> Budgeted context Token optimization is stream discipline. 17 | takeaway @matyo91
22. Slide 19 · 1/2 · représentation statique Resources Go deeper. darkwood/content/flow-pipe - Symfony demo repo app:flow-token-demo - run the pipeline locally darkwood/flow - orchestration package PHP.Watch - PHP 8.5 pipe operator Inspired by a LinkedIn post on agentic context - cited at the end of the article. → Noisy logs @matyo91 Version interactive (mise en ligne à venir) : https://slides.darkwood.com/slides/flow-pipe
23. Slide 19 · 2/2 · représentation statique Resources Go deeper. → Stream pipeline Agent-ready context @matyo91 Version interactive (mise en ligne à venir) : https://slides.darkwood.com/slides/flow-pipe