The Local-First S3 Index for LLM Data Infrastructure
—438 concepts·1992 relationships·51 guidesEach technology, standard, and architecture in the index belongs to one or more topics — the conceptual anchors that define the S3 / AI-memory-infrastructure ecosystem, sorted by how connected they are.
Amazon's Simple Storage Service and the broader ecosystem of S3-compatible object storage. The root concept of this e...
The storage paradigm of flat-namespace, HTTP-accessible binary objects with metadata. Data is addressed by bucket and...
The emerging tier of persistent, object-storage-backed memory architecture sitting between GPU HBM and cold S3 — the ...
The category of specifications (Iceberg, Delta, Hudi) that bring table semantics — schema, partitioning, ACID transac...
The convergence of data lake storage (raw files on object storage) with data warehouse capabilities — ACID transactio...
The layer of standardized orchestration fabrics, communication protocols, model gateways, and agent runtimes that sit...
The intersection of large language models and S3-centric data infrastructure. Scoped strictly to cases where LLMs ope...
Using S3 as the central data layer for machine learning workflows: storing training data, model checkpoints, feature ...
The practice of building and querying vector indexes over embeddings derived from data stored in S3.
The discipline of maintaining catalogs, schemas, statistics, and descriptive information about objects and datasets s...
The compliance, audit, lineage, and retention discipline applied to persistent AI memory — extending traditional data...
The practice of deploying S3-compatible object storage on infrastructure that is fully controlled by a specific organ...
The pattern of storing raw, heterogeneous data in object storage for later processing. Data arrives in its original f...
The architectural shift toward minimizing data movement between storage and inference compute — placing computation a...
The set of technologies eliminating CPU bounce-buffers between object storage and GPU memory — establishing direct me...
Deploying S3-compatible object storage at geographically distributed edge locations with synchronization to a central...
Techniques for tracking and managing changes to datasets stored in object storage over time, including snapshots, bra...
The discipline of building production retrieval systems that go beyond basic Retrieval-Augmented Generation (RAG) — o...
A purpose-built storage tier designed for single-digit millisecond latency, using a directory-based namespace within ...
Kubernetes-native provisioning and management of S3 buckets using operators, the Container Object Storage Interface (...
The orchestration of memory and shared state across multi-agent environments — the architectural pattern that enables...
A design philosophy that treats object metadata as a first-class, queryable resource rather than an afterthought. Ena...
The ability to query a dataset as it existed at a previous point in time by leveraging immutable snapshots and metada...
I run local AI. Why do I care about S3?
Guided path from local inference to the S3 storage ecosystem — storage, formats, retrieval, and the tradeoffs that matter.
Architectural shifts as they happen. Each post anchors on a pre-existing pain point and walks through what changed.
When the Engines Conceded the Object Store
In August this index called that object storage was reclassifying from cold archive to AI's active memory tier. October scores that call in two directions at once. Milvus rewrote its engine around lake-native storage for 3.0 and Polars turned out-of-core spill on by default for 2.0 — two majors in one month, both conceding that the object store holds the truth and memory is the cache. Meanwhile vLLM 0.32.0 shipped substantial KV-cache work and none of it touches an object-storage tier. The storage and query layers are doing what we said. The inference layer still is not.
What Fits on One Box Now
A practical pass at the local-first stack: what you can actually run on a single machine with your own object storage in late 2026, which layer to pick at each decision, and where one box genuinely stops being enough. October moved the line — Polars 2.0 turned out-of-core spill on by default and Milvus 3.0 rebuilt on lake-native storage — so workloads that needed a cluster last quarter no longer do. This is the decision path, with the guides that go deeper at each step.
When the Catalog Started Handing Out Keys
September 2026 turned the lakehouse catalog into an agent-callable control plane and, in the same motion, into the thing holding your object-storage credentials. OpenMetadata 2.0 ships MCP on by default and rebrands as a context layer. Unity Catalog 0.6.0 quietly expires tokens that were previously valid forever and re-keys its credential cache. And CVE-2026-34214 showed what the failure mode looks like in production: a query engine handing bucket keys to anyone with SQL write privilege. Three facts, one boundary.
Getting Data Into the Lakehouse — Choosing a CDC-to-Iceberg Path in 2026
The default way to move operational data into the lakehouse used to be a four-system pipeline: Debezium reads the database WAL, Kafka buffer...
48Choosing a Lakehouse Catalog — Polaris vs. Unity Catalog vs. Gravitino vs. Cloud-Native
In 2024 the catalog was an afterthought — somewhere to remember where the tables lived. By mid-2026 it is the **control plane** of the lakeh...
19The Post-MinIO Landscape — Self-Hosted S3 Branches Out
For nearly a decade, "self-hosted S3" meant MinIO. It was the default answer — simple, fast, single-binary, open-source. The February 2026 a...
37Picking an AI Memory Layer in 2026 — Mem0 vs. Zep vs. Build-Your-Own
The shift from stateless LLM inference to stateful, multi-agent systems forces a decision that didn't exist two years ago: where does agent ...