Apache Avro

What it is

A row-based data serialization format with rich schema definition and built-in schema evolution support. Schemas are stored with the data.

Where it fits

Avro is the ingestion format of the S3 ecosystem. Data flowing from Kafka, operational databases, and streaming systems into S3 often arrives in Avro — because Avro's schema-with-data approach handles the frequent schema changes typical of event streams.

Misconceptions / Traps

Key Connections

What it is

A row-based data serialization format specification with rich schema definition and built-in schema evolution support. Schemas are stored with the data, making files self-describing.

Why it exists

Data arriving into S3 often comes from streaming systems (Kafka) and operational databases where the schema changes frequently. Avro's schema-with-data approach and backward/forward compatibility rules make it ideal for ingestion layers where schema stability cannot be guaranteed.

Primary use cases

Streaming data ingestion into S3 (Kafka → S3), schema-evolving event logs, interchange format between systems writing to object storage.