Apache Fory Row Format Explained

Apache Fory Row Format Explained

Apache Fory is a multi-language serialization framework designed for high-performance use cases. It supports object graph serialization and a cache-friendly row format for analytics workloads. Row format enables random field access and partial serialization without deserializing entire objects.

Fory defines a binary row format optimized for data processing workloads where you only need to read some fields without reconstructing the entire object. Row format works with Java, Python, and C++ and integrates with Arrow for analytics. This lets you convert row-format data directly into Arrow record batches and tables for analytics workflows.

I studied Fory’s row format deeply while working on row format implementations in Go, Swift, Dart, and JavaScript. This article shares what I learned, how row format works, and what it takes to implement it correctly in new runtimes.

Row Format Layout

Row format binary consists of two primary regions:

+------------------+------------------+-----+------------------+------------------+
| Null Bitmap      | Field 0 Slot     | ... | Field N-1 Slot   | Variable Data    |
+------------------+------------------+-----+------------------+------------------+

Null Bitmap records which fields are null.
Field Slots hold each value or an offset+size pair for variable-length fields.
Variable Data holds the actual content for strings, lists, maps, and nested structs.

+------------------+
| Null Bitmap      |
+------------------+
| Field Slots      |
+------------------+
| Variable Data    |
+------------------+

This layout lets readers compute the exact byte position of any field using offsets, without scanning other fields.

Fixed and Variable Regions

Row format separates fixed content from variable content:

Fixed Region:
+------------------+------------------+
| Null Bitmap      | Field Slots      |
+------------------+------------------+

Variable Region:
+------------------+------------------+
| Strings, arrays, nested struct bytes|
+------------------+------------------+

Fixed region has a predictable size. Variable fields live in the variable region. Each variable-length field in the fixed region stores:

+----------------------+----------------------+
| Relative Offset (32) | Size (32)            |
+----------------------+----------------------+

Readers use the offset plus the row start address to locate the value in the variable section. Size gives the exact length to read.

Apache Arrow Integration

Fory row format works with Apache Arrow for analytics workloads. Arrow defines a language-agnostic memory layout optimized for large datasets and analytical operations. Fory uses Arrow schemas to share data with Arrow tools and formats.

In row format, the schema produced by inference (in Java or Python) is also an Arrow schema. This lets you convert row data into Arrow record batches or tables without translating the layout manually. You can then export or process the same data with analytics engines that support Arrow formats like Parquet, Flight, and dataframes.

In Java:

Schema schema = TypeInference.inferSchema(BeanA.class);
ArrowWriter arrowWriter = ArrowUtils.createArrowWriter(schema);
Encoder<BeanA> encoder = Encoders.rowEncoder(BeanA.class);
// write multiple rows as Arrow record batch
return arrowWriter.finishAsRecordBatch();

In Python:

import pyfory
encoder = pyfory.encoder(Foo)
encoder.to_arrow_record_batch([foo] * 10000)
encoder.to_arrow_table([foo] * 10000)

This integration gives row format a bridge to the wider analytics ecosystem. You do not need a separate conversion layer. The same schema used to encode rows drives conversion into Arrow formats.

Nested Structs

Nested structs are stored as complete row structures in the variable region.

Parent Row:
+------------------+--------------------+
| Fixed region     | Nested struct data |
+------------------+--------------------+

Nested Row:
+------------------+------------------+
| Null Bitmap      | Field Slots      |
+------------------+------------------+
| Nested Variable Data                |
+------------------+------------------+

The nested row itself has its own null bitmap, field slots, and variable data region. This supports arbitrary nesting without flattening everything into the parent.

Standard vs Compact Format

Fory defines two row formats: Standard and Compact.

Standard Format

Standard format is the baseline. It focuses on cross-language compatibility and simplicity. Key rules include:

• Field slots are fixed to 8 bytes each.
 • Null bitmap is aligned to 8-byte boundaries.
 • Variable data uses relative offset plus size.
 • Strict alignment rules keep layout consistent across languages.

This makes it possible to share row binaries between Java, Python, C++, and future languages.

Compact Format (Java Only)

Compact format is an optimized layout available only in Java. It is not available for any language yet. It reduces space at the cost of stricter ordering rules and relaxed alignment. Key differences:

• Fields use their natural width (1, 2, 4, or 8 bytes), not always 8 bytes.
 • Null bitmap is byte-aligned and placed after fields.
 • Fields are sorted by alignment requirements to reduce padding.
 • Bitmap is skipped completely if all fields are non-nullable.

Compact format produces smaller rows, useful in memory-tight scenarios, but it is Java-specific and not recommended for cross-language exchange.

Zero Copy Access

Row format supports zero copy reads. You wrap binary data with a reader object and invoke accessors like:

foo_row = pyfory.RowData(encoder.schema, binary)
foo_row.f2[100000]
foo_row.f4[200000].f2[5]

The reader computes exact byte positions and returns values directly. No intermediate objects are built for unused fields.

Java Implementation Insight

In Java, row format uses runtime code generation. It infers schema from a bean class, generates encoder/decoder source code, compiles it at runtime, and caches it. The compiled class then encodes and decodes rows without reflection overhead.

Python Implementation Insight

Python row format uses pyfory.format and pyarrow. Type annotations and dataclass fields produce a pyarrow.Schema. This schema drives the layout and direct access. Python readers access fields via the schema, mapping binary data into Python values lazily.

Why This Matters for Go, Swift, Dart, JavaScript

This GSoc project focuses only on implementing the standard row format rules as defined in the official row format specification. Compact format is not included. The implementation must follow the standard row format rules exactly, including:

• 8-byte alignment
 • fixed 8-byte field slots
 • null bitmap behavior
 • offset+size encoding for variable fields
 • deterministic padding

Compact format is not part of this task.

When Row Format Fits Your Workloads

Row format fits workloads where:

• You read some fields often and skip others.
 • You work with large nested structures.
 • You need cross-language binary exchange.
 • You process data in analytics pipelines integrated with Arrow.

Row format reduces CPU and memory use compared with traditional full-object serialization.

Row format gives you a predictable, efficient binary representation of structured data. It supports cross-language access and integration with analytics workflows. The upcoming support in Go, Swift, Dart, and JavaScript will open up new possibilities for high-performance, low-overhead data access across platforms.

If you are exploring row format in your language of choice, study the spec closely and try mapping your layout to it. Share your progress and questions with the Fory community on Slack or GitHub.