How I Added JavaScript Code Generation to Apache Fory’s IDL Compiler

How I Added JavaScript Code Generation to Apache Fory’s IDL Compiler

Apache Fory is a multi-language serialization framework. You define your data types once in a schema file. The compiler generates native serialization code for Java, Python, Go, Rust, C++, and C#. JavaScript was missing. The PR #3394 adds JavaScript. It took me over 1 month to build this JavaScript Generator.

The Problem

Fory uses .fdl files (Fory Definition Language) as the source of truth for data types. Write a schema once, run the compiler, and you get native serialization code in every supported language. Every language except JavaScript.

JavaScript developers faced two choices: write type definitions by hand or skip IDL-based serialization entirely. Neither works for a production-grade cross-language serialization framework.

How I Fixed This and Built the Generator

I started with no existing JavaScript IDL infrastructure. No generator, no integration test project, no CI step. The starting point was reading what already existed for other languages.

Phase 1: Understanding the Pattern

Before writing a single line, I read the existing generators: CSharpGenerator, SwiftGenerator. Every generator extends BaseGenerator. Every generator follows the same structure: read the schema AST, map types, write files. The Python fory_compiler handles file management and CLI wiring. The generator only needs to produce strings.

I also read the compiler docs under docs/compiler/ to understand how the FDL type system maps to language-specific types. This took time, but skipping it would have cost more time later.

Phase 2: The Wrong First Attempt

My first version generated TypeScript from inside the JavaScript runtime itself. I thought the right place for JavaScript codegen was in JavaScript. The first review comment from the maintainer was direct: this was wrong. Every language uses the Python fory_compiler. JavaScript should not be an exception.

This was a useful failure. Reading PR #3406 (the reference the maintainer pointed to) made the correct approach clear. The generator belongs in Python alongside every other generator.

Phase 3: Rewriting in Python

I rewrote JavaScriptGenerator from scratch in Python. The class follows the same BaseGenerator contract. I mapped all 25 FDL primitive kinds to TypeScript types, handled snake_case to camelCase field conversion, generated discriminated union types, and built registration helper functions so the generated code connects to the Fory runtime without extra wiring.

The CLI got a — javascript_out flag. The generator registry got a new entry. The integration test directory got a new javascript/ folder with Jest, a tsconfig, and round-trip tests.

Phase 4: Hitting the Runtime Bugs

Running the tests exposed something unexpected. The JavaScript Fory runtime itself had bugs. The xlang tests, which verify cross-language serialization against Java, had 10 failures. These were not caused by my code. They were pre-existing issues in the runtime that the new IDL tests exposed by exercising paths no earlier test had reached.

I debugged each failure methodically. The isMonomorphic() bug in typeInfo.ts caused struct fields to write typeInfo in schema-consistent mode when they should not. The map isAny() bug caused byte misalignment by skipping typeInfo for user-defined map value types. The empty-props struct bug caused infinite recursion because the generated write() method called back into itself through the serializer lookup.

Each fix required reading the Java implementation to understand the expected behavior, then tracing the JavaScript code to find where the divergence happened. The Java implementation was the reference for correctness throughout.

Phase 5: Getting Everything Green

After the runtime fixes, all 79 xlang tests passed. The IDL integration tests passed. The CI pipeline ran both the xlang suite and the IDL test suite on every push.

The full arc went from zero infrastructure to a working generator, integration tests, CI coverage, and runtime correctness fixes, across the Python compiler and the JavaScript runtime.

My Learnings from This

• Learned that cross language systems rely on strict binary compatibility, where small mismatches break decoding across runtimes
• Gained clarity on how null bitmap, type info, and field layout define correctness at the byte level
• Understood how monomorphic and non monomorphic types control whether type metadata is written or skipped
• Saw why schema consistent mode and compatible mode require different serialization behavior
• Observed that missing type info reads cause buffer misalignment and corrupt subsequent data
• Understood how DFS ordering ensures dependent types register before use and prevents runtime failures
• Saw that map serialization must align with Java behavior, especially for user defined types
• Gained insight into dynamic code generation and how small logic errors lead to recursion or crashes
• Observed how cross language testing, with Java invoking JavaScript through ts node, validates real interoperability
• Understood how deterministic binary tests ensure correctness across languages at the byte level

This work strengthened my understanding of how serialization systems operate at the byte level and how small inconsistencies break cross language guarantees. It also reinforced the importance of using a reference implementation, in this case Java, to guide correctness across runtimes. Building the generator and fixing runtime issues gave me hands on experience with compiler design, binary protocols, and multi language system integration.