Building Native Bitmap Functions for Apache Fluss

How a documentation task opened a six-month journey through Apache governance, Flink internals, and the gap between code that works and code that ships.

August 24, 2026 · 10 min read · Prajwal Banakar

GSoC Open Source Apache Fluss Distributed Systems Java RoaringBitmap

In February 2026, I picked up a GitHub issue in Apache Fluss asking for a documentation quickstart. It seemed like a good starting point — something concrete, something achievable. I had no idea it would eventually lead me to author a Fluss Improvement Proposal, write 7,000+ lines of code that shipped in Fluss 1.0, and fundamentally change how I think about software.

This is that story.

What is Apache Fluss?

Before going further, a quick explanation for anyone who hasn't heard of Fluss.

Most real-time analytics systems face a fundamental tension: you need data processed fast (streaming), but you also need to query it efficiently (analytics). Traditional stream processors like Apache Flink are great at computation but aren't designed for random-access reads. Data warehouses handle analytics well but are too slow for real-time writes.

Apache Fluss is a streaming storage system that sits between these two worlds. It's designed to store streaming data in a way that's immediately queryable. Think of it as a real-time database optimized for Flink — you write events into it continuously, and it maintains pre-aggregated results that you can query instantly without scanning millions of rows.

A concrete example: imagine you run an e-commerce platform with millions of users clicking on products every second. You want to know, right now, how many unique visitors each product page had in the last hour — broken down by country. With traditional approaches you'd either scan all events (too slow) or batch-compute (too stale). Fluss handles this by maintaining compressed bitmaps of visitor IDs per page, updated continuously as events arrive.

That's exactly what my project was about.

Where It Started

The task I picked up — issue #2659 — was to write a Real-Time Page User Profile quickstart tutorial. The idea was to demonstrate two Fluss features working together: auto-increment columns (which automatically assign integer IDs to incoming data) and the aggregation merge engine (which lets you pre-aggregate data directly in storage).

I wrote the tutorial. The reviewers asked for Docker Compose setup, reproducible steps, visual diagrams. I added those. Then came the feedback that stopped me cold:

"I believe we need bitmap functions like RB_CARDINALITY and RB_OR_AGG for aggregating the result bitmap. Without these functions, the results would be meaningless."

Fluss already had storage-level bitmap aggregation built in. But there was no way to call it from Flink SQL. Users had to download external JAR files and manually register functions. The tutorial couldn't demonstrate the feature properly because the feature was incomplete.

That was the gap. And it was sitting there in the codebase, documented nowhere, complained about by no one because most users just worked around it.

I decided to fix it.

Authoring FIP-37

In a personal project, you find a gap and you fill it. You write the code, it works, you move on.

In the Apache Software Foundation, there's a process. First you write a proposal — called a Fluss Improvement Proposal (FIP-37) — that describes the problem, the proposed solution, the API design, and what you're explicitly not doing. Then you post it to the community for discussion, hold a vote, and only then proceed with implementation.

The reason this process exists is not bureaucracy. It's because code in a production system has to be maintained by people who weren't involved in writing it, and used by companies who can't afford to upgrade every six months. Decisions made publicly, with documented rationale, create a trail that lets future contributors understand why things are the way they are.

My first draft of FIP-37 was ambitious. I proposed a native BITMAP type in the Flink SQL type system, server-side aggregation pushdown that would offload computation from Flink to the Fluss storage layer, and a comprehensive set of SQL functions. Three layers, lots of moving parts.

The community feedback cut it down to one layer.

The BITMAP type depended on a Flink proposal (FLIP-556) that wasn't finalized. The server-side pushdown would require the storage layer to run full-table scans without any memory budget or backpressure controls — a real production risk that could starve read/write operations. The committers suggested narrowing the FIP to what could actually ship correctly: SQL functions registered in the Flink catalog, operating on the existing BYTES columns.

I was disappointed for about a day. Then I realized: the feature I actually shipped was the right feature. Not the most impressive-on-paper feature. The right one.

Why RoaringBitmaps?

Imagine you have 500 million user IDs and you want to know how many unique ones visited a website today. The naive approach: store every visit, then COUNT(DISTINCT user_id). Problem: with billions of events, this is slow and memory-intensive.

A RoaringBitmap is a compressed data structure that stores sets of integers extremely efficiently. Instead of storing individual IDs, it stores which IDs are in the set, compressed. A bitmap representing "users 1, 2, 3, 1000000, 1000001" takes far less space than six integers, and set operations (union, intersection, difference) run in microseconds.

The critical insight for analytics: if you store bitmaps instead of raw counts, you can union them across dimensions without double-counting. "How many unique users visited page A this month?" — OR the 30 daily bitmaps. "How many visited page A but not page B?" — AND NOT the two page bitmaps. Exact counts, without approximation, without scanning raw event data.

This is what companies like ByteDance, Alibaba, and Meituan use for their real-time UV (unique visitor) analytics at billion-user scale.

The Implementation

The implementation ended up as three distinct layers, each building on the previous.

Layer 1 — Infrastructure. Before you can write bitmap functions, you need a way to serialize and deserialize bitmaps to/from bytes, and a base class that aggregate functions can extend. This sounds simple but took two weeks and 15 review comments. One comment made me realize I'd implemented something that would fail during Flink checkpoint recovery — I hadn't thought about distributed state at all.

Layer 2 — Function implementations. This is where the actual logic lives: 12 functions split across two phases. The aggregate functions (like rb_build_agg and rb_or_agg) accumulate bitmap state across rows. The scalar functions (like rb_cardinality and rb_contains) operate on individual bitmap values.

The most technically interesting piece was rb_xor_agg. XOR is a symmetric difference — elements that appear in an odd number of bitmaps. It's self-inverse, which means it supports retraction (undoing previously accumulated data). But my first implementation used a boolean initialized flag that couldn't represent a subtle state: what happens when you accumulate a bitmap and then retract it? The count returns to zero. The result should be NULL — no net input received. But initialized stays true, so the function returns an empty bitmap instead.

The fix was replacing boolean initialized with long nonNullCount, incrementing on accumulate and decrementing on retract. The code change was ten lines. Understanding why it was wrong took much longer.

Layer 3 — FlussCatalog registration. This is the piece that makes everything accessible. By adding each function to a map in FlinkCatalog, they become available automatically after USE CATALOG fluss_catalog — no manual registration, no external JAR. This is what closes the gap I found in February.

After the code was merged, a reviewer noticed that the catalog methods were resolving bitmap functions even when the database component of a fully qualified reference didn't exist. fluss_catalog.nonexistent_db.rb_build(...) was returning results instead of an error. This was a bug in the catalog semantics. I fixed it as a priority=critical issue for the 1.0 release — the kind of thing that only gets caught when someone reads the code carefully, not when they run the happy-path tests.

The Result in Practice

Here's the end result in practice. After USE CATALOG fluss_catalog, you can build a real-time unique visitor analytics system that looks like this:

-- Table stores one cumulative bitmap per page
CREATE TABLE page_uv (
    page_id   BIGINT,
    uv_bitmap BYTES,
    PRIMARY KEY (page_id) NOT ENFORCED
) WITH (
    'table.merge-engine'   = 'aggregation',
    'fields.uv_bitmap.agg' = 'rbm32'
);

-- Every 5 seconds, build per-page bitmaps from click events
-- Fluss OR-s each new bitmap into the stored one at the storage layer
INSERT INTO page_uv
SELECT page_id, rb_build_agg(user_id)
FROM TABLE(TUMBLE(TABLE click_events, DESCRIPTOR(proc_time), INTERVAL '5' SECOND))
GROUP BY page_id, window_start, window_end;

-- Query unique visitors
SELECT page_id, rb_cardinality(uv_bitmap) AS uv
FROM page_uv WHERE page_id = 1;

-- Who visited page A but not page B?
SELECT rb_cardinality(rb_andnot(a.uv_bitmap, b.uv_bitmap)) AS exclusive_visitors
FROM page_uv a, page_uv b WHERE a.page_id = 1 AND b.page_id = 2;

The Flink job is essentially stateless — it emits bitmap updates and forwards them to Fluss. All the accumulation happens at the storage layer. This is the quickstart tutorial I eventually shipped as part of Fluss 1.0.

Key Learnings

Review comments are information, not criticism. When my first PR came back with 15 comments, my instinct was to defend my choices. The better move was to treat each comment as a question the codebase was asking me: "Did you think about this?" Usually I hadn't.

Scope discipline is a technical skill. The hardest decision in FIP-37 wasn't a code decision — it was agreeing to remove the features I'd spent weeks designing. A proposal that ships is infinitely more valuable than a proposal that's ambitious.

Good tests force you to understand what you're testing. When I wrote tests that passed on Flink 1.20 but failed on Flink 2.2, it wasn't because my tests were broken. It was because I'd made assumptions about exception types that differed across versions. Fixing those tests taught me more about Flink internals than reading documentation.

Documentation is part of the feature. The SQL Functions reference page I wrote as PR #3823 took more iterations than some of the code PRs. Getting the null semantics right, making the examples self-contained, explaining the difference between scalar and aggregate functions for readers who aren't familiar with Flink — all of that matters as much as the code.

The community is real. This sounds obvious but it wasn't to me before this year. The people who reviewed my code are engineers at companies building real systems with real users. Their feedback comes from experience with failure modes I haven't encountered yet. That's not a power dynamic — it's just information asymmetry, and being on the learning side of it is a privilege.

Evaluations & 1.0 Release

At the midterm evaluation, my mentor wrote:

"He has shown good ownership of the project. He actively follows up on reviews, responds constructively to feedback, and improves the implementation when reviewers ask for stronger end-to-end coverage. He also contributes beyond the strict GSoC scope, including documentation, website improvements, and project health work."

That feedback reflected the work done beyond the core implementation — theme-aware diagrams for the website, client documentation for Python, Rust, and C++, and test coverage improvements.

With the final evaluation successfully completed, all deliverables are merged into upstream:

12 Native SQL Functions implemented for scalar and aggregate operations
FIP-37 officially shipped in Apache Fluss 1.0
The Real-Time Page User Profile quickstart tutorial and SQL Functions reference are live in the official docs

What's Next

I'm continuing to contribute to Apache Fluss post-GSoC. The rb64_ function family (64-bit bitmap support) is the natural next FIP. The quickstart tutorial is now merged. There are open issues worth picking up.

The longer-term goal is becoming an Apache Fluss committer — which requires sustained contribution across multiple areas of the codebase over time, not just one summer. That's fine. I now understand what that path looks like.

If you're considering GSoC, or open-source contribution more generally: the gap between "I can write code" and "this code belongs in production" is real, and closing it is the most valuable thing you can do for your engineering ability. The way to close it is to put your code in front of people who will tell you honestly what's wrong with it, and then fix it.

That's what this summer was.

The full technical report from my GSoC project is on GitHub. The step-by-step tutorial for building a real-time UV analytics system with Fluss is in the official documentation. All my contributions are at github.com/apache/fluss.