Why I Rewrote Our Hot Path in Rust

A Go service was burning 40% CPU on JSON serialization. A targeted Rust rewrite via FFI cut p99 latency by 3x.

The Problem

We had a video fingerprinting pipeline processing ~12k requests/sec. The core matching logic was fast — but the service was spending 40% of CPU time serializing response payloads. pprof made this obvious:

(pprof) top10
Showing nodes accounting for 4.2s, 42% of 10s total
      flat  flat%   sum%        cum   cum%
     2.1s 21.00% 21.00%      2.8s 28.00%  encoding/json.(*encodeState).marshal
     0.9s  9.00% 30.00%      0.9s  9.00%  encoding/json.(*encodeState).string
     0.6s  6.00% 36.00%      0.6s  6.00%  runtime.mallocgc

The response struct had nested slices of fingerprint matches — each response was ~15KB of JSON with hundreds of float64 arrays. Go’s encoding/json uses reflection, allocates per-field, and can’t be easily vectorized.

Why Not Just Use a Faster Go Serializer?

I tried the obvious fixes first:

None of these solved the fundamental issue: we were allocating thousands of small objects per response, and the GC was paying for it.

The Rust Approach

Instead of rewriting the entire service, I wrote a shared library (.so) that handles only serialization:

use serde::Serialize;
use serde_json::to_vec;

#[repr(C)]
pub struct SerializeResult {
    ptr: *mut u8,
    len: usize,
    cap: usize,
}

#[no_mangle]
pub extern "C" fn serialize_matches(
    data_ptr: *const u8,
    data_len: usize,
) -> SerializeResult {
    let input = unsafe {
        std::slice::from_raw_parts(data_ptr, data_len)
    };

    // Deserialize from a compact binary format (MessagePack)
    let matches: Vec<MatchResult> = rmp_serde::from_slice(input)
        .expect("invalid msgpack input");

    // Serialize to JSON using simd-json when available
    let mut output = to_vec(&matches).unwrap();
    let result = SerializeResult {
        ptr: output.as_mut_ptr(),
        len: output.len(),
        cap: output.capacity(),
    };
    std::mem::forget(output); // caller frees
    result
}

The Go side calls this via CGo:

// #cgo LDFLAGS: -L./lib -lfingerprint_serial
// #include "serial.h"
import "C"

func serializeMatches(packed []byte) []byte {
    result := C.serialize_matches(
        (*C.uchar)(unsafe.Pointer(&packed[0])),
        C.size_t(len(packed)),
    )
    defer C.free_buffer(result.ptr, result.len, result.cap)
    return C.GoBytes(unsafe.Pointer(result.ptr), C.int(result.len))
}

Results

After deploying to one pod and comparing:

The key win wasn’t just “Rust is faster” — it’s that we moved ~800 allocations per request out of Go’s GC entirely. The Rust code allocates once (the output buffer) and the Go side copies it out immediately.

When Is This Worth It?

A partial FFI rewrite makes sense when:

1. The hot path is isolated — serialization is a pure function with clear input/output boundaries
2. The bottleneck is allocation, not logic — if your code is slow because of algorithmic complexity, Rust won’t magically fix it
3. You can’t change the protocol — if we could switch to Protobuf everywhere, that would’ve been simpler
4. The team can maintain it — a 200-line Rust library with CI is manageable; rewriting the whole service isn’t

What I’d Do Differently

• Start with flamegraphs, not intuition — I almost rewrote the matching logic first, which was already fast
• Use cbindgen from day one — I hand-wrote the C header initially and got bitten by alignment issues
• Benchmark the FFI overhead itself — CGo has ~50ns overhead per call. For us that’s negligible at 12ms baseline, but for microsecond-level work it matters

The full serialization library is ~200 lines of Rust. The CGo wrapper is ~40 lines of Go. Total effort: 3 days including benchmarking and rollout. The service has been running this in production for 6 weeks with zero issues.

This is a pattern I’ll write more about — using Rust as a “performance escape hatch” from Go/Python/Node without committing to a full rewrite.