Show and Tell: bit-jev — structured decisions (option scoring, no answer decoding) on a BitNet backbone, exported as I2_S GGUF with a native CPU runner

[Zeaulo (Zeaulo)]

Hi all,

I'm sharing bit-jev, a small project that uses a BitNet b1.58 backbone for structured decisions rather than text generation. The caller declares questions and candidate options. A pointer head reads the hidden states and scores the options directly, so no answer tokens are generated. It supports three question types: choice, noul (yes/no), and score (ordinal).

This is an independent project. It is not affiliated with or endorsed by Microsoft. The backbone and the native inference base come from microsoft/BitNet.

How it relates to bitnet.cpp

• Pipeline: BitNet BF16 base → LoRA + pointer-head training → teacher-student distillation into a full student backbone without LoRA → export the student as I2_S GGUF, plus a separate float32 pointer head (head.f32).
• test/bootstrap_bitnet.py fetches pinned BitNet / llama.cpp commits and applies a ReLU² compatibility patch. A small native runner (core/native/main.cpp) then loads the GGUF backbone and scores the options with the f32 head.
• Checkpoint: jinghao1632/bit-jev-2b-distilled. The I2_S backbone is about 1.19 GB.

Numbers (single-question case, with caveats)

There is one fixed dev question: 703 input tokens, 77 candidate options, native I2_S CPU path. Native latency excludes model loading.

CPU path	Mean latency	Runs	Peak process RSS
AMD EPYC 9654, 32-core container quota / 32 threads	1,954.46 ms	6 (two launches)	1,625.88 MiB
Xeon Gold 6459C, 8 threads (historical)	3,127.88 ms	3	1,622.74 MiB
Xeon Gold 6459C, 16 threads (historical)	1,972.17 ms	3	1,624.52 MiB

An EPYC 16-thread check on the same host averaged about 3,210.12 ms. The EPYC build used g++ 11.4.0, CMake 3.31.10, Release, GGML_NATIVE=ON, and GGML_OPENMP=ON. The per-run timings, source commit, and model SHA-256 are in docs/benchmark-data. This is a single small dev case. It does not represent general latency or accuracy, and the input text is not public.

As a separate check, I also measured the public microsoft/bitnet-b1.58-2B-4T-gguf I2_S file at a pinned older revision. I chose that revision because of the reports in #608. With llama-bench -p 128 -n 32 -r 5 -t 8 -b 128 -ub 128 on a Ryzen 7 4800H (8 threads, CPU-only), the median prefill was 56.23 tok/s, the median decode was 4.57 tok/s, and peak RSS was 1,230.19 MiB. The commands, the compatibility patch, and the per-run samples are in docs/BENCHMARKS.md.

Try it

• Windows x86_64 (AVX2): pip install bit-jev, then python -m bit_jev.demo (local Gradio page). The wheel ships a prebuilt CPU runner.
• Linux / macOS: source build (Git, CMake 3.28+, C++17). See docs/CPU_QUICKSTART.md.
• Online CPU demo (runs on a free 2-vCPU ModelScope container): https://modelscope.cn/studios/JingHao9616/bit-jev-demo · Hugging Face entry: https://huggingface.co/spaces/jinghao1632/bit-jev-demo

Limits

• Only the Windows x86_64 AVX2 wheel has been tested. The Linux (x86_64/ARM64) and macOS (Intel/Apple Silicon) source builds have not been tested yet.
• Multi-question requests repeat the shared-state computation, because the runner evaluates one causal row per question.
• The code is Apache-2.0. The weights have no separate open-weights license. The training data included Yelp review records, and a permission request was still unanswered as of 2026-09-28. See the model card before use.

Where help would be useful

1. Source builds and timings on Linux x86_64/ARM64 and Apple Silicon: CPU model, threads, --batch, build flags, and latency_ms.
2. Re-running the public-base llama-bench measurement on other CPUs, so the I2_S numbers have more reference points.
3. Feedback on whether the pinned BitNet/llama.cpp + ReLU² patch setup is the right way to consume I2_S GGUF outside run_inference.py.

Thanks for BitNet. Feedback and reproduction reports are welcome.