Split View: BitNet 논문 분석: 1-Bit LLM의 시대 — 삼진 가중치부터 CPU 추론까지

BitNet Paper Analysis: The Era of 1-Bit LLMs — From Ternary Weights to CPU Inference

• Introduction: The Quantization Paradigm and the Meaning of 1-Bit
• BitNet v1: The Birth of BitLinear
  • 1-Bit Weights and the Sign Function
  • The Role of Straight-Through Estimator (STE)
  • Scaling Laws and Initial Results
• BitNet b1.58: The Innovation of Ternary Weights
  • The Power of {-1, 0, +1}
  • Absmean Quantization Function
  • Why Matching FP16 Performance Is Possible
• BitNet a4.8: Hybrid Quantization Strategy
  • Combining 4-Bit Activations with 1-Bit Weights
• BitNet b1.58 2B4T: The First Open-Source Native 1-Bit LLM
  • A 2B Model Trained on 4 Trillion Tokens
  • Comparison with Llama 3 and Qwen 2.5
• bitnet.cpp Inference Framework
  • Architecture Overview
  • Internal Operation of I2_S Kernels
  • ARM and x86 Platform Optimizations
• Performance Comparison: BitNet vs FP16 vs GPTQ vs AWQ
  • Inference Speed and Memory Comparison
  • Key Analysis
• Practical Applications and Deployment
  • Edge Device Deployment
  • Mobile Inference Scenarios
• Limitations and Considerations
  • The Reality of Training Costs
  • Limited Model Sizes and Ecosystem
  • Accuracy Warning: Risky Use Cases
• Failure Cases and Troubleshooting
  • Case 1: Build Failure - CMake Version Mismatch
  • Case 2: Segmentation Fault During Inference
  • Case 3: Out-of-Memory (OOM) Errors
  • Case 4: Incorrect Quantization Format Selection
  • Case 5: Divergence Issues During Training
• Future Prospects
  • NPU Support and Hardware Optimization
  • Toward Sustainable AI
  • Possibilities for Model Size Scaling
  • Training Efficiency Improvements
• References
• Quiz

Introduction: The Quantization Paradigm and the Meaning of 1-Bit

The number of parameters in large language models (LLMs) is growing explosively. GPT-4 class models have hundreds of billions of parameters, and storing them in FP16 alone requires hundreds of gigabytes of memory. During inference, memory bandwidth becomes the bottleneck, making it difficult to achieve reasonable speeds with a single GPU. To address these challenges, Post-Training Quantization (PTQ) techniques such as GPTQ, AWQ, and GGUF are widely used, but all of them quantize an already-trained FP16 model after the fact. When quantizing below 4-bit, performance degradation becomes pronounced, especially in terms of accuracy loss on knowledge-intensive tasks.

Microsoft Research's BitNet series fundamentally overturns this paradigm. By adopting Quantization-Aware Training (QAT), which constrains weights to 1-bit or 1.58-bit from the start of training, information loss from quantization is compensated during the learning process. The core insight is that multiplication operations in weight matrices can be replaced with additions and subtractions. When weights consist only of {-1, 0, +1}, matrix-vector products can be computed using only sign flips and cumulative additions, enabling inference without multipliers. This dramatically reduces energy consumption and enables efficient inference on general-purpose hardware such as CPUs and NPUs.

This article analyzes the papers from BitNet v1 (2023), BitNet b1.58 (2024), BitNet a4.8 (2024), and BitNet b1.58 2B4T (2025) in chronological order, and comprehensively covers the internal architecture of the official inference framework bitnet.cpp, real-world performance benchmarks, operational considerations and failure cases, and future prospects.

BitNet v1: The Birth of BitLinear

1-Bit Weights and the Sign Function

The BitNet v1 paper ("BitNet: Scaling 1-bit Transformers for Large Language Models"), published in October 2023, proposed the idea of replacing the nn.Linear layer in Transformers with BitLinear. In BitLinear, weights are quantized to binary values of {-1, +1} through the Sign function during training.

The core formula is as follows. For real-valued weights W, the binarized weights W_b are obtained:

W_b = Sign(W) = +1  (if W >= 0)
                -1  (if W < 0)

alpha = (1/nm) * sum(|W_ij|)   # scaling factor

Here, alpha is the mean of the absolute values of the original weights, serving to correct the scale of the binary weights. Activations are also quantized, with absmax quantization applied to convert them to b-bit integers.

import torch
import torch.nn as nn
import torch.nn.functional as F

class BitLinear_v1(nn.Module):
    """BitNet v1 BitLinear implementation (simplified educational version)"""
    def __init__(self, in_features, out_features, activation_bits=8):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.activation_bits = activation_bits
        self.Qb = 2 ** (activation_bits - 1)
        # Real-valued weights (updated during training)
        self.weight = nn.Parameter(torch.randn(out_features, in_features))

    def ste_binarize(self, w):
        """Binarization using Straight-Through Estimator"""
        # Forward pass: apply sign function
        # Backward pass: pass gradients through as-is (STE)
        w_bin = w.sign()
        # STE: block sign's gradient with detach(), maintain original w's gradient
        return w + (w_bin - w).detach()

    def activation_quant(self, x):
        """Activation absmax quantization"""
        gamma = x.abs().max()
        x_quant = torch.clamp(x * self.Qb / (gamma + 1e-5), -self.Qb, self.Qb - 1)
        return x_quant, gamma

    def forward(self, x):
        # Weight binarization + scaling factor
        w_bin = self.ste_binarize(self.weight)
        alpha = self.weight.abs().mean()

        # Activation quantization
        x_quant, gamma = self.activation_quant(x)

        # Integer matrix operations (addition/subtraction instead of multiplication)
        output = F.linear(x_quant, w_bin)

        # Dequantization: restore scale
        output = output * (alpha * gamma) / self.Qb
        return output

The Role of Straight-Through Estimator (STE)

The Sign function has a gradient of 0 at almost every point (it is undefined at the origin). Since backpropagation is impossible in this state, the Straight-Through Estimator (STE) proposed by Bengio et al. (2013) is used. The core of STE is to apply the Sign function during the forward pass, but during the backward pass, pass gradients through as if the Sign function were the identity function. Mathematically, the forward pass is w_bin = sign(w), and the backward pass directly propagates the gradient as dL/dw = dL/dw_bin.

An intuitive understanding of why this approximation works is as follows. When the real-valued weight w is sufficiently large in the positive direction, sign(w) = +1 is already the correct value, so no update is needed. When w is near 0, that is the decision boundary of sign, and in this region, STE's gradient estimation is most inaccurate. However, as training progresses, weights gradually converge toward +-1, so overall training stability is maintained.

Scaling Laws and Initial Results

BitNet v1 conducted experiments across model sizes from 125M to 30B. A notable finding is that 1-bit models follow scaling laws similar to FP16 models. As model size increases, perplexity decreases according to a power law, and above a certain size (approximately 6.7B), the performance gap with FP16 models narrows rapidly. However, in v1, there was still a performance gap compared to FP16 at the same parameter count.

BitNet b1.58: The Innovation of Ternary Weights

The Power of {-1, 0, +1}

BitNet b1.58 ("The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits"), published in February 2024, was the true turning point for BitNet. The key change was expanding weights from {-1, +1} to {-1, 0, +1}. The name "1.58 bits" comes from log2(3) approximately equal to 1.585. It means that the information required to represent a single ternary weight is about 1.58 bits.

To understand why the introduction of 0 is decisive, we need to look at it from the perspective of matrix operations. Positions where the weight is 0 can skip the computation entirely, effectively encoding explicit sparsity into the weights. This serves as feature filtering, allowing the model to learn which input channels to ignore at each neuron. BitNet v1's binary weights had to include all input channels, but ternary weights enable selective exclusion.

Absmean Quantization Function

BitNet b1.58's weight quantization uses the absmean function.

import torch
import torch.nn as nn
import torch.nn.functional as F

def weight_quant_ternary(w):
    """BitNet b1.58 ternary weight quantization (absmean-based)"""
    # Scaling factor: mean of absolute values of weights
    gamma = w.abs().mean()
    # Scale, round, and clamp to {-1, 0, +1}
    w_scaled = w / (gamma + 1e-5)
    w_ternary = torch.clamp(torch.round(w_scaled), -1, 1)
    # STE: forward uses quantized values, backward uses original gradients
    return w + (w_ternary - w).detach(), gamma

class BitLinear_b158(nn.Module):
    """BitNet b1.58 BitLinear implementation"""
    def __init__(self, in_features, out_features, activation_bits=8):
        super().__init__()
        self.weight = nn.Parameter(torch.randn(out_features, in_features) * 0.02)
        self.activation_bits = activation_bits
        self.Qb = 2 ** (activation_bits - 1)
        # Apply RMSNorm before activation
        self.norm = nn.RMSNorm(in_features)

    def activation_quant(self, x):
        """Activation absmax quantization (per-token)"""
        gamma = x.abs().max(dim=-1, keepdim=True).values
        x_q = torch.clamp(
            torch.round(x * self.Qb / (gamma + 1e-5)),
            -self.Qb, self.Qb - 1
        )
        return x_q, gamma

    def forward(self, x):
        # Activation normalization
        x = self.norm(x)

        # Ternary weight quantization
        w_q, w_scale = weight_quant_ternary(self.weight)

        # Activation quantization
        x_q, x_scale = self.activation_quant(x)

        # Integer operations: multiplication replaced with addition/subtraction/skip
        output = F.linear(x_q, w_q)

        # Dequantization
        output = output * (w_scale * x_scale) / self.Qb
        return output

Why Matching FP16 Performance Is Possible

The most surprising result of BitNet b1.58 is achieving perplexity comparable to FP16 Transformers at the 3B parameter scale. The reasons this is possible can be summarized as follows.

First, the introduction of 0 increases expressiveness. The transition from binary (2 states) to ternary (3 states) represents approximately a 58% increase in information capacity, from 1.0 bit to 1.58 bits. Second, the model adapts to quantization error during training. Since QAT is used, weight distributions converge to forms optimized for ternary representation. Third, the application of RMSNorm stabilizes activation distributions, reducing quantization error. Fourth, per-token activation quantization applies optimal scales for each token, maximizing dynamic range.

BitNet a4.8: Hybrid Quantization Strategy

Combining 4-Bit Activations with 1-Bit Weights

BitNet a4.8 ("BitNet a4.8: 4-bit Activations for 1-bit LLMs") is a follow-up study focusing on activation quantization. In BitNet b1.58, weights were extremely compressed to 1.58-bit, but activations were still maintained as 8-bit integers. a4.8 proposes a hybrid quantization technique that reduces activations to 4-bit while maintaining performance.

The key observation is that Transformer activation distributions are not uniform. Extremely large values (outliers) concentrate in certain channels, and quantizing these outliers to low bits causes severe information loss. a4.8 introduces two techniques to address this.

First, Sparsification and Decomposition. A certain percentage of top values from the activation tensor are separated and processed at higher precision (8-bit), while the rest are quantized to 4-bit. Second, per-channel scaling is applied to individually optimize the dynamic range of each channel.

The benefit of this hybrid approach is maximization of inference efficiency. Matrix operations with 1.58-bit weights and 4-bit activations can be performed with significantly fewer bit operations than conventional INT8xINT8, and high throughput can be achieved with custom kernels.

BitNet b1.58 2B4T: The First Open-Source Native 1-Bit LLM

A 2B Model Trained on 4 Trillion Tokens

BitNet b1.58 2B4T ("BitNet b1.58 2B4T Technical Report"), published in April 2025, is practically the most important milestone. Previous BitNet papers only reported research results without releasing model weights. 2B4T is the first open-source native 1-bit LLM with 2B (2 billion) parameters trained on 4T (4 trillion) tokens. Model weights can be downloaded directly from Hugging Face.

# BitNet b1.58 2B4T model download and bitnet.cpp inference environment setup
# 1. Clone repository
git clone --recursive https://github.com/microsoft/BitNet.git
cd BitNet

# 2. Install dependencies (conda environment recommended)
conda create -n bitnet python=3.11 -y
conda activate bitnet
pip install -r requirements.txt

# 3. Download model and build inference engine (all at once)
python setup_env.py --hf-repo microsoft/BitNet-b1.58-2B-4T-gguf \
    -q i2_s \
    --quant-embd

# 4. Run inference
python run_inference.py -m models/BitNet-b1.58-2B-4T-gguf/ggml-model-i2_s.gguf \
    -p "Microsoft Research recently released" \
    -n 128 \
    -t 4 \
    --temp 0.7

Comparison with Llama 3 and Qwen 2.5

The key achievement of 2B4T becomes apparent when compared to FP16 models of similar scale. The benchmark results reported in the paper are summarized below.

BitNet 2B4T, with 2B parameters, outperforms FP16 Llama 3.2 1B on most benchmarks, and while it does not match the 3B model, its model size is only 0.4GB, which is 15x smaller. This is a revolutionary result from a memory efficiency perspective, and the fact that it shows higher performance than Llama 3.2 1B (2GB) while using 5x less memory proves its practical value.

bitnet.cpp Inference Framework

Architecture Overview

bitnet.cpp is an inference engine optimized for 1-bit LLMs, built on the llama.cpp framework. Unlike general quantized models (GPTQ, AWQ), it provides dedicated kernels for ternary weights, performing inference without multiplication. The core components are as follows.

The I2_S (2-bit Integer, Signed) quantization format encodes ternary weights {-1, 0, +1} in 2 bits. Each weight is mapped to 00 (-1), 01 (0), 10 (+1), and 16 weights can be packed into a single 32-bit register.

The TL1 (Ternary Lookup 1) and TL2 (Ternary Lookup 2) kernels are matrix-vector product implementations specialized for ternary weights. TL1 uses sequential lookup, while TL2 uses parallel lookup that processes 2 weights simultaneously.

Internal Operation of I2_S Kernels

The core idea of the TL2 kernel is to combine 2 ternary weights (2 bits x 2 = 4 bits) into a single index and reference a lookup table. The combination of two ternary values gives 3x3 = 9 possibilities, which can be represented with a 4-bit index.

import numpy as np

def tl2_lookup_simulation(activations, ternary_weights_packed):
    """TL2 lookup table-based ternary matrix-vector product simulation"""
    # Pre-compute dot product results for all 9 possible combinations
    # of two consecutive weight pairs (w0, w1) with activations (a0, a1)
    # Replace w0*a0 + w1*a1 with lookup
    #
    # Encoding: w=(-1,0,1) -> (0,1,2), combination idx = w0_enc * 3 + w1_enc
    # idx=0: (-1,-1) -> -(a0+a1)
    # idx=1: (-1, 0) -> -a0
    # idx=2: (-1,+1) -> -a0+a1
    # idx=3: ( 0,-1) -> -a1
    # idx=4: ( 0, 0) -> 0
    # idx=5: ( 0,+1) -> a1
    # idx=6: (+1,-1) -> a0-a1
    # idx=7: (+1, 0) -> a0
    # idx=8: (+1,+1) -> a0+a1

    n = len(activations)
    result = 0
    for i in range(0, n, 2):
        a0, a1 = activations[i], activations[i+1]
        # Create lookup table (loaded into SIMD registers in actual implementation)
        lut = [
            -(a0 + a1), -a0, -a0 + a1,
            -a1, 0, a1,
            a0 - a1, a0, a0 + a1
        ]
        # Extract combination from packed index
        idx = ternary_weights_packed[i // 2]  # 0~8
        result += lut[idx]
    return result

# Verification
np.random.seed(42)
acts = np.random.randn(8).astype(np.float32)
# Ternary weights: [-1, 1, 0, 1, -1, 0, 1, -1]
weights = np.array([-1, 1, 0, 1, -1, 0, 1, -1])
# Generate packed indices
packed = []
for i in range(0, 8, 2):
    w0_enc = weights[i] + 1  # {-1,0,1} -> {0,1,2}
    w1_enc = weights[i+1] + 1
    packed.append(w0_enc * 3 + w1_enc)

ref = np.dot(acts, weights)
tl2 = tl2_lookup_simulation(acts, packed)
print(f"Reference dot product: {ref:.6f}")
print(f"TL2 lookup result:     {tl2:.6f}")
print(f"Match: {np.isclose(ref, tl2)}")

ARM and x86 Platform Optimizations

bitnet.cpp includes SIMD optimizations specialized for ARM (NEON/SVE) and x86 (AVX2/AVX-512) platforms. On ARM NEON, 16 INT8 activations are loaded into 128-bit registers, and the TBL instruction is used to perform direct lookups using ternary weight indices. On x86 AVX2, 256-bit registers are utilized to process 32 INT8 activations in parallel.

The benchmark script for performance profiling of bitnet.cpp is as follows.

# bitnet.cpp performance benchmark execution
cd BitNet

# Single-thread benchmark
python utils/benchmark.py \
    -m models/BitNet-b1.58-2B-4T-gguf/ggml-model-i2_s.gguf \
    -n 512 \
    -p 256 \
    --threads 1

# Multi-thread benchmark (matched to physical core count)
python utils/benchmark.py \
    -m models/BitNet-b1.58-2B-4T-gguf/ggml-model-i2_s.gguf \
    -n 512 \
    -p 256 \
    --threads 4

# Performance measurement across various prompt lengths
for prompt_len in 64 128 256 512 1024; do
    echo "=== Prompt length: $prompt_len ==="
    python utils/benchmark.py \
        -m models/BitNet-b1.58-2B-4T-gguf/ggml-model-i2_s.gguf \
        -n 128 \
        -p $prompt_len \
        --threads 4
done

Performance Comparison: BitNet vs FP16 vs GPTQ vs AWQ

Inference Speed and Memory Comparison

The following table compares the performance of various quantization methods with BitNet. The comparison targets are models of approximately 3B parameter scale, and inference was measured on the same hardware.

Key Analysis

There are several noteworthy points in this table. First, BitNet shows overwhelming speed in CPU inference. It is approximately 1.9x faster than GGUF Q4_K_M, which is the effect of multiplication elimination and ternary-specialized kernels. Second, the model size is 4.5x smaller than GPTQ 4-bit. This is a decisive advantage for edge device and mobile deployment. Third, energy efficiency is approximately 9x better than FP16. The elimination of multiplication operations is the key factor in energy savings.

However, on knowledge-intensive benchmarks such as MMLU, there is a performance gap compared to FP16 models with the same parameter count. This is a limitation of information density per parameter and must be compensated with larger BitNet models. PTQ methods (GPTQ, AWQ) preserve the knowledge of already-trained FP16 models as much as possible, so they maintain higher accuracy on a per-parameter basis.

Practical Applications and Deployment

Edge Device Deployment

BitNet's most promising application area is LLM inference on edge devices. The 0.4GB model size can be loaded on smartphones, IoT devices, Raspberry Pi, and other platforms, and multiplication-free inference is suitable for battery-sensitive mobile environments.

# Simple text generation pipeline example using BitNet model
# (Use bitnet.cpp's C++ API for actual deployment)
import subprocess
import json
import sys

class BitNetInference:
    """bitnet.cpp-based inference wrapper class"""
    def __init__(self, model_path, n_threads=4):
        self.model_path = model_path
        self.n_threads = n_threads
        self.binary = "./build/bin/llama-cli"  # bitnet.cpp build binary

    def generate(self, prompt, max_tokens=128, temperature=0.7, top_p=0.9):
        """Text generation"""
        cmd = [
            self.binary,
            "-m", self.model_path,
            "-p", prompt,
            "-n", str(max_tokens),
            "-t", str(self.n_threads),
            "--temp", str(temperature),
            "--top-p", str(top_p),
            "--no-display-prompt"
        ]
        try:
            result = subprocess.run(
                cmd,
                capture_output=True,
                text=True,
                timeout=120
            )
            if result.returncode != 0:
                raise RuntimeError(f"Inference failed: {result.stderr}")
            return result.stdout.strip()
        except subprocess.TimeoutExpired:
            raise TimeoutError("Inference timed out after 120 seconds")

    def benchmark(self, prompt_lengths=[64, 128, 256, 512]):
        """Performance measurement across various prompt lengths"""
        results = {}
        for length in prompt_lengths:
            prompt = "A " * length  # Dummy prompt
            cmd = [
                self.binary,
                "-m", self.model_path,
                "-p", prompt,
                "-n", "1",  # Generate 1 token only to measure prefill speed
                "-t", str(self.n_threads),
                "--no-display-prompt"
            ]
            result = subprocess.run(cmd, capture_output=True, text=True)
            # Parse performance metrics from stderr
            for line in result.stderr.split('\n'):
                if 'eval time' in line:
                    # Extract tokens/sec
                    parts = line.split()
                    for i, p in enumerate(parts):
                        if p == 'token/s)':
                            tok_per_sec = float(parts[i-1].strip('('))
                            results[length] = tok_per_sec
        return results

# Usage example
if __name__ == "__main__":
    model_path = "models/BitNet-b1.58-2B-4T-gguf/ggml-model-i2_s.gguf"
    engine = BitNetInference(model_path, n_threads=4)

    # Text generation
    output = engine.generate(
        "The key advantages of 1-bit LLMs are:",
        max_tokens=200,
        temperature=0.8
    )
    print(output)

Mobile Inference Scenarios

There are several considerations when deploying BitNet on mobile devices. The ARM processor's NEON instruction set is the baseline, and optimal performance is achieved on Apple Silicon (M1/M2/M3/M4) or Qualcomm Snapdragon's ARM cores. In mobile environments with limited memory bandwidth, the 0.4GB model can achieve sufficiently fast inference even with LPDDR5 bandwidth. A generation speed of approximately 20-30 tokens per second can be achieved, which is sufficient for real-time conversational applications.

Limitations and Considerations

The Reality of Training Costs

BitNet's biggest limitation is training cost. Since it uses QAT, the model must be trained from scratch, and existing FP16 models cannot be simply converted. BitNet b1.58 2B4T used 4 trillion tokens of training data, which requires substantial GPU time. The only publicly released model to date is at the 2B scale, and models of 7B or larger have not yet been announced. Since training large-scale models requires thousands of GPU-hours, it is realistically difficult for individuals or small teams to train their own BitNet models.

Limited Model Sizes and Ecosystem

The main constraints of the current BitNet ecosystem are summarized as follows.

• Only one model at the 2B scale has been released. Models at 7B, 13B, or 70B scale do not yet exist.
• Fine-tuning tools are not mature. Research is ongoing on whether parameter-efficient fine-tuning techniques like LoRA can be applied to ternary weights.
• GPU inference optimization is insufficient. bitnet.cpp is optimized for CPU inference, and GPU kernels are still in early development stages.
• Training frameworks are limited. PyTorch-based training code has been released, but integration with Megatron-LM or DeepSpeed is not complete.
• Multimodal extension has not been validated. Research applying ternary weights to Vision Transformers or Audio models is in early stages.

Accuracy Warning: Risky Use Cases

BitNet 2B4T shows respectable performance on general benchmarks, but caution is needed for certain tasks. Significant performance degradation compared to FP16 models of similar size has been observed in mathematical reasoning (GSM8K, MATH). In code generation (HumanEval), precise syntax generation capabilities may also be lacking. For multilingual tasks, non-English language performance is limited due to English bias in the training data. Use in safety-critical applications (medical, legal, financial) is not recommended without thorough validation.

Failure Cases and Troubleshooting

Case 1: Build Failure - CMake Version Mismatch

The most common issue when building bitnet.cpp is the CMake version requirement.

# Problem: Build failure when CMake version is under 3.22
# Error message:
# CMake Error at CMakeLists.txt:1:
#   CMake 3.22 or higher is required. You are running version 3.16.3

# Solution 1: Upgrade CMake (Ubuntu)
sudo apt remove cmake
pip install cmake --upgrade
# or
sudo snap install cmake --classic

# Solution 2: Install CMake in conda environment
conda install -c conda-forge cmake>=3.22

# Solution 3: Build from source
wget https://github.com/Kitware/CMake/releases/download/v3.28.3/cmake-3.28.3.tar.gz
tar xzf cmake-3.28.3.tar.gz
cd cmake-3.28.3
./bootstrap && make -j$(nproc) && sudo make install

# Retry build
cd BitNet
python setup_env.py --hf-repo microsoft/BitNet-b1.58-2B-4T-gguf \
    -q i2_s --quant-embd

Case 2: Segmentation Fault During Inference

On certain ARM processors, NEON-optimized kernels can cause segmentation faults due to unaligned memory access. This issue primarily occurs on older ARM chipsets (ARMv7 or below) or when using non-standard memory allocators.

The recovery procedure is as follows. First, check AVX/NEON support. On x86, check with lscpu | grep avx; on ARM, check for the neon flag in /proc/cpuinfo. If SIMD is not supported, add the -DBITNET_NO_SIMD=ON flag during build to use fallback kernels. In this case, performance will be degraded but operation will be stable.

Case 3: Out-of-Memory (OOM) Errors

Although the model size is a small 0.4GB, additional memory is needed for KV cache and activation buffers during inference. OOM can occur when processing long sequences (over 4096 tokens) in environments with less than 2GB of system RAM.

Mitigation strategies include limiting context length (--ctx-size 2048), reducing batch size (--batch-size 256), or managing memory by disabling mmap (--no-mmap).

Case 4: Incorrect Quantization Format Selection

If a BitNet model is quantized with general GGUF quantization (Q4_K_M, etc.) instead of the I2_S format, the already 1.58-bit weights get "quantized" to 4-bit, causing unnecessary bloat and performance degradation. The dedicated i2_s format must be used.

Case 5: Divergence Issues During Training

The most common problem when training BitNet directly is divergence at the beginning of training. STE-based training is more unstable than FP16 training and is sensitive to the learning rate. Using the same learning rate as typical FP16 training (e.g., 3e-4) may cause divergence. Recommendations include lowering the learning rate to 1e-4 or below, setting the warmup ratio to 5-10%, and maintaining a sufficiently large batch size (512 or above). Gradient clipping (max_norm=1.0) also helps with stability.

Future Prospects

NPU Support and Hardware Optimization

BitNet's long-term vision lies in dedicated hardware support. Since matrix operations with ternary weights consist essentially of only additions and subtractions, dedicated NPUs (Neural Processing Units) without multipliers can be designed. Since multipliers are a major source of chip area and power consumption, removing them improves energy efficiency by orders of magnitude.

Chip manufacturers such as Intel, Qualcomm, and Apple are strengthening low-bit computation support in their NPUs, and ultra-low-bit models at the BitNet level naturally align with these hardware trends. In particular, Apple's Neural Engine is already optimized for INT8 operations, and if INT2-level support is added, BitNet inference could be further accelerated.

Toward Sustainable AI

As the environmental impact of AI emerges as a major topic of discussion, the energy efficiency improvements presented by BitNet can serve as one pillar of sustainable AI development. An energy efficiency improvement of over 10x compared to FP16 can dramatically reduce data center power consumption, and lowering cloud dependency through edge deployment also contributes to carbon footprint reduction.

Possibilities for Model Size Scaling

What performance BitNet models will show when scaled from the current 2B to 7B, 13B, and 70B is the most anticipated research direction. As confirmed in BitNet v1's scaling law analysis, the performance gap with FP16 narrows as model size increases. If a 70B-scale BitNet emerges, its model size would be approximately 14GB (compared to 140GB for FP16), capable of running on a single GPU or high-end CPU, and its performance is expected to approach that of FP16 70B.

Additionally, the combination of Mixture of Experts (MoE) and BitNet is an intriguing direction. When ternary weight sparsity combines with MoE's conditional computation, extremely large model capacity can be utilized with extremely low computation. For example, one can envision a BitNet MoE architecture with 100B total parameters, where only 6B parameters are active per token and active memory is only about 1.2GB.

Training Efficiency Improvements

Improving BitNet's training efficiency is also an active research area. Currently, the QAT approach requires training from scratch, which is costly, but Post-Training Ternarization techniques that convert from a pre-trained FP16 model to ternary weights are being researched. If this becomes practical, it would enable leveraging existing FP16 model assets while gaining BitNet's inference efficiency, significantly lowering the adoption barrier for BitNet.

References

1. BitNet v1 Paper: Wang et al., "BitNet: Scaling 1-bit Transformers for Large Language Models", 2023. https://arxiv.org/abs/2310.11453

2. BitNet b1.58 Paper: Ma et al., "The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits", 2024. https://arxiv.org/abs/2402.17764

3. BitNet b1.58 2B4T Technical Report: Microsoft Research, "BitNet b1.58 2B4T Technical Report", 2025. https://arxiv.org/abs/2504.12285

4. bitnet.cpp Official Repository: Microsoft, BitNet Inference Framework. https://github.com/microsoft/BitNet

5. BitNet b1.58 2B4T Hugging Face Model: Microsoft, Open-Source Native 1-bit LLM. https://huggingface.co/microsoft/bitnet-b1.58-2B-4T

6. bitnet.cpp Paper: Zhu et al., "bitnet.cpp: Efficient Edge Inference for Ternary LLMs", 2024. https://arxiv.org/abs/2410.16144

7. Bengio et al., 2013: Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. The original paper for STE (Straight-Through Estimator).

8. GPTQ: Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers", 2023. Post-Training Quantization comparison baseline.

Quiz