Skip to content
BranchesTags

Repository files navigation

⚡ OpenNLP GPU Extension

Third-party GPU acceleration layer for Apache OpenNLP - transparent 2–5× speedups with NVIDIA CUDA, AMD ROCm, Intel OpenCL, and intelligent CPU fallback.

License GitHub Stars GitHub Forks Last Commit Repo Size Issues Java OpenNLP Maven JitPack Build Code Size PRs Welcome Docs

Important

This is an independent, third-party GPU acceleration extension for Apache OpenNLP and is not officially endorsed or maintained by the Apache Software Foundation.

Table of Contents

🎯 Overview

What Is This Project?

OpenNLP GPU Extension is an independent third-party hardware acceleration layer that transparently routes Apache OpenNLP compute-intensive matrix operations to GPU hardware, delivering 2–5× throughput improvements for NLP workloads while maintaining 100% API compatibility with all standard OpenNLP model interfaces.

The extension operates as a drop-in decorator around existing OpenNLP models. No changes to training pipelines, serialized model files, or application calling code are required. When GPU hardware is present and configured, dense matrix operations (GEMM, softmax, TF-IDF, cosine similarity) execute on GPU kernels; when no GPU is detected, a numerically-identical pure-Java implementation silently handles all operations.

Why OpenNLP Was Chosen

Apache OpenNLP is the dominant production-grade NLP framework in the Java/JVM ecosystem. Enterprises standardized on Java cannot easily switch to Python-native frameworks like spaCy or Hugging Face without introducing cross-language inter-process calls, retraining costs, and operational complexity. OpenNLP was specifically chosen as the GPU acceleration target because:

Reason Detail
Java-native Integrates directly into Spring Boot, Jakarta EE, and enterprise JVM stacks without subprocess overhead
Stable API contracts MaxentModel, TokenizerModel, and NameFinderME interfaces are stable across releases; the decorator pattern is reliable
Apache governance Apache License 2.0; Apache Software Foundation oversight ensures long-term stability and commercial compatibility
Lightweight models Serialized .bin model files are compact, versioned, and deployable without a framework runtime on the target server
Extensibility Interface-based design means GpuMaxentModel implements MaxentModel with no changes to model loading or application logic
Active maintenance OpenNLP 2.5.8 fixes SentenceDetector abbreviation handling (OPENNLP-1809/1810/1811) and updates ONNX Runtime to 1.24.3

Why GPU Acceleration for NLP?

Traditional NLP workloads are dominated by dense matrix operations that run sequentially on single CPU threads:

  • Maximum Entropy evaluation: dot products between high-dimensional feature vectors and weight matrices (thousands of features × hundreds of outcomes per document)
  • Named Entity Recognition: per-token matrix multiplications across sequence windows in every sentence
  • TF-IDF document scoring: vocabulary-scale sparse-to-dense matrix operations across entire corpora
  • Cosine similarity search: pairwise distance calculations that scale O(N²) with corpus size

GPUs execute thousands of these operations simultaneously. A modern GPU with 10,000+ CUDA cores processes a 512×512 matrix multiplication as a single parallel batch that would require thousands of sequential CPU instructions. The result: the same per-document accuracy at a fraction of the wall-clock time, directly translating to smaller SLA requirements or larger processing windows under the same compute budget.

How this helps (practical + technical)

Think of CPU vs GPU like this: a CPU is a small team of expert workers that are great at varied tasks and branch-heavy logic, while a GPU is a very large team that performs the same numeric operation on many values at once. NLP feature and scoring pipelines contain many repeatable numeric operations (dot products, matrix multiplications, normalization), so they map naturally to GPU execution.

Concretely, acceleration comes from three things:

  1. Data parallelism: many tokens/documents are processed at the same time.
  2. Operation parallelism: many multiply-add operations of the same formula run concurrently.
  3. Higher arithmetic throughput: GPUs are built to sustain very high floating-point throughput for vector/matrix math.
Work item CPU-style execution GPU-style execution Why GPU usually wins here
Dot products for MaxEnt outcomes Iterate outcome-by-outcome, feature-by-feature Thousands of multiply-add lanes run in parallel Same arithmetic pattern repeated across large vectors
NER token window scoring Sequence loop over tokens/windows Multiple token windows scored concurrently Independent token-window math can be batched
TF-IDF matrix construction Build term weights mostly row-by-row Batch rows/blocks and compute vectorized weights Regular memory access + repeated formulas
Cosine similarity over corpus Pairwise loops become expensive quickly Parallel pair blocks computed together Large $O(N^2)$ pair sets are highly parallelizable

Note

GPU does not make algorithms “more accurate”; it makes the same algorithm run faster when the workload is parallelizable.

What GPU is doing that CPU typically cannot do as efficiently is not “new math,” but massively concurrent execution density for identical operations. CPU cores are fewer and optimized for general-purpose control flow; GPU cores are far more numerous and optimized for throughput on homogeneous numeric kernels.

Tip

Parallelization helps most when you have enough work per launch (batches, larger matrices, many documents). Very tiny workloads may see little gain because dispatch/transfer overhead can dominate.

Who this is for:

  • Java NLP engineers processing high-volume batch workloads (10K+ documents/hour) who need lower latency without framework migration
  • MLOps teams deploying OpenNLP on GPU-enabled cloud instances (AWS g4dn/p3, GCP a2, Azure NCv3)
  • Researchers benchmarking GPU acceleration for classical NLP algorithms
  • Organizations with existing OpenNLP deployments who need GPU benefits without retraining models or changing application code

(back to top ↑)

🆕 Recent Updates (What Changed, Why, Benefits)

This section summarizes the latest implementation and documentation changes so contributors and adopters can quickly understand what is new and why it matters.

What was added

Change What it is
Unified TF-IDF core A single TfIdfAlgorithms path now drives CPU/OpenCL/CUDA/ROCm TF-IDF behavior.
Advanced vectorization controls N-gram blending, BM25/sublinear weighting, smoothing strategies, DF cutoffs, and class-balanced scoring options.
Persisted reproducibility state Versioned VocabularyState with explicit compatibility policy (V2 current + V1 migration defaults).
Dense vector compression Optional FLOAT16 and INT8 persistence paths (with load-time reconstruction).
Guardrail tests Cross-backend parity plus latency-bound assertions to detect regressions early.
Expanded documentation Decision matrix, algorithm/formula rationale, collapsible API reference, additional Mermaid diagrams, and research references.
Mermaid hardening Diagram labels were adjusted to parser-safe text to improve GitHub rendering reliability.

Why these were added

  • To reduce drift between backend implementations and keep feature behavior deterministic.
  • To improve retrieval/classification quality for real-world corpora with mixed term distributions.
  • To make training/inference pipelines reproducible and auditable through versioned state.
  • To lower storage and memory pressure in production vector caches.
  • To give operators practical rollout guidance (when to use GPU, when not to, and how to validate safely).

Practical benefits

Benefit area Outcome
Quality Better relevance and class-sensitive feature retention through richer scoring options.
Reliability Deterministic shared logic + benchmark guardrails reduce backend regression risk.
Reproducibility Versioned vocabulary/DF metadata keeps inference aligned with training assumptions.
Efficiency Compression options reduce artifact size and memory footprint at scale.
Adoption speed Richer README guidance reduces onboarding time and misconfiguration risk.

Important

These updates focus on making existing OpenNLP pipelines faster and safer to operate, without forcing framework migration or model-format changes.

(back to top ↑)

✨ Key Features

Icon Feature Description Impact Status
⚡ GPU-Accelerated Matrix Ops GEMM, transpose, and activation functions dispatched to GPU kernels 2–5× throughput ✅ Stable
🔄 Auto CPU Fallback Silent, transparent fallback to pure-Java when GPU unavailable Zero downtime ✅ Stable
🎯 Drop-in API Compatibility GpuMaxentModel implements OpenNLP MaxentModel interface exactly No code changes ✅ Stable
🖥️ Multi-Backend CUDA 11+, ROCm 5+, OpenCL 1.2+, CPU (runtime-selected) Broad hardware support 🔄 In Progress
☁️ Cloud Accelerators AWS Inferentia and Google TPU providers with CPU fallback; Neuron/XLA bridges planned Cloud-native NLP 🔄 In Progress
📊 Performance Monitor Real-time thread-safe metrics, latency alerts, memory tracking Operational observability ✅ Stable
🔍 GPU Diagnostics CLI Standalone tool to probe drivers, SDKs, and runtime environment DevOps-friendly ✅ Stable
🧪 Extensive Test Suite 30+ test classes: unit, integration, stress, compatibility, benchmark High confidence ✅ Stable

Highlights:

  • 115 Java source files covering ML models (MaxEnt, Perceptron, Naive Bayes, Neural), GPU backends, monitoring, and tooling
  • Structured commenting on all core interfaces and compute classes: requirement, purpose, inputs, outputs, and failure-mode documentation
  • Java 21 LTS compilation target with full OpenNLP 2.5.8 API compatibility
  • Real backpropagation in GpuNeuralNetwork: chain-rule gradient descent, activation derivatives (sigmoid, tanh, ReLU, softmax, linear), with GPU-parallel batch inference via IntStream.parallel()
  • JOCL-based hardware detection: CudaUtil.isAvailable(), OpenCLUtil.isAvailable(), and RocmUtil.isAvailable() all enumerate real devices via JOCL with no placeholder returns
  • Zero stub methods: all public API methods have production implementations or documented CPU-fallback paths; no return new Object() or return false // Stub remain
  • Benchmarks against CpuComputeProvider reference implementation to validate numerical correctness

(back to top ↑)

🧭 Decision Matrix (When to Use / Not Use)

This section is intentionally practical: if you only read one part before rollout, read this one. It tells you where the extension shines, where it does not, and why.

Tip

If your workload is mostly batch inference or high-concurrency scoring, start with GPU enabled and benchmark. If your workload is tiny or latency-insensitive, keep CPU fallback as default and enable GPU only where it proves value.

Quick chooser by workload type

Workload pattern Recommended mode Why this mode works When to avoid
10K+ docs per hour, repeated model eval GPU primary + CPU fallback Kernel launch overhead is amortized; high parallelism wins If GPU memory is too small for your batch profile
Low-volume internal API, predictable load CPU default, GPU optional Simpler operations, less tuning overhead If strict p95 latency target is difficult to meet on CPU
Spiky traffic (bursts) GPU with bounded batch size Handles sudden parallel work better If queueing delay from oversized batches hurts latency
On-prem regulated workloads GPU on local servers No external inference calls required If operational team cannot support GPU driver lifecycle
Cost-focused cloud workload Mixed mode by endpoint Use GPU for heavy endpoints only If constant GPU idle time dominates bill

Backend comparison (what each one does differently)

Backend Strengths Trade-offs Best for Not ideal for
CPU fallback Most portable, easiest debugging, deterministic baseline Lower throughput at scale Local dev, CI, small workloads Large corpus scoring at tight SLAs
OpenCL Vendor-agnostic path across NVIDIA/AMD/Intel Capability can vary by driver stack Mixed hardware fleets Teams expecting one-click homogeneous behavior
CUDA Strong tooling/perf ecosystem on NVIDIA Vendor lock-in NVIDIA-heavy production Cross-vendor portability requirements
ROCm/HIP Native AMD acceleration path Stack maturity varies by distro/GPU AMD-centric environments Teams without ROCm ops experience
Cloud accelerators Elastic infrastructure options Runtime integration complexity Managed cloud NLP pipelines Strictly offline/on-prem environments

Selection logic in one diagram

flowchart TD
    A[Start Deployment Plan] --> B{Batch or Concurrency Heavy?}
    B -->|Yes| C[Enable GPU Path]
    B -->|No| D[Use CPU Fallback First]
    C --> E{Hardware Vendor Mix?}
    E -->|Mixed| F[Prefer OpenCL-first Strategy]
    E -->|Mostly NVIDIA| G[Prefer CUDA-first Strategy]
    E -->|Mostly AMD| H[Prefer ROCm-first Strategy]
    F --> I[Benchmark and tune batch size]
    G --> I
    H --> I
    D --> J[Track latency and throughput baseline]
    J --> K{SLA pressure?}
    K -->|Yes| C
    K -->|No| L[Stay CPU default]
Loading

Rollout checklist by stage

Stage Goal Concrete checks Exit criteria
Baseline Understand current CPU behavior Measure throughput, p95 latency, memory Stable baseline report captured
Enablement Turn on GPU path safely GpuDiagnostics pass, fallback enabled No functional regressions
Optimization Increase efficiency Tune batch size, memory pool, warm-up Throughput and/or p95 improved
Guardrails Prevent silent drift Parity tests and latency thresholds CI catches regression before release

Important

Always keep CPU fallback enabled in production. This gives you graceful degradation instead of incident-level outages when hardware, drivers, or native dependencies change.

(back to top ↑)

💡 Use Cases & Applications

Real-World Application Scenarios

1. High-Volume Batch Document Processing

Legal discovery, content moderation, financial document analysis, and compliance scanning involve processing tens of thousands of documents per hour. GPU batch sizing:

  • Stacks 64–256 document feature vectors per kernel launch
  • Processes each batch in a single GPU call replacing hundreds of sequential CPU invocations
  • Sustains linear throughput scaling as document volume grows

2. Real-Time NLP APIs

Low-latency REST endpoints for text classification, sentiment analysis, or entity detection:

  • Sub-50ms inference on complex MaxEnt models under concurrent load
  • Reduced p99 latency outliers eliminated through GPU parallel evaluation
  • Handle burst traffic without horizontal scaling

3. Enterprise Document Intelligence

ETL pipelines for CRM, HR, compliance, and knowledge management systems:

  • GPU-accelerated TF-IDF across large document corpora
  • Batch cosine similarity for document deduplication and clustering
  • Faster named entity extraction across multilingual document sets

4. Clinical NLP & Healthcare

On-premises clinical text processing (EHR structuring, ICD coding, clinical concept extraction) where:

  • Privacy constraints prevent cloud API calls; a local GPU server is required
  • High-accuracy MaxEnt models are used for medical term classification
  • Throughput matters for overnight batch processing of patient notes

5. Research & Academic Benchmarking

Researchers using OpenNLP as a classical NLP baseline can:

  • Measure GPU vs. CPU throughput for traditional probabilistic models
  • Compare accuracy/latency tradeoffs across CUDA, ROCm, and OpenCL backends
  • Prototype GPU-accelerated feature engineering before committing to deep learning pipelines

6. Cloud GPU Cost Optimization

Teams on GPU cloud instances can:

  • Maximize GPU utilization by running OpenNLP inference alongside vision or audio model serving
  • Use spot/preemptible instances cost-effectively due to pipelined batch processing
  • Scale inference horizontally with bit-identical results across CPU fallback and GPU nodes

Platform Use Case Matrix

Industry Workload OpenNLP Component GPU Benefit
Legal Contract entity extraction GpuNerModel Batch throughput on large corpora
Finance Earnings call sentiment GpuMaxentModel Sub-100ms per-document scoring
Healthcare Clinical concept extraction Custom MaxEnt Privacy-safe on-prem GPU inference
E-commerce Query intent classification GpuMaxentModel Low-latency real-time API
Media Article topic classification MaxEnt ensemble GPU batch for trending topic detection
HR / Recruitment Resume skill extraction GpuNerModel High-volume batch processing
Compliance Document classification audit GpuPerceptronModel Reproducible GPU-verified results
News / Search Multilingual document dedup TF-IDF + cosine similarity O(N²) → GPU-parallel similarity

(back to top ↑)

🏗️ Architecture

flowchart TD
    A[NLP Application] --> B[OpenNlpGpuAdapter]
    B --> C{"GpuConfig<br/>gpu.available?"}
    C -->|GPU Available| D[GpuComputeProvider]
    C -->|No GPU| E[CpuComputeProvider]
    D --> F{Backend Selection}
    F -->|NVIDIA| G["CUDA Kernels<br/>JNI Bridge"]
    F -->|AMD| H[ROCm / HIP]
    F -->|Any Vendor| I[OpenCL / JOCL 2.0.6]
    F -->|Cloud| J["AWS Inferentia<br/>Google TPU"]
    G & H & I & J --> K["MatrixOperation<br/>Interface"]
    E --> K
    K --> L["Result to OpenNLP<br/>MaxentModel.eval"]
    L --> M["GpuPerformanceMonitor<br/>Metrics & Alerts"]
Loading

Component responsibilities:

Component Package Role
OpenNlpGpuAdapter integration Entry point; selects provider; wraps OpenNLP models
ComputeProvider common Hardware-agnostic interface for all compute backends
GpuConfig common Configuration value object (GPU flag, pool size, batch size)
CpuComputeProvider compute Pure-Java reference implementation; always available
GpuComputeProvider compute OpenCL-backed provider with CPU fallback delegation
OperationFactory compute Factory for selecting concrete MatrixOperation implementations
GpuMaxentModel ml.maxent Drop-in MaxentModel decorator with GPU dispatch
GpuPerformanceMonitor monitoring Thread-safe singleton metrics and alerting
GpuDiagnostics tools CLI tool for environment pre-flight checks

(back to top ↑)

⚙️ How the System Works Internally

At a high level, the extension acts as a scheduler and adapter. It does not replace OpenNLP models; it wraps them and routes expensive numeric operations to the most appropriate compute path.

Request lifecycle (runtime path)

flowchart LR
    A[Input Text / Context Features] --> B[OpenNLP Wrapper Model]
    B --> C[Feature Extraction Layer]
    C --> D[Compute Provider Selector]
    D --> E[GPU Backend Operation]
    D --> F[CPU Fallback Operation]
    E --> G[Result Aggregation]
    F --> G
    G --> H[Outcome Probabilities / Labels]
Loading

TF-IDF state lifecycle (train to inference)

stateDiagram-v2
    [*] --> BuildVocabulary
    BuildVocabulary --> ComputeDF
    ComputeDF --> ScoreTerms
    ScoreTerms --> PersistVocabularyState
    PersistVocabularyState --> LoadAtInference
    LoadAtInference --> VectorizeIncomingDocs
    VectorizeIncomingDocs --> [*]
Loading

Data contract boundaries

Layer Input Output Why this boundary exists
OpenNLP wrapper token/context arrays model-ready numeric features Preserve OpenNLP API compatibility
Feature extraction text or token stream sparse/dense vectors Keep algorithm changes isolated
Compute provider matrices/vectors transformed matrices/probabilities Swap hardware path without app changes
Monitoring operation timings/counters metrics and alerts Operational visibility and regression detection

Note

This separation is why the project can evolve algorithms (e.g., smoothing, DF cutoffs, BM25) without forcing application-level refactors.

(back to top ↑)

🔄 Usage Flow

sequenceDiagram
    participant App as "NLP Application"
    participant Adapter as "OpenNlpGpuAdapter"
    participant Factory as "ComputeProviderFactory"
    participant GPU as "GpuComputeProvider"
    participant Model as "GpuMaxentModel"
    participant Monitor as "GpuPerformanceMonitor"

    App->>Adapter: new OpenNlpGpuAdapter()
    Adapter->>Factory: selectProvider(GpuConfig)
    Factory-->>Adapter: GpuComputeProvider or CpuFallback
    App->>Model: new GpuMaxentModel(baseModel, config)
    Model->>GPU: initialize()
    GPU-->>Model: ready or silently falls back
    App->>Model: eval(context[])
    Model->>GPU: matrixMultiply / extractFeatures
    GPU-->>Model: double[] probabilities
    Model-->>App: probabilities
    Model->>Monitor: recordOperation(latencyNs, memoryMB)
    Monitor-->>App: alert if threshold exceeded
Loading

Step-by-step usage:

# 1. Clone
git clone https://github.com/hkevin01/opennlp-gpu.git
cd opennlp-gpu

# 2. Compile (skips native cmake build by default)
mvn clean compile

# 3. Run GPU diagnostics to check your environment
mvn exec:java -Dexec.mainClass=org.apache.opennlp.gpu.tools.GpuDiagnostics

# 4. Run tests
mvn test -Dtest=GpuTestSuite

(back to top ↑)

🛠️ Technology Stack

Technology Version Purpose Why Chosen Alternative
Apache OpenNLP 2.5.8 NLP model API contract Industry-standard Java NLP; stable API Stanford NLP, spaCy
Java 21 LTS Runtime and implementation LTS stability; virtual threads; modern records Kotlin, Scala
JOCL 2.0.6 OpenCL Java bindings Cross-vendor GPU without native CUDA lock-in LWJGL, pure JNA
SLF4J 2.0.17 Logging facade Framework-neutral; no log framework lock-in Log4j2, java.util.logging
JUnit 5 5.13.1 Testing framework Parameterized tests; extension model; parallel execution TestNG
CMake 4+ Native library build Cross-platform C++/CUDA build system Makefile, Meson
Maven 3.9+ Build and dependency management Industry standard; reproducible builds Gradle

(back to top ↑)

📐 Technical Specifications

GPU Architecture Support

GPU Family Architecture Min Compute / Version OpenCL Level Backend
NVIDIA Turing (RTX 20xx, T4) sm_75 CUDA 11+ 3.0 CUDA + OpenCL
NVIDIA Ampere (RTX 30xx, A100) sm_80 CUDA 11+ 3.0 CUDA + OpenCL
NVIDIA Ada Lovelace (RTX 40xx) sm_89 CUDA 12+ 3.0 CUDA + OpenCL
NVIDIA Hopper (H100, H200) sm_90 CUDA 12+ 3.0 CUDA + OpenCL
AMD RDNA2 (RX 6000 series) GFX1030 ROCm 5.0+ 2.0 ROCm / HIP
AMD RDNA3 (RX 7000 series) GFX1100 ROCm 5.5+ 2.0 ROCm / HIP
Intel Arc (A-series) Xe-HPG N/A 3.0 OpenCL via JOCL
Any OpenCL 1.2+ device N/A N/A 1.2 JOCL cross-vendor

System Requirements

Component Minimum Recommended
Java JDK 21 LTS 21 LTS or 26
Maven 3.9 3.9+
GPU VRAM 2 GB 8 GB+
JVM Heap 512 MB 2–4 GB
NVIDIA Driver 520.x 535.x+
CUDA Toolkit 11.0 12.0+
ROCm 5.0 5.5+
OpenCL ICD 1.2 3.0
CMake (native build only) 3.16 4.x

GPU Kernel Inventory

All kernels are implemented in CUDA C++ (kernels.cu), HIP/ROCm (kernels.cpp), and have equivalent pure-Java CPU reference implementations validated for numerical correctness to ≤1e-5 tolerance:

Kernel Dimensions Block / Tile Size Algorithm
matMulKernel M×K · K×N → M×N 16×16 shared-mem tiles Tiled SGEMM
softmaxKernel N-element vector 256 threads/block Numerically stable (subtract max)
tfidfKernel N docs × M terms 32×32 TF × log(N/df)
cosineSimilarityKernel N pairs × D dims 256 threads L2-normalized dot product
ngramExtractKernel N tokens × L window 128 threads/block Sliding-window n-gram

Performance Targets (FP32, Batch = 64)

Reference measurements on NVIDIA RTX 3080 (10 GB VRAM). Actual performance varies by GPU model, driver version, batch size, and input dimensions. CPU fallback is always available and numerically identical.

Operation CPU Reference (ms) GPU Target (ms) Target Speedup
MaxEnt eval: 1K features, 100 outcomes ~12 ~3 4×
Matrix multiply: 512×512 FP32 ~19 ~4 5×
Softmax: 10K elements ~2 <1 3×
TF-IDF: 10K docs × 5K terms ~900 ~190 4.7×
Cosine similarity: 1K pairs × 512 dims ~24 ~6 4×

Build Variants

Maven Profile Command Artifacts Hardware Required
Default (Java-only) mvn clean package JAR + CPU fallback None
Native CUDA mvn clean package -Pnative JAR + CUDA .so kernels CUDA Toolkit 11+
Native ROCm mvn clean package -Pnative -Drocm=true JAR + HIP .so kernels ROCm 5.0+
Test suite (CPU mode) mvn test -Dtest=GpuTestSuite Test results None

(back to top ↑)

📊 GPU Backend Distribution

pie title GPU Backend Support Coverage
    "OpenCL (JOCL cross-vendor)" : 45
    "CUDA (NVIDIA)" : 30
    "ROCm/HIP (AMD)" : 15
    "Cloud (Inferentia + TPU)" : 10
Loading
Backend Vendor Status Requirement
OpenCL via JOCL Any (NVIDIA, AMD, Intel) 🔄 JNI bridge in progress OpenCL 1.2+ ICD
CUDA via JNI NVIDIA 🔄 Native kernels in progress CUDA Toolkit 11+, driver
ROCm / HIP AMD 🔄 JOCL enumeration complete; HIP native kernels planned ROCm 5.0+, compatible GPU
AWS Inferentia Amazon 🔄 CPU fallback active; AWS Neuron SDK bridge planned Neuron SDK on inf1/inf2
Google TPU Google 🔄 CPU fallback active; XLA bridge planned TPU v3/v4 on GCP
CPU Fallback Any ✅ Production ready JVM only

Note

The CPU fallback (CpuComputeProvider) is fully production-ready and used as the numerical reference for all GPU kernel correctness tests. GPU backends are progressively integrated as the JNI bridge matures.

(back to top ↑)

🚀 Setup & Installation

Prerequisites

Requirement Minimum Recommended
Java JDK 21 21 LTS or 26
Maven 3.9 3.9+
GPU (optional) OpenCL 1.2+ CUDA 11+ or ROCm 5+
CMake (optional) 3.16 4.x (for native build)

Clone & Build

git clone https://github.com/hkevin01/opennlp-gpu.git
cd opennlp-gpu

# Standard build (Java only, no native GPU kernels)
mvn clean package

# Full native build (requires CUDA/ROCm/OpenCL headers)
mvn clean package -Pnative

Maven Dependency (via JitPack)

Tip

Use a tagged release (e.g. 1.0.0) for stable builds, or main-SNAPSHOT to track the latest commit on main.

Maven (pom.xml):

<repositories>
    <repository>
        <id>jitpack.io</id>
        <url>https://jitpack.io</url>
    </repository>
</repositories>

<dependencies>
    <!-- Apache OpenNLP -->
    <dependency>
        <groupId>org.apache.opennlp</groupId>
        <artifactId>opennlp-tools</artifactId>
        <version>2.5.8</version>
    </dependency>

    <!-- GPU Extension (tagged release) -->
    <dependency>
        <groupId>com.github.hkevin01</groupId>
        <artifactId>opennlp-gpu</artifactId>
        <version>1.0.0</version>
    </dependency>
</dependencies>

Gradle (build.gradle):

repositories {
    maven { url 'https://jitpack.io' }
}

dependencies {
    implementation 'org.apache.opennlp:opennlp-tools:2.5.8'
    implementation 'com.github.hkevin01:opennlp-gpu:1.0.0'
}

Gradle Kotlin (build.gradle.kts):

repositories {
    maven("https://jitpack.io")
}

dependencies {
    implementation("org.apache.opennlp:opennlp-tools:2.5.8")
    implementation("com.github.hkevin01:opennlp-gpu:1.0.0")
}

Environment Setup (GPU)

# Enable GPU detection (set to true when GPU hardware is present and drivers loaded)
export JAVA_TOOL_OPTIONS="-Dgpu.available=true -Dgpu.vendor=NVIDIA -Dgpu.device=RTX4090"

# Verify environment
mvn exec:java -Dexec.mainClass=org.apache.opennlp.gpu.tools.GpuDiagnostics

(back to top ↑)

⚡ Quick Start

import opennlp.tools.tokenize.TokenizerModel;
import org.apache.opennlp.gpu.common.GpuConfig;
import org.apache.opennlp.gpu.integration.OpenNlpGpuAdapter;
import org.apache.opennlp.gpu.ml.maxent.GpuMaxentModel;

// 1. Configure GPU
GpuConfig config = new GpuConfig();
config.setGpuEnabled(true);         // Enable GPU acceleration
config.setMemoryPoolSizeMB(512);    // Pre-allocate 512 MB GPU pool
config.setBatchSize(64);            // Process 64 samples per kernel launch

// 2. Create the GPU adapter (auto-selects best available backend)
OpenNlpGpuAdapter adapter = new OpenNlpGpuAdapter();

// 3. Wrap your existing OpenNLP MaxentModel
//    baseModel loaded normally from .bin file
GpuMaxentModel gpuModel = new GpuMaxentModel(baseModel, config);

// 4. Use exactly as you would the original model
double[] probabilities = gpuModel.eval(new String[]{"word", "suffix=ing", "prev=VBZ"});
String bestOutcome = gpuModel.getBestOutcome(probabilities);

// 5. Check runtime stats
System.out.println("Using GPU: " + gpuModel.isUsingGpu());
System.out.println("Speedup:   " + gpuModel.getSpeedupFactor() + "×");
gpuModel.cleanup(); // Release GPU resources

Tip

Set -Dgpu.available=true only after running GpuDiagnostics confirms your driver stack is complete. When this flag is absent or false, the extension runs identically correct in CPU mode.

(back to top ↑)

🔧 Core Capabilities

🧮 Matrix Operations

The MatrixOperation interface provides 20+ operations:

Category Methods Backend
BLAS-style multiply, add, subtract, transpose, scalarMultiply CPU ✅ / GPU 🔄
ML-specific dotProduct, vectorNorm, elementWiseMultiply, matrixVectorMultiply CPU ✅ / GPU 🔄
Activations sigmoid, tanh, relu, softmax (numerically stable) CPU ✅ / GPU 🔄
Statistics mean, variance, normalize CPU ✅ / GPU 🔄
Utility copyArray, fillArray, findMax, findMin CPU ✅ / GPU 🔄

Note

DummyMatrixOperation (CPU) implements every method with correct algorithms, including numerically-stable softmax with exp(x - max(x)) and epsilon-guarded normalization. All GPU backends are validated against it.

🤖 ML Model Wrappers

📋 Supported OpenNLP Model Types
Model Type GPU Wrapper Class OpenNLP Interface
Maximum Entropy GpuMaxentModel MaxentModel
Perceptron GpuPerceptronModel MaxentModel
Naive Bayes GpuNaiveBayesModel MaxentModel
Neural Network GpuNeuralNetworkModel Custom
Attention Layer GpuAttentionLayer Custom
Advanced Neural AdvancedGpuNeuralNetwork Custom
MaxEnt Trainer GpuMaxentTrainer EventTrainer

All wrappers follow the same decorator pattern: accept the base OpenNLP object, add GPU dispatch, and fall back to the base when GPU is unavailable.

📡 Performance Monitoring

GpuPerformanceMonitor monitor = GpuPerformanceMonitor.getInstance();
monitor.setAlertThresholdMs(500);          // Alert on ops > 500ms
monitor.setMemoryAlertThreshold(0.75);     // Alert at 75% GPU memory
monitor.setMaxHistorySize(5000);           // Keep last 5000 records/op

// After inference...
OperationMetrics metrics = monitor.getMetrics("matrixMultiply");
System.out.println("Avg latency: " + metrics.getAverageLatencyMs() + "ms");

(back to top ↑)

🧠 Advanced TF-IDF Vectorization

The project now uses a single, shared TF-IDF engine (TfIdfAlgorithms) across CPU/OpenCL/CUDA/ROCm wrappers to eliminate backend drift and keep scoring deterministic.

What changed

  • N-gram blend vectors: combine unigram/bigram/trigram terms in one feature space with configurable linear weights.
  • Weighting schemes: raw TF-IDF, sublinear TF-IDF, and BM25.
  • Smoothing and pruning:
    • IDF smoothing strategies: STANDARD_SMOOTH, PROBABILISTIC_IDF, BM25_IDF
    • Document-frequency controls: minDocumentFrequency / maxDocumentFrequency
  • Class-balanced feature scoring:
    • Information Gain and Chi-square feature selection
    • Macro-averaging support
    • Optional class-prior weighting for imbalanced datasets
  • Persistent reproducibility state:
    • VocabularyState persists vocabulary, DF statistics, blend weights, smoothing strategy, and DF cutoffs.
    • Versioned format with explicit migration policy (V1 → V2 fallback defaults).
  • Compressed dense vectors:
    • Persist dense vectors as FLOAT32, FLOAT16, or INT8 with load-time reconstruction.
  • Guardrail benchmark assertions:
    • Cross-backend parity assertions plus bounded latency-drift checks in tests.

Why it was added

  • Reduce algorithm divergence across backend wrappers.
  • Improve feature quality on heterogeneous text (phrases + tokens).
  • Make ranking behavior tunable for IR-style and classification-style workloads.
  • Improve reproducibility for train/infer pipelines through persisted vectorizer state.
  • Reduce storage/memory overhead when persisting large dense corpora.

Practical benefits

  • Better relevance: phrase-aware vectors and BM25-style weighting improve retrieval fidelity.
  • Fairer selection under imbalance: class-prior-aware scoring helps minority-class signal survive top-k pruning.
  • More stable production behavior: explicit smoothing/DF controls reduce noisy rare-term effects.
  • Smaller artifacts: float16/int8 persistence lowers disk and memory pressure for cached vectors.
  • Safer backend evolution: parity + latency guardrails detect regressions earlier.

API quick example

import org.apache.opennlp.gpu.features.GpuFeatureExtractor;
import org.apache.opennlp.gpu.features.TfIdfAlgorithms;

// extractor created as usual
GpuFeatureExtractor extractor = new GpuFeatureExtractor(provider, config, matrixOp);

// 1) Tune feature-engineering behavior
extractor.setNGramBlendOptions(TfIdfAlgorithms.NGramBlendOptions.linearMix(1.0, 0.7, 0.3));
extractor.setFeatureSelectionMethod(TfIdfAlgorithms.FeatureSelectionMethod.CHI_SQUARE);
extractor.setClassBalanceOptions(new TfIdfAlgorithms.ClassBalanceOptions(true, true)); // macro + prior weighting
extractor.setIdfSmoothingStrategy(TfIdfAlgorithms.IDFSmoothingStrategy.PROBABILISTIC_IDF);
extractor.setDocumentFrequencyCutoffs(2, Integer.MAX_VALUE);

// 2) Train-time vectorization with labels (for discriminative selection)
TfIdfAlgorithms.VectorizationResult train = extractor.extractTfIdfVectors(
    trainDocs,
    50000,
    TfIdfAlgorithms.WeightingScheme.BM25,
    trainLabels
);

// 3) Persist vocabulary state for reproducible inference
extractor.saveVocabularyState(java.nio.file.Path.of("tfidf-vocab-state.bin"));

// 4) Inference-time loading and vectorization with the same vocabulary/DF settings
extractor.loadVocabularyState(java.nio.file.Path.of("tfidf-vocab-state.bin"));
TfIdfAlgorithms.VectorizationResult infer = extractor.extractTfIdfVectorsWithLoadedVocabulary(
    incomingDocs,
    TfIdfAlgorithms.WeightingScheme.BM25
);

float[][] denseVectors = infer.getDenseVectors();

Persisted-state migration policy

State version Read support Behavior
V2 (current) ✅ Loads full metadata (vocabulary, DF, blend weights, smoothing, DF cutoffs).
V1 (legacy) ✅ Auto-migrates with safe defaults (STANDARD_SMOOTH, minDf=1, maxDf=Integer.MAX_VALUE).
Unknown/future ❌ Fails fast with clear error to avoid silent incompatibility.

Dense vector compression formats

Format Storage behavior Precision profile Best use case Caution
FLOAT32 Full precision Highest numeric fidelity Research baselines, strict parity Largest footprint
FLOAT16 Half-precision Good practical trade-off Large-scale caching with moderate tolerance Minor quantization noise
INT8 8-bit + per-row scale Aggressive compression Very large inference stores Greater reconstruction error

(back to top ↑)

🧮 Algorithms & Formula Choices

This section explains what algorithms were selected, what they do, and why they were favored over alternatives. The wording is intentionally between layman and technical depth.

TF-IDF family choices

Method Formula intuition Why chosen Common alternative Why not default alternative
Raw TF-IDF count in doc × rarity in corpus Strong baseline, easy to reason about Binary term presence Loses useful repetition signal
Sublinear TF-IDF $\log(1+tf)$ dampens repetition Reduces over-weighting repeated tokens Raw TF-only Too sensitive to term burstiness
BM25 Saturating TF + length normalization Better retrieval-style ranking quality Plain TF-IDF Less robust for varying doc lengths

Smoothing and DF controls

Option What it does Why needed in production
STANDARD_SMOOTH Adds stability to IDF denominator and offset Prevents extreme values for small corpora
PROBABILISTIC_IDF Uses odds-style rarity signal Useful when term discrimination needs stronger contrast
BM25_IDF BM25-compatible rarity scaling Keeps weighting family internally consistent
minDocumentFrequency Drops very rare terms Reduces noise and overfitting risk
maxDocumentFrequency Drops overly common terms Removes low-information global terms

Class-balanced scoring rationale

Strategy What it means Why it matters
Macro averaging Treat each class with equal weight in score aggregation Prevents majority classes from dominating selection
Class-prior weighting Up-weights minority class evidence Helps retain minority-class signal in top-k feature pruning
Chi-square Measures dependence between term and class Works well for discriminative vocabulary selection
Information gain Measures entropy reduction from term presence Strong general-purpose class relevance signal

Formula selection pipeline

flowchart TD
    A[Tokenized Documents] --> B{Weighting Scheme}
    B -->|Raw| C[Raw TF IDF]
    B -->|Sublinear| D[Sublinear TF IDF]
    B -->|BM25| E[BM25 TF and IDF]
    C --> F{Feature Selection}
    D --> F
    E --> F
    F -->|Frequency| G[Top by corpus stats]
    F -->|Chi square| H[Class dependence ranking]
    F -->|Information gain| I[Entropy reduction ranking]
    G --> J[Final Vocabulary]
    H --> J
    I --> J
Loading

Why these choices vs end-to-end neural embeddings?

Dimension This project approach End-to-end neural embedding stack
Integration effort Drop-in for existing OpenNLP apps Usually requires pipeline redesign
Explainability High (interpretable term-level features) Lower by default
Ops complexity Moderate (drivers + runtime checks) Higher (model serving infra + retraining lifecycle)
Cold start cost Low Higher
Best for Classical NLP modernization Greenfield neural-first architectures

Tip

The design goal here is not “replace all neural NLP.” It is “give existing OpenNLP systems a performance and feature-quality upgrade with low migration risk.”

(back to top ↑)

📚 Collapsible API Reference

GpuFeatureExtractor (high-level feature APIs)
API Purpose Typical use
extractNGramFeatures(...) Build n-gram count/frequency vectors Fast lexical baselines
extractTfIdfFeatures(...) Dense TF-IDF features for corpus Classification/retrieval inputs
extractTfIdfVectors(...) Rich vectorization result (dense+sparse+state) Advanced tuning and persistence
setNGramBlendOptions(...) Blend uni/bi/tri-grams Phrase sensitivity tuning
setFeatureSelectionMethod(...) Frequency/IG/Chi-square feature pruning Controlled vocabulary size
setClassBalanceOptions(...) Macro/prior weighting behavior Imbalanced dataset handling
setIdfSmoothingStrategy(...) Choose IDF smoothing family Stability vs discrimination tuning
setDocumentFrequencyCutoffs(...) Min/max DF filtering Noise and stop-term reduction
saveVocabularyState(...) / loadVocabularyState(...) Persist/reload feature-state metadata Reproducible train/infer alignment
TfIdfAlgorithms (shared algorithm core)
API group Key methods Why it exists
Vectorization vectorizeDocuments(...), vectorizeDocumentsWithVocabulary(...) Single-source behavior across backends
State persistence saveVocabularyState(...), loadVocabularyState(...) Versioned reproducibility
Dense persistence saveDenseVectors(...), loadDenseVectors(...) Storage/memory optimization paths
Token normalization tokenizeNormalized(...) Centralized text normalization policy
Cache controls clearCache(), getCacheSize() Repeat-run speed and deterministic testing
Operational toggles and guardrails
Concern Mechanism Recommendation
Runtime safety CPU fallback paths Keep enabled in all environments
Regression detection Backend parity + latency guardrail tests Run in CI before release
Explainability Term-level vectorization + DF metadata Persist state for audits
Performance stability Batch size + memory pool tuning Tune per deployment profile

(back to top ↑)

⚙️ Configuration

All settings are controlled via GpuConfig (a plain Java value object):

Property Default Description
gpuEnabled false Master GPU switch
memoryPoolSizeMB 256 Pre-allocated GPU memory pool size (MB)
batchSize 32 Samples per GPU kernel launch
maxMemoryUsageMB 1024 Hard memory cap per provider (MB)
debugMode false Verbose diagnostic output

System properties (read at runtime):

Property Example Description
gpu.available true Master GPU presence flag
gpu.vendor NVIDIA Reported vendor name
gpu.device RTX 4090 Device display name
gpu.driver 535.0 Driver version string
gpu.memory.total 24576 Total VRAM in MB
gpu.speedup.factor 3.5 Reported speedup for stats reporting

(back to top ↑)

🔍 Diagnostics

Run the built-in hardware probe before deploying:

mvn exec:java -Dexec.mainClass=org.apache.opennlp.gpu.tools.GpuDiagnostics

Sample output:

🔍 OpenNLP GPU Acceleration - Hardware Diagnostics
==================================================
[System Information]
  OS:           Linux 6.x.x-zen
  Java Version: 26.0.2 ✅ Compatible
  JAVA_HOME:    /usr/lib/jvm/java-26-openjdk ✅ Set and valid
[GPU Hardware Detection]
  AMD GPU:      ✅ Detected: AMD Radeon RX 7900 XTX
[AMD Drivers]
  AMD ROCm Driver: ✅ Installed and working
[OpenCL Runtime]
  OpenCL:       ✅ 2 platform(s), 3 device(s)
[OpenNLP GPU Integration]
  Extension JAR: ✅ Loaded successfully

🎉 GPU acceleration is ready to use!

Exit code 0 = ready, 1 = setup incomplete.

(back to top ↑)

🗺️ Project Roadmap

gantt
    title OpenNLP GPU Extension Roadmap
    dateFormat  YYYY-MM-DD
    section Phase 1: Foundation
        Core Interfaces & CPU Fallback     :done,    p1a, 2025-01-01, 2025-04-01
        ComputeProvider Hierarchy          :done,    p1b, 2025-01-01, 2025-04-01
        GpuConfig & Monitoring             :done,    p1c, 2025-03-01, 2025-05-01
    section Phase 2: ML Models
        MaxEnt / Perceptron / Naive Bayes  :done,    p2a, 2025-04-01, 2025-07-01
        Neural Network & Attention         :done,    p2b, 2025-05-01, 2025-08-01
        GPU Diagnostics Tool               :done,    p2c, 2025-06-01, 2025-08-01
    section Phase 2.5: Feature Engineering Hardening
        Unified TF IDF engine across backends            :done, p25a, 2026-05-01, 2026-06-30
        N gram blending and BM25 sublinear weighting     :done, p25b, 2026-05-10, 2026-06-30
        Class balanced scoring IG and Chi square          :done, p25c, 2026-05-15, 2026-06-30
        IDF smoothing and min max DF controls             :done, p25d, 2026-05-20, 2026-06-30
        VocabularyState versioning and V1 to V2 migration :done, p25e, 2026-05-25, 2026-06-30
        Dense vector quantization FLOAT16 and INT8        :done, p25f, 2026-06-01, 2026-06-30
        Cross backend parity and latency guardrails       :done, p25g, 2026-06-05, 2026-06-30
    section Phase 3 - Native GPU (Active)
        OpenCL JNI Bridge                  :active,  p3a, 2025-09-01, 2026-06-01
        CUDA Kernel Integration            :active,  p3b, 2025-10-01, 2026-07-01
        ROCm / HIP Integration             :         p3c, 2026-03-01, 2026-09-01
    section Phase 4 - Cloud & Production
        AWS Inferentia Integration         :         p4a, 2026-06-01, 2026-10-01
        Google TPU Integration             :         p4b, 2026-07-01, 2026-11-01
        Maven Central Release              :         p4c, 2026-10-01, 2026-12-01
Loading
Phase Goals Target Status
Phase 1 Core interfaces, CPU fallback, monitoring Q1-Q2 2025 ✅ Complete
Phase 2 ML model wrappers, diagnostics, test suite Q2-Q3 2025 ✅ Complete
Phase 2.5 TF-IDF/feature-engineering hardening (blending, balancing, smoothing, migration, quantization, guardrails) Q2 2026 ✅ Complete
Phase 3 OpenCL + CUDA JNI kernels, ROCm integration Q4 2025–Q3 2026 🔄 Active
Phase 4 Cloud accelerators, Maven Central, production hardening Q4 2026 ⭕ Planned

(back to top ↑)

📈 Development Status

pie title Component Readiness (% complete)
    "CPU Fallback (100%)" : 100
    "Monitoring (100%)" : 100
    "Diagnostics (100%)" : 100
    "ML Wrappers (100%)" : 100
    "OpenCL JOCL Detection (80%)" : 80
    "CUDA/ROCm JOCL Detection (75%)" : 75
    "Cloud Providers — CPU Fallback (70%)" : 70
    "Native GPU Kernels — JNI Bridge (25%)" : 25
Loading
Version Phase Stability Java OpenNLP Key Limitation
1.0.0 Phase 1-2 Beta 21 2.5.8 Hardware GPU kernel execution requires native JNI bridge (CPU fallback active)

Warning

Hardware GPU kernel execution (isAvailable() == true + real device dispatch) requires the in-progress JNI bridge to be compiled with -Pnative and a compatible driver stack verified by the GpuDiagnostics tool. JOCL-based provider detection (CudaUtil.isAvailable(), OpenCLUtil.isAvailable(), RocmUtil.isAvailable()) is fully implemented and returns real hardware results. Until the native kernel bridge is wired, all matrix compute routes silently through CpuComputeProvider.

(back to top ↑)

🔬 Further Reading & Research References

The implementation choices here are practical engineering adaptations of widely used IR/NLP methods. If you want the deeper theoretical background, these are strong references:

Topic Reference Why it is relevant
BM25 foundations Robertson, S. and Zaragoza, H. (2009), The Probabilistic Relevance Framework: BM25 and Beyond Canonical explanation of BM25 behavior and ranking trade-offs
Information Retrieval fundamentals Manning, C. D., Raghavan, P., Schütze, H. (2008), Introduction to Information Retrieval Core TF-IDF, DF, and retrieval math intuition
Feature selection (text classification) Yang, Y. and Pedersen, J. O. (1997), A Comparative Study on Feature Selection in Text Categorization Practical comparison of IG/Chi-square for text features
Neural attention context Vaswani et al. (2017), Attention Is All You Need (arXiv:1706.03762) Useful contrast point versus classical feature-engineered pipelines
Compression/quantization context Dettmers et al. (2022), LLM.int8() (arXiv:2208.07339) Modern perspective on low-bit numeric compression trade-offs

Note

This project intentionally emphasizes compatibility and explainability for existing OpenNLP systems. The references above include both classical IR and modern deep-learning context to clarify why these choices were made.

(back to top ↑)

🤝 Contributing

Contributions are welcome! This project follows the standard GitHub pull-request workflow.

# Fork, then:
git clone https://github.com/YOUR_USERNAME/opennlp-gpu.git
cd opennlp-gpu
git checkout -b feature/my-improvement
# Make changes, add tests
mvn clean test
git commit -m "feat: describe your change"
git push origin feature/my-improvement
# Open a Pull Request on GitHub
📋 Contribution Guidelines

Code Style

  • Java 21 syntax; no Lombok (removed to reduce annotation processor complexity)
  • All new public APIs must include structured Javadoc comments (Requirement, Purpose, Inputs, Outputs, Failure Modes)
  • Follow existing package structure: common/, compute/, ml/, monitoring/, tools/

Testing Requirements

  • Unit tests in src/test/java/ matching the source package
  • New GPU backends must include a CPU-parity test verifying numerical equivalence
  • Stress tests for any concurrent code (stress/ test package)

Pull Request Checklist

  • mvn clean compile passes with zero errors
  • mvn test -Dtest=GpuTestSuite,MatrixOpsTest passes
  • No new Xlint:all warnings introduced
  • GpuDiagnostics still reports correctly

(back to top ↑)

📜 Attribution

This project extends Apache OpenNLP but is not part of the Apache Software Foundation.

Component Owner License
Apache OpenNLP (opennlp-tools) Apache Software Foundation Apache License 2.0
JOCL Marco Hutter / jocl.org MIT License
This GPU Extension OpenNLP GPU Extension Contributors Apache License 2.0 
OpenNLP GPU Extension
Copyright 2025 OpenNLP GPU Extension Contributors

This software includes code from Apache OpenNLP:
Copyright 2011-2025 The Apache Software Foundation

(back to top ↑)

📄 License

Distributed under the Apache License, Version 2.0. See LICENSE for full text.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

(back to top ↑)

Built with ❤️ for the Java NLP community

Apache OpenNLP · Report Bug · Request Feature

About

GPU acceleration extensions for Apache OpenNLP** to dramatically boost natural language processing performance with seamless integration and zero accuracy loss.

Topics

gpu-computinggpu-nlplang-javanlp-toolsplatforms-portfolioplatforms-toolsscope-mediumstatus-activestatus-stable

Resources

Readme

Code of conduct

Code of conduct

Contributing

Contributing
Activity

Stars

3 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages