⚡ Asynchronous Ingestion

Architecture of the mass processing engine with Virtual Threads and Spring Batch.

The core of the Pipeline is designed to process massive logistics telemetry files asynchronously. The architecture decouples the HTTP reception from the heavy I/O processing, guaranteeing service availability.

🧵 Concurrency with Virtual Threads (Java 21)

To prevent the exhaustion of the server’s physical threads during heavy loads, the Spring Batch Pipeline delegates concurrent execution to an executor backed by Java 21 native virtual threads.

// Configuration in BatchConfig.java
@Bean
public TaskExecutor taskExecutor() {
    return new TaskExecutorAdapter(Executors.newVirtualThreadPerTaskExecutor());
}

📦 Batch Processing and Bulk Inserts

To achieve high database throughput without saturating the container’s RAM, the writing process combines the native capabilities of Spring Batch with Hibernate and PostgreSQL JDBC Batching.

🎯 Strategic Chunk Size

The import Step is configured to read, process, and write in batches (chunks) of 500 records at a time.

// Chunk configuration in BatchConfig.java
@Bean
public Step importTelemetryStep(...) {
    return new StepBuilder("importTelemetryStep", jobRepository)
            .<TelemetryCsvRecord, LogiflowTelemetry>chunk(500, transactionManager)
            // ...
            .build();
}

🛡️ Resilience and Fault-Tolerant

A critical aspect of mass ingestion of external data is resilience against corrupt files or malformed rows. The pipeline is configured under a strict fault-tolerant skip policy (faultTolerant).

If the reader detects a parsing error or an unexpected exception in a row, the engine isolates the record, increments the skip counter, and continues processing the rest of the file without interrupting the entire batch transaction, supporting a maximum limit of up to 500 exclusions.

// Fault-tolerant configuration in BatchConfig.java
.faultTolerant()
.skip(FlatFileParseException.class)
.skip(Exception.class)
.skipLimit(500)