> ## Documentation Index
> Fetch the complete documentation index at: https://alan-ramirez-dev.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# ⚡ Asynchronous Ingestion

> Architecture of the mass processing engine with Virtual Threads and Spring Batch.

The core of the **Pipeline** is designed to process massive logistics telemetry files asynchronously. The architecture decouples the HTTP reception from the heavy I/O processing, guaranteeing service availability.

## 🧵 Concurrency with Virtual Threads (Java 21)

To prevent the exhaustion of the server's physical threads during heavy loads, the Spring Batch Pipeline delegates concurrent execution to an executor backed by Java 21 native virtual threads.

```java theme={null}
// Configuration in BatchConfig.java
@Bean
public TaskExecutor taskExecutor() {
    return new TaskExecutorAdapter(Executors.newVirtualThreadPerTaskExecutor());
}
```

## 📦 Batch Processing and Bulk Inserts

To achieve high database throughput without saturating the container's RAM, the writing process combines the native capabilities of Spring Batch with Hibernate and PostgreSQL JDBC Batching.

### 🎯 Strategic Chunk Size

The import `Step` is configured to read, process, and write in batches (chunks) of **500 records** at a time.

```java theme={null}
// Chunk configuration in BatchConfig.java
@Bean
public Step importTelemetryStep(...) {
    return new StepBuilder("importTelemetryStep", jobRepository)
            .<TelemetryCsvRecord, LogiflowTelemetry>chunk(500, transactionManager)
            // ...
            .build();
}
```

## 🛡️ Resilience and Fault-Tolerant

A critical aspect of mass ingestion of external data is resilience against corrupt files or malformed rows. The pipeline is configured under a strict fault-tolerant skip policy (`faultTolerant`).

If the reader detects a parsing error or an unexpected exception in a row, the engine isolates the record, increments the skip counter, and continues processing the rest of the file without interrupting the entire batch transaction, supporting a maximum limit of up to 500 exclusions.

```java theme={null}
// Fault-tolerant configuration in BatchConfig.java
.faultTolerant()
.skip(FlatFileParseException.class)
.skip(Exception.class)
.skipLimit(500)
```