> ## Documentation Index
> Fetch the complete documentation index at: https://alan-ramirez-dev.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# 🚚 Introduction to the ETL Pipeline

> Logiflow ETL API - Asynchronous Mass Ingestion Engine (Backend)

Welcome to the documentation of the third microservice of the project. My Portfolio's Asynchronous Mass Ingestion ETL Pipeline!

This project functions as a high-performance asynchronous microservice, designed for the extraction, transformation, and mass loading of logistics telemetry logs. It solves the architectural challenge of processing tens of thousands of records simultaneously without blocking or hindering the user interface, while protecting the database against attacks and corrupt files.

## 🚀 Features and Backend Architecture

<CardGroup cols={2}>
  <Card title="Virtual Threads (Loom)" icon="microchip">
    Delegation of heavy I/O work to a native Java 21 `TaskExecutorAdapter`, freeing the HTTP connection instantly (`202 Accepted`).
  </Card>

  <Card title="JDBC Bulk Inserts" icon="database">
    Synergy between Spring Batch (`chunk(500)`), Hibernate (`allocationSize=500`) and the PostgreSQL driver for mass insertions.
  </Card>

  <Card title="Multi-layer Defense (WAF)" icon="shield-half">
    `@Transactional` isolation and simulated WAF against SQL injection. Use of `AtomicBoolean` to block concurrency attacks (TOCTOU).
  </Card>

  <Card title="Fault-Tolerant Resilience" icon="truck">
    Skip policies (`SkipPolicy`). Corrupt or empty records are isolated and silently discarded without aborting the entire batch.
  </Card>

  <Card title="Secure Infrastructure" icon="docker">
    Multi-stage deployment on Alpine Linux, operating under a `spring` user (no-root) and data persistence via volumes.
  </Card>
</CardGroup>

## 🛠️ Tech Stack

| Technology       | Version | Project Purpose                                           |
| :--------------- | :------ | :-------------------------------------------------------- |
| **Spring Boot**  | `^3.x`  | Orchestration, asynchronous RESTful controllers and IoC   |
| **Java**         | `21`    | Main language (Virtual Threads and Records)               |
| **Spring Batch** | `^5.x`  | ETL for file reading (NIO) and Chunk processing           |
| **PostgreSQL**   | `17`    | Relational store with batch writing capabilities          |
| **Docker**       | `v2`    | Containerization and local network environment deployment |

## 💻 Development Commands

The API exposes its services on port `8081` and requires PostgreSQL on port `5435`.

```bash theme={null}
# Initializes PostgreSQL with persistent volume
docker-compose up -d

# Compiles and generates the executable artifact (.jar)
mvn clean package -DskipTests

# Starts the application locally
mvn spring-boot:run
```

## 📡 API Documentation (Endpoints)

The API operates under the base route `/api/v1/etl`. Failed responses return a standard structure `{"error": "Reason for failure"}`.

### 🏭 Ingestion and Query Engine

| Method | Endpoint   | Description                                                                   | Required Payload      |
| :----- | :--------- | :---------------------------------------------------------------------------- | :-------------------- |
| `POST` | `/upload`  | Starts the asynchronous file ingestion on disk and starts the Virtual Thread. | `MultipartFile` (CSV) |
| `GET`  | `/status`  | *Polling* method to check the Batch Job progress in real time.                | None                  |
| `GET`  | `/preview` | Returns the 5 most recent logistic records based on the index.                | None                  |

### 📟 Interactive Console and Support

| Method   | Endpoint | Description                                                               | Required Payload  |
| :------- | :------- | :------------------------------------------------------------------------ | :---------------- |
| `POST`   | `/query` | Direct SQL engine for evaluation in the dashboard's interactive terminal. | JSON with `query` |
| `DELETE` | `/reset` | Total purge (Demo Mode). Clears the tables and Spring Batch metadata.     | None              |