🚚 Introduction to the ETL Pipeline

Logiflow ETL API - Asynchronous Mass Ingestion Engine (Backend)

Welcome to the documentation of the third microservice of the project. My Portfolio’s Asynchronous Mass Ingestion ETL Pipeline!

This project functions as a high-performance asynchronous microservice, designed for the extraction, transformation, and mass loading of logistics telemetry logs. It solves the architectural challenge of processing tens of thousands of records simultaneously without blocking or hindering the user interface, while protecting the database against attacks and corrupt files.

🚀 Features and Backend Architecture

Virtual Threads (Loom)

Delegation of heavy I/O work to a native Java 21 TaskExecutorAdapter, freeing the HTTP connection instantly (202 Accepted).

JDBC Bulk Inserts

Synergy between Spring Batch (chunk(500)), Hibernate (allocationSize=500) and the PostgreSQL driver for mass insertions.

Multi-layer Defense (WAF)

@Transactional isolation and simulated WAF against SQL injection. Use of AtomicBoolean to block concurrency attacks (TOCTOU).

Fault-Tolerant Resilience

Skip policies (SkipPolicy). Corrupt or empty records are isolated and silently discarded without aborting the entire batch.

Secure Infrastructure

Multi-stage deployment on Alpine Linux, operating under a spring user (no-root) and data persistence via volumes.

🛠️ Tech Stack

Technology	Version	Project Purpose
Spring Boot	^3.x	Orchestration, asynchronous RESTful controllers and IoC
Java	21	Main language (Virtual Threads and Records)
Spring Batch	^5.x	ETL for file reading (NIO) and Chunk processing
PostgreSQL	17	Relational store with batch writing capabilities
Docker	v2	Containerization and local network environment deployment

💻 Development Commands

The API exposes its services on port 8081 and requires PostgreSQL on port 5435.

# Initializes PostgreSQL with persistent volume
docker-compose up -d

# Compiles and generates the executable artifact (.jar)
mvn clean package -DskipTests

# Starts the application locally
mvn spring-boot:run

📡 API Documentation (Endpoints)

The API operates under the base route /api/v1/etl. Failed responses return a standard structure {"error": "Reason for failure"}.

🏭 Ingestion and Query Engine

Method	Endpoint	Description	Required Payload
POST	/upload	Starts the asynchronous file ingestion on disk and starts the Virtual Thread.	MultipartFile (CSV)
GET	/status	Polling method to check the Batch Job progress in real time.	None
GET	/preview	Returns the 5 most recent logistic records based on the index.	None

📟 Interactive Console and Support

Method	Endpoint	Description	Required Payload
POST	/query	Direct SQL engine for evaluation in the dashboard’s interactive terminal.	JSON with query
DELETE	/reset	Total purge (Demo Mode). Clears the tables and Spring Batch metadata.	None