Reporting service

Introduction

This project is a Node.js application using TypeScript and NestJS. It generates Excel reports from user data stored in a MySQL database, streams the data efficiently using Node.js streams, and uploads the resulting report to AWS S3. The design focuses on memory efficiency and scalability for large datasets.

Technologies Used

• Node.js v22.14.0
• TypeScript
• NestJS
• MySQL (via TypeORM)
• AWS SDK v3 (@aws-sdk/client-s3) and MinIO for development experience
• ExcelJS
• Node.js Streams (Readable, Transform, Writable)

How Node.js Streams Work

Node.js streams allow processing of data piece by piece, without loading everything into memory. There are four main types:

• Readable: Source of data (e.g., database query)
• Writable: Destination for data (e.g., file, network)
• Duplex: Both readable and writable
• Transform: Duplex stream that can modify data as it passes through

flowchart LR
    A[Readable Stream] -- data --> B[Transform Stream]
    B -- transformed data --> C[Writable Stream]

Loading

Application Data Flow

The following diagram illustrates the data flow in app.service.ts:

flowchart TD
    DB["Get user info from DB"]
    RS["To Readable Stream"]
    TS["Transform Stream to S3 worksheet"]
    WS["Writable Stream for S3 object"]
    S3U["S3 Uploader service"]
    S3["AWS S3 Bucket Uploaded"]

    DB --> RS
    RS --> TS
    TS --> WS
    WS --> S3U
    S3U --> S3

Loading

Project Setup Commands

• Install dependencies: npm install
• Build the project: npm run build
• Start the application: npm start
• Start in development mode: npm run start:dev
• Run tests: npm run test
• Run command to generate a report: npm run command:generate-report

Environment Variables

Copy .env.dist to .env and fill in the required values.