BullMQ Module¶

Overview¶

The BullMQ module provides queue management for background job processing in the BidScript application. It uses Redis as a backend for job storage and scheduling, enabling asynchronous task execution, job prioritisation, and retry mechanisms.

Configuration¶

The BullMQ module is configured in the following files:

bullmq.module.ts: Defines the BullMQ module
bullmq.providers.ts: Provides queue configurations
linked-references-worker.module.ts: Configures workers for linked references
linked-references.worker.ts: Implements the linked references worker

Key Components¶

BullMQModule¶

The BullMQModule is the main module that exposes the queue providers:

@Module({
  providers: [BullMQProvider, LinkedReferencesQueueProvider],
  exports: ["FLUSH_QUEUE", "LINKED_REFERENCES_QUEUE"],
})
export class BullMQModule {}

Queue Providers¶

The module defines providers for different queue types:

FLUSH_QUEUE: For general background tasks
LINKED_REFERENCES_QUEUE: For processing linked references

Workers¶

Workers process jobs from the queues:

LinkedReferencesWorker: Processes jobs in the linked references queue
LinkedReferencesWorkerModule: Module for the linked references worker

Integration with Other Modules¶

The BullMQ module integrates with:

Redis Module: For queue backend storage
Document Processing: For async document processing
Notification Module: For handling notification jobs

Environment Variables¶

The module relies on the following environment variables:

REDIS_HOST: Redis host for the queue
REDIS_PORT: Redis port
REDIS_PASSWORD: Redis password

Usage Example¶

// Example of adding a job to the LinkedReferencesQueue
import { Inject, Injectable } from "@nestjs/common";
import { Queue } from "bullmq";

@Injectable()
export class SomeService {
  constructor(
    @Inject("LINKED_REFERENCES_QUEUE")
    private linkedReferencesQueue: Queue
  ) {}

  async scheduleReferenceProcessing(data: any): Promise<void> {
    await this.linkedReferencesQueue.add("process-references", data, {
      attempts: 3,
      backoff: {
        type: "exponential",
        delay: 1000,
      },
    });
  }
}

Dependencies¶

bullmq: Modern queue implementation for Node.js
@nestjs/bull: NestJS integration for Bull/BullMQ
ioredis: Redis client used by BullMQ