Background Workers & BullMQ Processing

​

The Workers Law

​

BullMQ Processing Pipeline

​

Dedicated Worker Processors

​

1. marketing-voice.processor.ts

• Processes autonomous outbound AI voice campaigns.
• Integrates normalizeVoiceLeadVariables from @brokeros/constants to sanitize customer names, project titles, and phonetic pronunciations.
• Uses tryCarrierBridgeDispatch from @brokeros/int-voice to initiate calls via Vobiz, Exotel, Twilio, or Telnyx.
• Enforces strict carrier trunk concurrency limits (e.g., maximum 25 concurrent calls per trunk) to prevent SIP line exhaustion.

​

2. marketing-whatsapp.processor.ts

• Processes outbound Meta message template broadcasts and drip automations.
• Regulates dispatch velocity according to Meta Messaging Tiers:
  • Tier 1: 1,000 unique customers per rolling 24 hours
  • Tier 2: 10,000 unique customers per rolling 24 hours
  • Tier 3: 100,000 unique customers per rolling 24 hours
• Modular sub-runners in processors/whatsapp/:
  • whatsapp-broadcast.runner.ts: Batch broadcast distribution
  • whatsapp-automation.runner.ts: Conditional drip workflow execution
  • whatsapp-template.syncer.ts: Synchronizes approved templates from Meta

​

3. marketing-sms.processor.ts

• Dispatches bulk promotional and transactional SMS messages.
• Appends DLT principal entity IDs and template IDs for telecommunications compliance in India.
• Batches gateway transmissions (Gupshup, Twilio, Sinch, AWS SNS) in chunks of 100 recipients.

​

4. marketing-email.processor.ts

• Streams personalized HTML emails through SendGrid, Brevo, or AWS SES.
• Injects tracked short-links and transparent 1x1 tracking pixels for open and click attribution.
• Suppresses previously bounced or complained recipient addresses automatically.

​

5. scheduled-messages.processor.ts

• Processes scheduled 1-on-1 team inbox messages across SMS, Email, and WhatsApp.
• Runs on a high-frequency 10-second interval scanner to pick up due messages promptly.
• Delegates dispatch execution directly to the deep @brokeros/messaging module (dispatchScheduledMessage):
  • Atomic Reservation: Acquires a mutex lock via conditional database transition (status: 'SCHEDULED' transitioning to status: 'PROCESSING'), preventing race conditions with concurrent worker nodes or immediate API triggers.
  • Provider Execution: Dispatches via cached adapter instances across SMS (Twilio, Gupshup, Sinch, AWS SNS), Email (SendGrid, Brevo, AWS SES), or WhatsApp (Meta Cloud API).
  • Audit Preservation: Transitions message state in-place to DELIVERED, SENT, or FAILED while maintaining thread conversation summaries without deleting audit rows.
• Resilient against worker restarts and power cycles: any message in SCHEDULED status with scheduledAt <= NOW() is picked up immediately upon restart.

​

Job Configuration & Retry Policies

export const defaultJobOptions = {
  attempts: 3,
  backoff: {
    type: 'exponential',
    delay: 5000, // 5s, 10s, 20s
  },
  removeOnComplete: {
    age: 86400, // Retain completed job logs for 24 hours
    count: 5000,
  },
  removeOnFail: {
    age: 604800, // Retain failed jobs for 7 days for post-mortem analysis
  },
};

​

Running Workers Locally

# Starts BullMQ worker processors watching Redis on port 6379
pnpm --filter @brokeros/workers start:dev

# Runs BullMQ processor E2E tests
pnpm test:workers:e2e