Migrating Workflows from Temporal to BullMQ

This guide helps you migrate your ToolJet workflow scheduling system from the legacy Temporal-based architecture to the new BullMQ-based system.

note

This migration guide is for ToolJet version v3.20.37-LTS and later. Versions prior to v3.20.37-LTS used a Temporal-based workflow system.

Overview

ToolJet has replaced Temporal with BullMQ for workflow scheduling, significantly simplifying deployment while maintaining all existing functionality. This change eliminates the need for separate Temporal server infrastructure and provides built-in monitoring capabilities.

Why Migrate?

Benefits of BullMQ-based Workflows

Simplified Architecture: No need for separate Temporal server deployment
Existing Infrastructure: Leverages your existing Redis instance
Better Resource Management: Flexible worker modes for optimized scaling
Improved Visibility: Real-time job status tracking and retry capabilities

Architecture Comparison

Feature	Temporal (Old)	BullMQ (New)
External Services	Temporal Server + Redis	Redis only
Deployment Complexity	High (multi-service)	Low (single-service)
Infrastructure Cost	Higher	Lower

How It Works

The new BullMQ-based workflow system operates as follows:

Workflow Scheduling: When you schedule a workflow, it's stored in PostgreSQL and a corresponding job is created in Redis using BullMQ
Job Queues: Two BullMQ queues manage workflows:
- workflow-schedule-queue: Handles scheduled workflow triggers
- workflow-execution-queue: Manages workflow execution
Worker Processing: ToolJet instances with WORKER=true pick up jobs from these queues and execute them
Schedule Recovery: On startup, the Schedule Bootstrap Service automatically loads all active schedules from PostgreSQL and recreates them in Redis, ensuring no workflows are lost during deployments
State Updates: The frontend polls workflow execution states every 3 seconds via batch API calls

Migration Steps

1. Review Current Setup

Check your current deployment for Temporal-related configurations:

Environment Variables to Remove:

# Old Temporal variables - REMOVE THESE
ENABLE_WORKFLOW_SCHEDULING=true
WORKFLOW_WORKER=true
TOOLJET_WORKFLOWS_TEMPORAL_NAMESPACE=default
TEMPORAL_SERVER_ADDRESS=temporal:7233

Services to Remove:

Temporal server containers/pods
Temporal worker containers/pods

2. Set Up Redis

warning

External Redis Requirement: When running separate worker containers or multiple instances, an external stateful Redis instance is required for job queue coordination. The built-in Redis only works when the server and worker are in the same container instance (single instance deployment).

Redis Requirements:

Persistence: AOF (Append Only File) must be enabled
Memory Policy: maxmemory-policy must be set to noeviction (required by BullMQ)
Version: Redis 6.x or higher (Redis 7.x recommended)

Example Redis Configuration:

# Persistence
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec

# Memory Management
maxmemory-policy noeviction

# RDB Snapshots
save 900 1
save 300 10
save 60 10000

Redis Setup by Platform:

Kubernetes (EKS, AKS, GKE, OpenShift)

Deploy stateful Redis:

kubectl apply -f https://tooljet-deployments.s3.us-west-1.amazonaws.com/kubernetes/redis-stateful.yaml

This creates:

StatefulSet with persistent storage
Headless Service
ConfigMap with production-ready configuration
Secret for password authentication

Configure environment variables:

REDIS_HOST=redis-service.default.svc.cluster.local
REDIS_PORT=6379
REDIS_PASSWORD=your-secure-password

AWS ECS

Use Amazon ElastiCache for Redis:

Create Redis cluster with:
- Engine version: Redis 7.x
- Node type: cache.t3.medium or higher
- Automatic failover enabled
Configure parameter group:
- Set maxmemory-policy to noeviction
- Set appendonly to yes
Add environment variables:

REDIS_HOST=your-elasticache-endpoint.cache.amazonaws.com
REDIS_PORT=6379
REDIS_PASSWORD=your-redis-password

Azure Container Apps

Use Azure Cache for Redis:

Create Redis instance (Standard or Premium tier)
Configure Redis settings for AOF and noeviction policy
Add environment variables:

REDIS_HOST=your-redis.redis.cache.windows.net
REDIS_PORT=6379
REDIS_PASSWORD=your-redis-password
REDIS_TLS=true

Google Cloud Run

Use Google Cloud Memorystore for Redis:

Create Redis instance with:
- Redis version 7.x
- High availability enabled
Configure Redis settings via gcloud CLI
Add environment variables:

REDIS_HOST=your-memorystore-ip
REDIS_PORT=6379
REDIS_PASSWORD=your-redis-password

3. Update Environment Variables

Add the new BullMQ workflow environment variables:

Required Variables:

# Worker Mode (required)
# Set to 'true' to enable job processing
WORKER=true

# Redis connection settings
REDIS_HOST=localhost         # Default: localhost
REDIS_PORT=6379              # Default: 6379
REDIS_USERNAME=              # Optional: Redis username (ACL)
REDIS_PASSWORD=              # Optional: Redis password
REDIS_DB=0                   # Optional: Redis database number (default: 0)
REDIS_TLS=false              # Optional: Enable TLS/SSL (set to 'true')

Note: Only REDIS_HOST and REDIS_PORT are required. Authentication and TLS are optional.

Optional Variables:

# Workflow Processor Concurrency (optional)
# Number of workflow jobs processed concurrently per worker
# Default: 5
TOOLJET_WORKFLOW_CONCURRENCY=5

# Workflow Timeout (optional)
# Maximum execution time for a workflow in seconds
# Default: 60
WORKFLOW_TIMEOUT_SECONDS=60

# Redis Configuration (optional)
REDIS_USERNAME=          # Redis username (ACL)
REDIS_DB=0              # Redis database number (default: 0)
REDIS_TLS=false         # Enable TLS/SSL (set to 'true')

4. Deploy Updated Configuration

Update your ToolJet deployment with the new configuration:

For Kubernetes:

# Update your deployment.yaml with new environment variables
kubectl apply -f deployment.yaml

# Restart pods to apply changes
kubectl rollout restart deployment/tooljet

For Docker/Docker Compose:

# Update your .env file or docker-compose.yml
docker-compose down
docker-compose up -d

For AWS ECS:

Update task definition with new environment variables
Create new revision and update service

For Azure Container Apps:

Update environment variables in container app settings
Save and restart

5. Remove Temporal Infrastructure

After confirming the new setup works:

For Kubernetes:

# Remove Temporal deployments
kubectl delete deployment temporal-server
kubectl delete deployment temporal-worker
kubectl delete service temporal-service

For Docker Compose:

# Remove Temporal services from docker-compose.yml
# - temporal-server
# - temporal-worker

For ECS:

Stop and delete Temporal task definitions
Remove Temporal services

6. Verify Migration

Check Workflow Scheduling: Create a new scheduled workflow in ToolJet
Verify Execution: Trigger a workflow and confirm it executes successfully
Check Logs: Review application logs for any errors

Scaling Workflows with Dedicated Workers

For production deployments with extensive workflow usage, it's recommended to deploy dedicated worker instances that only process jobs without serving HTTP traffic.

Why Dedicated Workers?

Better Resource Allocation: Separate compute resources for API and job processing
Independent Scaling: Scale workers based on job queue depth
Improved Reliability: HTTP server issues don't affect job processing
Cost Optimization: Use different instance sizes for API vs workers

Architecture

            ┌─────────────────────────────────────────────────────────────┐
            │                    ToolJet Deployment                       │
            ├─────────────────────────────────────────────────────────────┤
            │                                                             │
            │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐       │
            │  │ Web Server   │  │  Worker 1    │  │  Worker 2    │       │
            │  │              │  │              │  │              │       │
            │  │ WORKER=true  │  │ WORKER=true  │  │ WORKER=true  │       │
            │  │              │  │              │  │              │       │
            │  │ HTTP Requests│  │ Process Jobs │  │ Process Jobs │       │
            │  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘       │
            │         │                 │                 │               │
            │         └─────────────────┼─────────────────┘               │
            │                           │                                 │
            └───────────────────────────┼─────────────────────────────────┘
                                        │
                                        ▼
                               ┌─────────────────┐
                               │  External Redis │
                               │   (Stateful)    │
                               │                 │
                               │  - Job Queue    │
                               │  - Persistence  │
                               └─────────────────┘

Deployment Configuration

Kubernetes Example

ToolJet App Deployment (tooljet-deployment.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tooljet-deployment
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      component: tooljet
  template:
    metadata:
      labels:
        component: tooljet
    spec:
      imagePullSecrets:
        - name: docker-secret
      containers:
        - name: tooljet
          image: tooljet/tooljet:ee-lts-latest
          imagePullPolicy: Always
          args: ["npm", "run", "start:prod"]
          resources:
            limits:
              memory: "2000Mi"
              cpu: "2000m"
            requests:
              memory: "1000Mi"
              cpu: "1000m"
          ports:
            - containerPort: 3000
          readinessProbe:
            httpGet:
              port: 3000
              path: /api/health
            successThreshold: 1
            initialDelaySeconds: 10
            periodSeconds: 5
            failureThreshold: 6
          env:
            - name: PG_HOST
              valueFrom:
                secretKeyRef:
                  name: server
                  key: pg_host
            - name: PG_USER
              valueFrom:
                secretKeyRef:
                  name: server
                  key: pg_user
            - name: PG_PASS
              valueFrom:
                secretKeyRef:
                  name: server
                  key: pg_password
            - name: PG_DB
              valueFrom:
                secretKeyRef:
                  name: server
                  key: pg_db
            - name: LOCKBOX_MASTER_KEY
              valueFrom:
                secretKeyRef:
                  name: server
                  key: lockbox_key
            - name: SECRET_KEY_BASE
              valueFrom:
                secretKeyRef:
                  name: server
                  key: secret_key_base
            - name: TOOLJET_HOST
              valueFrom:
                secretKeyRef:
                  name: server
                  key: tj_host
            - name: REDIS_HOST
              value: redis-service.default.svc.cluster.local
            - name: REDIS_PORT
              value: "6379"
            - name: TOOLJET_DB
              value: "tooljet_db"
            - name: TOOLJET_DB_USER
              value: "replace_with_postgres_database_user"
            - name: TOOLJET_DB_HOST
              value: "replace_with_postgres_database_host"
            - name: TOOLJET_DB_PASS
              value: "replace_with_postgres_database_password"
            - name: PGRST_HOST
              value: localhost:3002
            - name: PGRST_SERVER_PORT
              value: "3002"
            - name: PGRST_JWT_SECRET
              value: "replace_jwt_secret_here"
            - name: PGRST_DB_PRE_CONFIG
              value: postgrest.pre_config
            - name: PGRST_DB_URI
              value: postgres://TOOLJET_DB_USER:TOOLJET_DB_PASS@TOOLJET_DB_HOST:port/tooljet_db
            - name: PGRST_LOG_LEVEL
              value: "info"
            - name: DEPLOYMENT_PLATFORM
              value: "k8s"
---
apiVersion: v1
kind: Service
metadata:
  name: tooljet-service
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 3000
  selector:
    component: tooljet

Worker Deployment (tooljet-worker.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tooljet-worker
spec:
  replicas: 2  # Scale based on job queue depth
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      component: tooljet-worker
  template:
    metadata:
      labels:
        component: tooljet-worker
    spec:
      imagePullSecrets:
        - name: docker-secret
      containers:
        - name: tooljet-worker
          image: tooljet/tooljet:ee-lts-latest
          imagePullPolicy: Always
          args: ["npm", "run", "start:prod"]
          resources:
            limits:
              memory: "2000Mi"
              cpu: "2000m"
            requests:
              memory: "1000Mi"
              cpu: "1000m"
          # No ports - workers don't serve HTTP
          env:
            # Worker-specific environment variables
            - name: WORKER
              value: "true"
            - name: TOOLJET_WORKFLOW_CONCURRENCY
              value: "10"
            - name: REDIS_HOST
              value: redis-service.default.svc.cluster.local
            - name: REDIS_PORT
              value: "6379"
            # All other environment variables same as ToolJet app
            - name: PG_HOST
              valueFrom:
                secretKeyRef:
                  name: server
                  key: pg_host
            - name: PG_USER
              valueFrom:
                secretKeyRef:
                  name: server
                  key: pg_user
            - name: PG_PASS
              valueFrom:
                secretKeyRef:
                  name: server
                  key: pg_password
            - name: PG_DB
              valueFrom:
                secretKeyRef:
                  name: server
                  key: pg_db
            - name: LOCKBOX_MASTER_KEY
              valueFrom:
                secretKeyRef:
                  name: server
                  key: lockbox_key
            - name: SECRET_KEY_BASE
              valueFrom:
                secretKeyRef:
                  name: server
                  key: secret_key_base
            - name: TOOLJET_HOST
              valueFrom:
                secretKeyRef:
                  name: server
                  key: tj_host
            - name: TOOLJET_DB
              value: "tooljet_db"
            - name: TOOLJET_DB_USER
              value: "replace_with_postgres_database_user"
            - name: TOOLJET_DB_HOST
              value: "replace_with_postgres_database_host"
            - name: TOOLJET_DB_PASS
              value: "replace_with_postgres_database_password"
            - name: PGRST_HOST
              value: localhost:3002
            - name: PGRST_SERVER_PORT
              value: "3002"
            - name: PGRST_JWT_SECRET
              value: "replace_jwt_secret_here"
            - name: PGRST_DB_PRE_CONFIG
              value: postgrest.pre_config
            - name: PGRST_DB_URI
              value: postgres://TOOLJET_DB_USER:TOOLJET_DB_PASS@TOOLJET_DB_HOST:port/tooljet_db
            - name: PGRST_LOG_LEVEL
              value: "info"
            - name: DEPLOYMENT_PLATFORM
              value: "k8s"

Key Points:

ToolJet App: Serves HTTP traffic on port 3000, WORKER is unset (defaults to false)
Worker: Only processes jobs with WORKER=true, no ports exposed
Both deployments use the same secrets and database configuration
Worker has additional workflow-specific env var: TOOLJET_WORKFLOW_CONCURRENCY
Update REDIS_HOST to point to your deployed Redis service

Docker Compose Example

Docker Compose Configuration

version: '3.8'

services:
  tooljet:
    tty: true
    stdin_open: true
    container_name: Tooljet-app
    image: tooljet/tooljet:ee-lts-latest
    platform: linux/amd64
    restart: always
    env_file: .env
    ports:
      - 80:80
    depends_on:
      - postgres
      - redis
    environment:
      SERVE_CLIENT: "true"
      PORT: "80"
    command: npm run start:prod

  tooljet-worker-1:
    tty: true
    stdin_open: true
    platform: linux/amd64
    container_name: tooljet-worker-1
    image: tooljet/tooljet:ee-lts-latest
    restart: always
    env_file: .env
    depends_on:
      - postgres
      - redis
    environment:
      WORKER: "true"
      TOOLJET_WORKFLOW_CONCURRENCY: 10
      REDIS_HOST: redis
      REDIS_PORT: 6379
    command: npm run start:prod

  redis:
    image: redis:7
    volumes:
      - redis-data:/data
    command: redis-server --appendonly yes --maxmemory-policy noeviction

volumes:
  redis-data:

Key Points:

tooljet service: Web server with WORKER unset (defaults to false), serves HTTP on port 80
tooljet-worker-1 service: Dedicated worker with WORKER=true, no ports exposed
Both services use the same .env file for shared configuration
env_file: .env loads common environment variables (database credentials, secrets, etc.)
Environment-specific variables are set directly in the environment section
Redis configured with AOF persistence and noeviction policy

AWS ECS Example

ECS Task Definitions

Web Service Task Definition:

{
  "family": "tooljet-web",
  "containerDefinitions": [
    {
      "name": "tooljet",
      "image": "tooljet/tooljet:ee-lts-latest",
      "portMappings": [
        {
          "containerPort": 3000,
          "protocol": "tcp"
        }
      ],
      "environment": [
        {"name": "WORKER", "value": "false"},
        {"name": "REDIS_HOST", "value": "your-elasticache-endpoint"}
      ]
    }
  ]
}

Worker Service Task Definition:

{
  "family": "tooljet-worker",
  "containerDefinitions": [
    {
      "name": "tooljet-worker",
      "image": "tooljet/tooljet:ee-lts-latest",
      "portMappings": [],  // No ports needed
      "environment": [
        {"name": "WORKER", "value": "true"},
        {"name": "TOOLJET_WORKFLOW_CONCURRENCY", "value": "10"},
        {"name": "REDIS_HOST", "value": "your-elasticache-endpoint"}
      ]
    }
  ]
}

Worker Scaling Considerations

When to scale workers:

Queue depth consistently > 100 jobs
Job processing latency increases
Workflows timing out

Scaling strategies:

Horizontal: Add more worker replicas
Vertical: Increase TOOLJET_WORKFLOW_CONCURRENCY
Hybrid: Combine both approaches

Monitoring metrics:

Job completion time
Failed job count
Redis memory usage
Application logs

Monitoring and Troubleshooting

Common Issues

Workflows Not Executing

Symptoms: Workflows scheduled but not running

Solutions:

Check WORKER=true is set in at least one instance

Verify Redis connection:

# From ToolJet container
redis-cli -h $REDIS_HOST -p $REDIS_PORT ping

Check worker logs for errors
Verify maxmemory-policy noeviction in Redis

Jobs Failing Repeatedly

Symptoms: Workflow jobs fail repeatedly

Solutions:

Check application logs for error messages
Verify workflow node configurations
Check Redis memory usage (may be full)
Review WORKFLOW_TIMEOUT_SECONDS setting

Schedules Lost After Restart

Symptoms: Scheduled workflows don't trigger after restart

Solutions:

Check Schedule Bootstrap Service logs
Verify Redis persistence (AOF) is working
Confirm PostgreSQL connection is stable
Check Redis has sufficient memory

FAQ

Support

If you encounter issues during migration:

Community: Join our Slack Community
Email: [email protected]
GitHub Issues: Report bugs

Overview​

Why Migrate?​

Benefits of BullMQ-based Workflows​

Architecture Comparison​

How It Works​

Migration Steps​

1. Review Current Setup​

2. Set Up Redis​

3. Update Environment Variables​

4. Deploy Updated Configuration​

5. Remove Temporal Infrastructure​

6. Verify Migration​

Scaling Workflows with Dedicated Workers​

Why Dedicated Workers?​

Architecture​

Deployment Configuration​

Kubernetes Example​

Docker Compose Example​

AWS ECS Example​

Worker Scaling Considerations​

Monitoring and Troubleshooting​

Common Issues​

Workflows Not Executing​

Jobs Failing Repeatedly​

Schedules Lost After Restart​

FAQ​

Support​

Overview

Why Migrate?

Benefits of BullMQ-based Workflows

Architecture Comparison

How It Works

Migration Steps

1. Review Current Setup

2. Set Up Redis

3. Update Environment Variables

4. Deploy Updated Configuration

5. Remove Temporal Infrastructure

6. Verify Migration

Scaling Workflows with Dedicated Workers

Why Dedicated Workers?

Architecture

Deployment Configuration

Kubernetes Example

Docker Compose Example

AWS ECS Example

Worker Scaling Considerations

Monitoring and Troubleshooting

Common Issues

Workflows Not Executing

Jobs Failing Repeatedly

Schedules Lost After Restart

FAQ

Support