Batch Processing

Process thousands of leads in parallel with real-time progress tracking and intelligent job management.

What is Batch Processing?

Batch processing allows you to run enrichment operations (validation, AI, scraping) on multiple leads simultaneously instead of one at a time. This dramatically speeds up large-scale operations.

How It Works

When you start a batch operation:

  1. Selected leads are queued for processing
  2. Multiple leads are processed in parallel (concurrent requests)
  3. Real-time progress updates via Server-Sent Events (SSE)
  4. Results are saved as each lead completes
  5. Job history tracks all operations

Real-Time Progress Tracking

Enrichabl uses Server-Sent Events to stream live updates as your batch processes:

  • Processed count: How many leads completed
  • Success rate: Percentage of successful operations
  • Error count: Number of failures
  • Estimated time: Time remaining (if available)
Stay on the page: You can navigate away during processing. Progress continues in the background, and you can check job history later.

Concurrent Processing Limits

Enrichabl respects API provider rate limits by controlling concurrency:

  • Email validation: Batch size depends on provider limits
  • AI enrichment: OpenAI rate limits (tokens per minute)
  • Web scraping: Firecrawl concurrent request limits

Optimizing Batch Size

  • Small batches (100-500): Test prompts and validate approach
  • Medium batches (500-5,000): Standard production runs
  • Large batches (5,000+): Split if hitting rate limits

Pausing and Resuming Jobs

You can pause long-running jobs and resume them later:

  1. Click Pause Job during processing
  2. Current requests complete, new ones are queued
  3. Resume anytime from Job History
Note: Pausing doesn't stop in-flight API requests. Already-started requests will complete.

Job History

Every batch operation is logged in Job History:

  • Operation type (validation, AI enrichment, scraping)
  • Start and end time
  • Total leads processed
  • Success and error counts
  • Error messages for failed leads

Viewing Failed Leads

Click on a job in history to see which leads failed and why. Common errors:

  • API rate limit: Wait and retry
  • Invalid API key: Check Settings → API Keys
  • Missing field: Lead doesn't have required data (e.g., website URL)
  • Provider timeout: Service was temporarily unavailable

Best Practices

Before Running Large Batches

  • Test on 10-20 leads first
  • Validate emails before expensive AI enrichment
  • Check API provider credits/limits
  • Save your pipeline (auto-saves, but good to check)

During Processing

  • Monitor progress for the first few minutes
  • Check for pattern of errors (indicates config issue)
  • You can navigate away - progress continues

After Completion

  • Review Job History for errors
  • Retry failed leads if errors were temporary
  • Export enriched data

Troubleshooting

Batch Processing Very Slow

Cause: API rate limits, complex AI prompts, or high load

Solution: Use faster AI models (GPT-3.5 vs GPT-4), reduce batch size, or spread over time

High Failure Rate

Cause: Missing data, invalid API keys, or provider issues

Solution: Check job history for error patterns, verify API keys, filter out leads missing required fields

Job Appears Stuck

Cause: Network issue or provider timeout

Solution: Refresh the page. If still stuck after 5 minutes, contact support.

Next Steps