Skip to content

Conversation

@snopoke
Copy link
Contributor

@snopoke snopoke commented Nov 7, 2025

This commit adds a complete guide demonstrating 4 different approaches to running Python async code within Celery tasks that use Django models.

What's Included

Documentation

  • RESEARCH.md: Detailed research on all approaches with references
  • IMPLEMENTATION_PLAN.md: Implementation details and success criteria
  • Root README.md: Comprehensive comparison and decision guide

Approach 1: asyncio.run() in Sync Tasks

  • Uses asyncio.run() to execute async code in standard Celery tasks
  • Requires connections.close_all() before async context
  • Simple and straightforward for basic async tasks
  • Complete Django project with demo tasks and documentation

Approach 2: Django's async_to_sync

  • Uses Django's official asgiref utilities
  • Automatic thread-local state management
  • Bidirectional sync/async conversion with sync_to_async
  • Complete Django project with multiple example patterns

Approach 3: Gevent Worker Pool

  • High-concurrency approach using green threads
  • No async/await syntax needed - uses standard blocking code
  • Excellent for I/O-bound tasks (100+ concurrent operations)
  • Requires monkey-patching and gevent worker pool
  • Complete Django project with concurrency demonstrations

Approach 4: Threading with Async Code

  • Maximum isolation using ThreadPoolExecutor
  • True parallel async operations in separate threads
  • Complex workflows with independent async operations
  • Complete Django project with advanced patterns

Key Features

Each approach includes:

  • Complete working Django application
  • Database models (APILog, Task)
  • Multiple example Celery tasks
  • Management command for running demos
  • Comprehensive README with pros/cons
  • Error handling examples
  • Production considerations

Quick Start

Each project can be run independently:

cd approach-X
uv sync
python manage.py migrate
celery -A myproject worker --loglevel=info
python manage.py run_demo

Decision Guide

  • Simple async tasks → Approach 1
  • Django integration → Approach 2
  • High I/O concurrency → Approach 3
  • Complex workflows → Approach 4

See root README.md for detailed comparison and recommendations.

This commit adds a complete guide demonstrating 4 different approaches
to running Python async code within Celery tasks that use Django models.

## What's Included

### Documentation
- RESEARCH.md: Detailed research on all approaches with references
- IMPLEMENTATION_PLAN.md: Implementation details and success criteria
- Root README.md: Comprehensive comparison and decision guide

### Approach 1: asyncio.run() in Sync Tasks
- Uses asyncio.run() to execute async code in standard Celery tasks
- Requires connections.close_all() before async context
- Simple and straightforward for basic async tasks
- Complete Django project with demo tasks and documentation

### Approach 2: Django's async_to_sync
- Uses Django's official asgiref utilities
- Automatic thread-local state management
- Bidirectional sync/async conversion with sync_to_async
- Complete Django project with multiple example patterns

### Approach 3: Gevent Worker Pool
- High-concurrency approach using green threads
- No async/await syntax needed - uses standard blocking code
- Excellent for I/O-bound tasks (100+ concurrent operations)
- Requires monkey-patching and gevent worker pool
- Complete Django project with concurrency demonstrations

### Approach 4: Threading with Async Code
- Maximum isolation using ThreadPoolExecutor
- True parallel async operations in separate threads
- Complex workflows with independent async operations
- Complete Django project with advanced patterns

## Key Features

Each approach includes:
- Complete working Django application
- Database models (APILog, Task)
- Multiple example Celery tasks
- Management command for running demos
- Comprehensive README with pros/cons
- Error handling examples
- Production considerations

## Quick Start

Each project can be run independently:
```bash
cd approach-X
uv sync
python manage.py migrate
celery -A myproject worker --loglevel=info
python manage.py run_demo
```

## Decision Guide

- Simple async tasks → Approach 1
- Django integration → Approach 2
- High I/O concurrency → Approach 3
- Complex workflows → Approach 4

See root README.md for detailed comparison and recommendations.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants