NetQueue 🚦

Smart, resilient CLI for developer network tasks

NetQueue is a developer-focused command runner that understands network instability.
It queues, retries, resumes, and orchestrates shell commands intelligently — even when the Internet is unreliable, filtered, or temporarily unavailable.

---

✨ Why NetQueue?

Modern developer workflows break easily when the network is unstable:

• Automatically queue & retry commands when offline or errors occur
• Resume large downloads without restarting from zero
• Handle proxies & VPNs seamlessly
• Orchestrate dependent tasks & flows like a mini CI system
• Get smart explanations & auto-fix suggestions for failures

NetQueue solves this.

---

🔑 Key Features

• ✅ Offline-aware persistent command queue
• 🔁 Automatic retry with exponential backoff
• 🌐 Multi-layer connectivity detection (DNS / TCP / HTTP)
• 🧠 Smart executors (Git-aware, shell-aware)
• 📥 Resumable downloads (pause / resume)
• 🧩 Dependency graph (DAG) & YAML flows
• 🪝 Lifecycle hooks (pre / post / retry / success)
• 🧾 Structured logs (JSON & text)
• 📦 Export / Import queued tasks
• 🔌 Proxy & VPN support (manual + auto-switch)
• 🧪 Dry-run & Explain mode

---

📦 Installation

Requirements

• Go 1.21+

Build from source

git clone https://github.com/mousav1/netqueue
cd netqueue
go build -o netqueue ./cmd/netqueue
./netqueue --help

On Windows, the binary will be netqueue.exe.

---

🚀 Quick Start

Run a command (queue if offline)

netqueue run "curl https://example.com"

Behavior:

• ✅ Online → runs immediately
• ❌ Offline → queued automatically when Internet is not usable
• ⚠️ Network error → queued & retried according to profile retry policy

You can also force smart detection and profiles:

netqueue run --smart "git push origin main"
netqueue run --profile git "git push origin main"

---

🧠 Smart Execution (Git-aware)

NetQueue automatically detects Git commands and applies a specialized execution policy.

netqueue run --smart "git push -u origin main"
# or
netqueue smart "git push -u origin main"

Smart behavior:

• Detects common Git network failures (exit 128, SSL, timeout, DNS)
• Queues on failure instead of losing progress
• Retries automatically when connectivity is restored (via daemon)

---

🧪 Dry-run & Explain Mode

Preview what NetQueue would do — without side effects.

netqueue run --dry-run "git push origin main"
netqueue explain "git push origin main"
netqueue run --dry-run --explain "git push origin main"

Explain output includes:

• Selected profile and whether it was auto-detected (smart) or explicit
• Connectivity requirements and level (DNS/TCP/HTTP)
• Retry and backoff policy
• Error classification rules
• Hooks that would trigger
• Final decision: run_now | queue | blocked

---

📥 Resumable Downloads

Download large files safely across disconnects.

netqueue download https://example.com/file.zip \
  --output ~/Downloads/file.zip

How it works

• Saves byte offset + metadata in SQLite
• Pauses automatically on disconnect
• Resumes from last byte on reconnect

---

🗂 Queue Management

netqueue queue list
netqueue queue list --long
netqueue queue remove <id>
netqueue queue show <id>  --format json
netqueue queue run <id> [<id>...]

Add tasks manually

netqueue queue add "git push origin main"
netqueue queue add "npm publish" --delay 60
netqueue queue add "npm publish" --after push

Flags:

• --at RFC3339 timestamp to schedule exact time
• --delay seconds from now
• --id custom task id
• --after dependency task ids (all must succeed)

Add downloads

netqueue queue add-download https://example.com/file.zip \
  --output ~/file.zip
netqueue queue add-download https://example.com/file.zip --output ~/Downloads/file.zip --at 2026-01-02T12:00:00Z
netqueue queue add-download https://example.com/file.zip --output ~/Downloads/file.zip --id dl1 --after push

Flags:

• --output required output file path
• --at RFC3339 schedule time
• --delay seconds from now
• --id custom task id
• --after dependency task ids

---

🧹 Queue State Management

Safely manage queued task history and recover from failures.

Clean (Delete Tasks)

netqueue queue clean --status succeeded
netqueue queue clean --status failed --older-than 7d
netqueue queue clean --all --confirm

Safety:

• running tasks are never deleted unless --force is passed
• --all requires --confirm
• Defaults refuse dangerous actions without filters

Reset State (failed → pending)

netqueue queue reset --status failed
netqueue queue reset --id <task_id>

Effect:

• status → pending
• attempt → 0
• next_run → now
• last_error → NULL

Retry Reset (only retry counter)

netqueue queue retry-reset --status failed

Effect:

• attempt → 0
• status unchanged

Cancel Pending Tasks

netqueue queue cancel --status pending
netqueue queue cancel --flow release

Effect:

• status → failed
• reason → cancelled_by_user
• exit_code → 130 (emitted in logs)

Archive Tasks (Export + Remove)

Export matched tasks (with dependencies) to a JSON file, then remove them safely.

netqueue queue archive --status succeeded --older-than 7d --output ~/Downloads/netqueue-archive.json
netqueue queue archive --status succeeded --older-than 7d --dry-run

Notes:

• Writes one JSON object per line (task fields + dependency ids)
• Refuses archiving running tasks unless --force
• --output optional; defaults to ~/.netqueue/archive-YYYYMMDDTHHMMSS.json
• Archive + deletion happens in a single SQL transaction

Shared Flags

All commands support:

--status <pending|running|succeeded|failed|queued|blocked>
--id <task_id>
--flow <flow_id>
--older-than <duration>   # e.g. 24h, 7d
--dry-run
--explain
--force
--confirm
--output <file>           # only for 'archive'

--older-than accepts Go-style durations (e.g., 24h) and day suffix (e.g., 7d).

Dry-run

No database changes. Prints affected count and filters:

Dry-run:
  matched_tasks=4
  action=delete
  filters=status=succeeded

Explain Mode

Explains decision logic in SQL-like terms:

Action: reset
Reason:
  status=failed
  next_run=now
  attempt=0

Edge Cases & Safety

• Refuses to delete running tasks unless --force
• Refuses --all without --confirm
• Returns clear errors on invalid state combinations
• Idempotent operations for safe repeated usage

---

🟢 System status:

netqueue status

Example output:

Connectivity: usable
Queue: pending=<n> running=<n> succeeded=<n> failed=<n>
Recovered running tasks: <n>   # printed on first run after recovery
Daemon: running|stopped

---

🔄 Background Daemon

Run tasks automatically when conditions are met.

netqueue daemon start
netqueue daemon stop

Daemon responsibilities

• Executes tasks when Internet is usable
• Respects dependencies (DAG)
• Applies retry & backoff
• Graceful shutdown on signals
• Auto-resumes interrupted running tasks (see below)

Auto-Resume Running Tasks

NetQueue automatically recovers tasks stuck in running after crashes, restarts, or power loss.

• On daemon startup (and when you run netqueue status), any stale running task is reset to pending with attempt incremented and next_run=now
• A task is stale if last_heartbeat is older than max(2*heartbeat_interval, 15s); heartbeat is updated every 5s while a task executes
• Safety:
  • Tasks with a fresh heartbeat are not recovered
  • Recovery is idempotent and transactional
  • Running tasks record worker_id; recovered tasks clear both worker_id and last_heartbeat
• Visibility:
  • netqueue status prints Recovered running tasks: N on the first run after recovery
  • netqueue logs shows structured events of type task_recovered

---

🧩 Profiles / Presets

Profiles define execution policy, not the command itself.

• Network requirements
• Retry strategy
• Error classification
• Hooks
• Timeouts

📁 Location:

By default, NetQueue looks for user profiles in:

~/.netqueue/profiles/*.yml

These are merged with builtin profiles embedded in the binary.

Example: Git profile

name: git
match:
  command_prefix: [git, sudo git]
network:
  required: true
  min_level: http
retry:
  strategy: exponential
  max_attempts: 5
errors:
  retry_on:
    - regex:connection timed out
    - regex:could not resolve host
  fail_fast_on:
    - authentication failed

---

🪝 Hooks (Lifecycle Automation)

Hooks are shell commands triggered on task lifecycle events.

Supported events

• pre_run
• post_run
• on_success
• on_fail
• on_retry
• on_queue

Example

hooks:
  pre_run:
    - echo "Starting $NETQUEUE_COMMAND"
  on_success:
    - notify-send "Task succeeded"

Hooks run:

• In isolated processes
• With timeouts
• Without affecting task success

---

🧬 Dependency Graph (DAG) & Flows

Define complex workflows using YAML.

Example flow

flow: release
tasks:
  - id: git_push
    run: git push origin main
    profile: git

  - id: npm_publish
    run: npm publish
    after: [git_push]

Add flow:

netqueue flow add release.yml

---

🧾 Structured Logs

NetQueue produces machine-readable logs.

netqueue logs
netqueue logs --format json
netqueue logs --task <id>
netqueue logs --flow <flow_id>
netqueue logs --status succeeded
netqueue logs --follow

JSON fields:

• timestamp
• task_id
• flow_id
• profile
• command
• status
• attempt
• exit_code
• reason
• action
• type
• affected
• filters

---

📤 Export / Import

Backup, migrate, or share your queue.

netqueue export --format json --output tasks.json
netqueue import tasks.json --merge

Supports:

• JSON / YAML
• Merge / Overwrite
• Dry-run validation

---

🌐 Proxy & VPN Support

netqueue run --proxy myproxy "curl ifconfig.me"
netqueue run --vpn workvpn "ssh internal-server"

Auto-switch

If multiple proxies are configured, NetQueue can rotate them automatically on failure.

---

⚙️ Configuration

Configuration priority:

Environment Variables
  > .env
    > config.yaml
      > Defaults

Generate config

netqueue config init
netqueue env init

Supports:

• Portable installs
• Dev / Prod separation
• Temporary overrides

---

🧪 Testing

go test ./...
go test -cover ./...

📄 License

MIT