Add celery queues

[phette23]

Is your feature request related to a problem? Please describe.

Set a default --queues celery,low for the worker and worker-beat container commands to work with the new jobs scheduler. It probably makes sense to expose a parameter to configure these as well.

See also #218.

Describe the solution you'd like

1. --queues celery,low added to the worker-beat-deployment and worker-deployment templates
2. Related worker.queues and workerBeat.queues values to specify custom queues

Describe alternatives you've considered

I don't think you can specify celery queues with an environment variable (e.g. CELERY_QUEUES), otherwise this feature would essentially already be present with the worker.extraEnvVars parameter.

Additional context

This may be an InvenioRDM v13 feature, where invenio-jobs introduced the scheduler but also specified a new "low" priority queue. Without it, jobs sent to the "low" queue never complete. When I run v13 locally, the celery commands look like this:

celery --app invenio_app.celery worker --beat --events --loglevel INFO --queues celery,low

but the commands of this chart's running containers lack the final --queues flag:

# worker
celery -A invenio_app.celery worker -c 2 -l INFO
# beat
celery -A invenio_app.celery beat -l INFO -s /var/run/celery/celery-schedule --pidfile /var/run/celery/celerybeat.pid

[lindhe]

@phette23 does #225 resolve this issue or would you rather see some other solution?

[phette23]

I'm not sure what #225 does — this lets you customize the celery commands if you're using flower? We are not using flower right now, so it wouldn't help. Unless I'm missing something, it does not let you customize the commands per pod either (e.g. specify different command for worker, beat, scheduler).

[lindhe]

I'm clearly being stupid here. I went ahead a bit fast and only updated flower – sorry for that.

What I wanted to check with you is if it'd help to make it possible to use a custom command so celery can be stated with whatever flags you want. I thought that parameterizing the command would be easier to implement and more generic than introducing values only for the --queues flag.

Anyway – your suggestion sounds reasonable so if you'd rather see that than my more generic solution I'm happy to review a PR with those improvements.