Flow doesn't follow Queue's default job options

[hicaro]

The default job options set on the queue level are not obeyed when enqueuing jobs with the flow producer.

For instance, creating a queue like this:

const queue = new Queue('a-queue', {
  connection: redisClient,
  defaultJobOptions: {
    attempts: 3,
    delay: 500,
    removeOnComplete: 25,
    removeOnFail: 25,
    backoff: {
      type: 'exponential',
      delay: 5000,
    },
  },
  limiter: {
    groupKey: 'someId',
  },
});

The job dispatched by the flow producer will have the following properties:

{
  "attempts": 0,
  "delay": 0
}

Is there anything else I should be doing for it to work correctly?

[roggervalf]

hi @hicaro, for a flow producer, you could pass flowQueueOpts when adding a flow, that flowQueueOpts could handle the defaultJobOpts:
https://github.com/taskforcesh/bullmq/blob/master/docs/gitbook/api/bullmq.flowproducer.add.md
https://github.com/taskforcesh/bullmq/blob/master/docs/gitbook/api/bullmq.flowopts.md
https://github.com/taskforcesh/bullmq/blob/master/docs/gitbook/api/bullmq.flowqueuesopts.md

[vuhrmeister]

I just ran into the same issue.

Yes, I can pass the options into the job, but is that really the way to go? I'm expecting the job to always respect the default queue options. Why not for flows?

[manast]

@vuhrmeister ok, there seems to be some confusion here. So when you want to work with a flow you need to use the FlowProducer class to add new jobs to the Queue https://docs.bullmq.io/guide/flows
Let me know if this is the issue or I have misunderstood it.

[vuhrmeister]

Yes, I do use the FlowProducer and that works all fine as described.

My point is: Even the FlowProducer doesn't work for itself but uses defined queues to process a job, being the parent or the children. And queues are defined somewhere. And queues have some defaultJobOptions. And even a Flow job is a job which gets processed by the queue. And that's why I expect the executed job to respect the queues defaultJobOptions.

[manast]

@vuhrmeister could you provide a small piece of code that illustrates what you mean that can help me understand?

[hicaro]

@manast, I agree with @vuhrmeister. If the target queue in which the flow producer will place the job has default job options, can't those options be observed if none were provided when adding the job through the flow producer?

For instance, taking the queue from the initial example of this issue, if we add a job as (without the opts property):

await flowProducer.add({
  name: 'a-job',
  queueName: 'a-queue',
});

Why couldn't it use the default options as:

{
    attempts: 3,
    delay: 500,
    removeOnComplete: 25,
    removeOnFail: 25,
    backoff: {
      type: 'exponential',
      delay: 5000,
    },
}

[manast]

@hicaro can you write a complete example on what you expect? I still don't get what you mean.

[hicaro]

@manast take this as the queue:

const queue = new Queue('a-queue', {
  connection: redisClient,
  defaultJobOptions: {
    attempts: 3,
    delay: 500,
    removeOnComplete: 25,
    removeOnFail: 25,
    backoff: {
      type: 'exponential',
      delay: 5000,
    },
  },
});

What the point of this issue is allowing developers to be able to do this:

await flowProducer.add({
  name: 'a-job',
  queueName: 'a-queue',
});

Instead of this since we have a defaultJobOptions in the queue definition:

await flowProducer.add({
  name: 'a-job',
  queueName: 'a-queue',
  opts: {
    attempts: 3,
    delay: 500,
    removeOnComplete: 25,
    removeOnFail: 25,
    backoff: {
      type: 'exponential',
      delay: 5000,
    },
  }
});

[manast]

@hicaro queue in your example is just an instance unrelated to the flowProducer instance, the defaultJobOptions are not persisted in Redis or anything like that. It is impossible for the flowProducer to know about these settings.

[hicaro]

Okay, I was under the impression that those options would be persisted somewhere. Would it be possible to have the flow producer instance have its own default job options? It gets very repetitive to copy and paste the same options every time a new flow is created.

[manast]

Can't you just store the default options in a const and just use the same const on all the calls to flowProducer.add?

[manast]

@hicaro actually it is possible to pass default opts in the constructor to the FlowProducer by passing an object mapping the queue name to the options as @roggervalf pointed out earlier: #1034 (comment)

[vuhrmeister]

Thx @hicaro for posting some details. That's exactly my issue.

@manast For me it's fine passing the options to the flow options when calling flowProducer.add(). In my case children have a different queue due to different requirements on the backoff logic. That's why passing it to the constructor doesn't work for me.

Anyway, it's just confusing since you still run those jobs on a queue. It's still a job. And that's why it's not self explanatory that the default options on the queue doesn't count for the flow. So if it's really not possible then it should be properly documented. I'm sure that this doesn't only confuse me.

[hicaro]

@manast I see what you mean, but it is still tied to an individual call of .add, which also generates repetitive code. Ideally, if we had the same map provided into the FlowProducer constructor, we could set it and forget it.

I agree with @vuhrmeister, we might not be the only ones confused by this.