VolumePopulator scheduling on the wrong node?

[remlabm]

Hello,

I'm having an issue that only recently started after upgrading to v0.14.0.

I have three nodes with the same OpenEBS-ZFS StorageClass. I have WaitForFirstConsumer set, but when scheduling the pod, the VolumePopulator will create the PVC/PVs on the wrong node. Even when forcing the node using nodeSelector, the VolumePopulator will schedule on a different node.

Similar to this issue:
#1255

Any ideas on why or where to start debugging this issue? I haven't tried to downgrade back to v0.13.1 yet.

[tesshuflower]

@remlabm unfortunately these can be a bit hard to debug. I can't think of anything specifically in v0.14.0 that would suddenly cause this to happen.

I guess from your description I'm assuming this stems from the fact that your volumesnapshots are not portable.

I think this can be an issue even if you created your PVC from snapshot (without volumepopulator being involved). With waitForFirstConsumer, the pod (which will mount the PVC) gets assigned to a node, and then it will get populated from the volumesnapshot - which may not work depending on what node was picked.

I'm not sure if there's some mechanism in the storage driver to get around this.

Having said all the above, it sounds like you are intentionally assigning the pod that will mount the PVC to a specific node which should be able to use the volumesnapshot?

Let me list some of the steps the volsync volumepopulator controller does, in case this helps (the PVC here is your volumesnapshot PVC):

• When the storage driver for the PVC is using waitForFirstConsumer, then the volume snapshot controller will not do anything with the PVC until it gets assigned to a node. That should happen after your pod gets created.
• Essentially after the pod gets created and assigned to a node (this is not done by the volume populator but normally the kube-scheduler), the PVC should get the annotation: volume.kubernetes.io/selected-node to indicate the node that has been selected.
• until the annotation is set on the PVC, the volume populator will not do anything further, in the VolSync log you may see entries something like this: "VolumeBindingMode is WaitForFirstConsumer, need to wait for nodeName annotation"
• only after the PVC gets this annotation will the volsync volume populator controller proceed and try to populate a PVC (on that assigned node) from the volumesnapshot.

So - wondering if perhaps your pod is actually not getting scheduled correctly? Can you recreate this same thing with the pod on its own (not using this particular PVC)? That might be good to see as a comparison.