[SPIKE] - Resolve P2P integration on PluginPreset Level

[uwe-mayer]

The current implementation of Plugin to Plugin integration #1426 resolves the references on the Plugin level.
We had some initial discussions about wether it makes sense to move the resolver to the PluginPreset.

Let's try and gather

• Limitations of the current implementation
• Limitations of resolving on PP
• Improvements with resolving on PP
• Implementation details and needed efforts

[IvoGoman]

One example that could benefit from earlier resolution of expressions would be this

Where I have a Thanos PluginPreset that configures it's GRPC hosts as such:

Thanos PluginPreset
spec:
  plugin:
    optionValues:
      - name: thanos.query.ingress.grpc.hosts
        expression: |
          - host: "thanos-grpc.obs.${global.greenhouse.metadata.region}.cloud.sap"
            paths:
              - path: /
                pathType: ImplementationSpecific

Then there is a Global Thanos PluginPreset, which requires to know all available stores:

# Thanos Global PluginPreset
spec:
  plugin:
    optionValues:
      - name: thanos.query.stores
        valueFrom:
          ref:
            kind: PluginPreset
            name: thanos-qa
            expression: ${spec.optionValues.filter(v, v.name == "thanos.query.ingress.grpc.hosts")[0].value.map(v,v.host+":443")}

Here we have a couple issues:

1. The targeted Plugin has an expression instead of a value, which makes the CEL expression considerably harder.
2. The targeted Plugin has an expression, so during evaluation of the valueFromRef the actual value of the expression is not known.

Resolving the CEL expressions during the reconciliation of the PluginPreset, will make this possible.
In that case the actual Plugin created from Thanos PluginPreset would have an value instead of the expression:

spec:
  optionValues:
  - value:
      - host: "thanos-grpc.obs.eu-west-123.cloud.dev"
        paths:
          - path: /
            pathType: ImplementationSpecific
    name: thanos.query.ingress.grpc.hosts

This would on the one hand make the integration easier/possible if one Plugin needs a value that is configured as an expression.
Also this would make it easier for the endusers, since they now can see the actual configuration of their Plugin in-Cluster. Beforehand, this was only possible on the Flux HelmRelease or on the target cluster.

There are also some downsides to this approach:

• change to a targeted Plugin, results in reconciliation of the PluginPreset, requiring fetching all Clusters, Plugins, the targeted PluginPreset and all Plugins
• a standalone Plugin would still require the resolving of CEL expressions, in order to integrate with other Plugins. Otherwise, this feature would be exclusive to PluginPresets.