How to provide environment variables to command components (SDK V2)

[extremilio]

Hey!

We're migrating from SDK V1 to V2.
I have to say it's been a headache.

We use yaml command components to run/deploy our scripts with SDK v2.
The yaml command components are loaded into a pipeline_component, and then the pipeline_component runs in a job, using the Python V2 SDK.
The above works, the job and its components execute.

But, how do we set environment variables for these (yaml) components so that they are reachable at runtime?
Can we set it in the command component yaml? In the loaded component? In the pipeline? In the Environment? I've searched and tried almost everything by now, but to no avail.

There's even a forum question about it, where the mods are suggesting the commandComponent.schema.json supports it, which it doesn't:
https://learn.microsoft.com/en-us/answers/questions/2237919/how-to-pass-environment-variable-into-azureml-comp?comment=answer-2015022&page=1#comment-2360727

Side comment is that a lot of the documentaton on SDK V2 references the mldesigner package, which has dependencies to SDK v1 (azureml.core) - and from what I gather this is contraproductive since SDK v1 is out or support next summer? And not even supporting above Python 3.9?
At least when I've tried to use mldesigner, I get errors that I need to add azureml.core as well.

[xiangyan99]

Thanks for reaching out.

Could you please specify which library and version you are using?

[github-actions]

Hi @extremilio. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[extremilio]

Thanks for reaching out.

Could you please specify which library and version you are using?

Hi!

Azure ML SDK v2
azure-ai-ml 1.30.0

[VKuzina]

Is there any update on this issue? I am having the same problems, no idea how to pass env_variables to my component while creating a batch endpoint.

[grabowskanna]

I’m sharing the solution I found in case it helps others who run into the same issue — the key is to wrap the Command component with a Pipeline component and deploy Pipeline component to batch endpoint.

Pipeline defined via Python decorator

When defining a pipeline using the pipeline decorator, environment variables can be set directly on individual steps after instantiating the component:

component_1 = ml_client.components.get(name="component_1", version="1")
component_2 = ml_client.components.get(name="component_2", version="1")

@pipeline()
def my_pipeline(
    pipeline_job_input: Input,
):
    step_1 = component_1(input_data=pipeline_job_input)
    step_2 = component_2(input_data=step_1.outputs.output_data)

    step_1.environment_variables = {"YOUR_VAR": "value"}

    return {
        "pipeline_job_output": step_2.outputs.output_data,
    }

Pipeline component defined via YAML

When defining a pipeline component in YAML, environment variables can be specified per job in the jobs section:

$schema: https://azuremlschemas.azureedge.net/latest/pipelineComponent.schema.json
type: pipeline

name: train_pipeline_component

inputs:
  training_data:
    type: uri_folder

outputs:
  trained_model:

jobs:
  train_job:
    type: command
    component: ./train/train.yml
    inputs:
      training_data: ${{parent.inputs.training_data}}
    outputs:
      model_output: ${{parent.outputs.trained_model}}
    compute: ${{parent.inputs.train_node_compute}}
    environment_variables:
      YOUR_VAR: "value"

Hope this will help