DATABRICKS_HOST env variable is ignored when using CLI inside DAB directory

[josepmartinez-vista]

Describe the issue

When executing non-bundle commands (e.g., databricks current-user me) from within a directory containing a Databricks Asset Bundle (databricks.yml file exists), the CLI uses the bundle's default target of databricks.yml file over the explicitly defined DATABRICKS_HOST environment variable. That should never be used as we are running a non-bundle command.

Behavior:

• Non-DAB commands inside DAB folder: The CLI ignores DATABRICKS_HOST and attempts to connect to the default target defined in databricks.yml.
• Non-DAB commands outside DAB folder: The CLI correctly uses DATABRICKS_HOST.
• DAB commands: Correctly respects DATABRICKS_HOST defined.

Steps to reproduce the behavior

1. Create a databricks.yml file in the current directory with a dev target set as default:

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://dev-environment.cloud.databricks.com
  stg:
    mode: production
    workspace:
      host: https://stg-environment.cloud.databricks.com

2. Set environment variables to point to a different environment (Staging):

export DATABRICKS_HOST=https://stg-environment.cloud.databricks.com
export DATABRICKS_TOKEN=dapi_stg_token_here

3. Execute a non-bundle command with debug logging in DAB folder (where databricks.yml file exists):

databricks current-user me --debug

Expected Behavior

The CLI should use the environment variable DATABRICKS_HOST. The command should execute against the Staging environment.

Actual Behavior

The CLI detects the bundle, loads the dev target because it is marked as default: true, and attempts to authenticate against the Dev environment, resulting in an Invalid Token error (as the token provided is for Staging).

OS and CLI version

• OS: Linux (Docker python:3.11-alpine)
• Databricks CLI Version: v0.288.0

Is this a regression?

Unknown, but consistently reproducible since at least v0.283.0.

Debug Logs

The logs clearly show the SelectDefaultTarget mutator forcing the target to dev despite the environment variables:

$ cd databricks
$ databricks current-user me --debug
15:35:06 Info: start pid=44 version=0.288.0 args="databricks, current-user, me, --debug"
15:35:06 Debug: Found bundle root at /my_repo_root/databricks (file /my_repo_root/databricks/databricks.yml) pid=44
15:35:06 Info: Phase: load pid=44
15:35:06 Debug: Apply pid=44 mutator=EntryPoint
15:35:06 Debug: Apply pid=44 mutator=scripts.preinit
15:35:06 Debug: No script defined for preinit, skipping pid=44 mutator=scripts.preinit
15:35:06 Debug: Apply pid=44 mutator=ProcessRootIncludes
15:35:06 Debug: Apply pid=44 mutator=ProcessRootIncludes mutator=ProcessInclude(resources/my_job_a.job.yml)
15:35:06 Debug: Apply pid=44 mutator=ProcessRootIncludes mutator=ProcessInclude(resources/my_job_b.job.yml)
15:35:06 Debug: Apply pid=44 mutator=VerifyCliVersion
15:35:06 Debug: Apply pid=44 mutator=EnvironmentsToTargets
15:35:06 Debug: Apply pid=44 mutator=ComputeIdToClusterId
15:35:06 Debug: Apply pid=44 mutator=InitializeVariables
15:35:06 Debug: Apply pid=44 mutator=DefineDefaultTarget(default)
15:35:06 Debug: Apply pid=44 mutator=validate:unique_resource_keys
15:35:06 Debug: Apply pid=44 mutator=SelectDefaultTarget
15:35:06 Debug: Apply pid=44 mutator=SelectDefaultTarget mutator=SelectTarget(dev)
15:35:06 Debug: GET /api/2.0/preview/scim/v2/Me
< HTTP/2.0 400 Bad Request
< Invalid Token pid=44 sdk=true
15:35:06 Debug: non-retriable error: Invalid Token pid=44 sdk=true
Error: Invalid Token
15:35:06 Info: failed execution pid=44 exit_code=1 error="Invalid Token"
15:35:06 Debug: no telemetry logs to upload pid=44