Cannot reach OpenSearch from Lambda

[bruderj15]

Hello.

I am operating Localstack Pro and deploy an OpenSearch instance and some Lambdas locally via Cloudformation.
Works great, but I cannot seem to connect to OpenSearch from within my Lambdas.

I studied localstack/localstack#5723 but cannot get the hang of it.
I feel like I tried everything under the sun.
I will share my setup and attempts below.

No idea what more to try.
I would appreciate help very much.
Thank you.

---

Cloudformation Snippet

  OpenSearchDomain:
    Type: AWS::OpenSearchService::Domain
    Properties:
      DomainName: "test-domain"
      EngineVersion: "OpenSearch_3.1"
      ClusterConfig:
        InstanceType: t3.small.search
        InstanceCount: 1
        ZoneAwarenessEnabled: false
      EBSOptions:
        EBSEnabled: true
        VolumeSize: 10
        VolumeType: gp3
      DomainEndpointOptions:
        CustomEndpointEnabled: true
        CustomEndpoint: "http://localhost:4566/test-domain"
        
  ProductMaterializeOpenSearchLambda:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: !Sub "product-lambda-materialize-opensearch-${StageName}"
      Runtime: provided.al2023
      Handler: lib.handler
      Role: !GetAtt ProductMaterializeOpenSearchRole.Arn
      Code:
        S3Bucket: !Ref ArtifactBucket
        S3Key: !Sub "product-lambda-materialize-opensearch-${StageName}-${CommitSHA}.zip"
      MemorySize: 512
      Timeout: 60
      EphemeralStorage:
        Size: 512

Notes

• Lambdas are successfully deployed and invoked
• I map random host-ports to localstack, for simplicity let's assume 4566
• OpenSearch is reachable from Host via http://localhost:4566/test-domain
• I start Localstack via Rust Testcontainers
• The closest attempts I got were able to reach Localstack from within the Lambda but die off with S3-Key not found exception: l.aws.handlers.logging : exception during call chain: Unable to find operation for request to service es: POST /test-domain/shops/_search

Attempts to reach OpenSearch from within Lambda

• http://host.docker.internal:4566/test-domain via "LAMBDA_DOCKER_FLAGS=--add-host=host.docker.internal:host-gateway"
• http://172.17.0.2:4566/test-domain where 172.17.0.2 was resolved from LOCALSTACK_HOST
• http://localhost.localstack.cloud:4566/eu-central-1/opensearch/test-domain aligned with Unable to Connect to localstack's OpenSearch with the Endpoint provided localstack#5723 (comment)
• http://test-domain.eu-central-1.es.localstack.cloud:4566
• http://test-domain.eu-central-1.opensearch.localstack.cloud:4566
• all above variants with port 9200 instead

---

[lethanix]

Hi @bruderj15! I'm with the LocalStack Support Team. I've been looking into this, but I'm having some trouble reproducing the issue on my end. Would you mind sharing LocalStack logs and/or a simple reproduction sample so I can take a closer look?