Custom metadata headers with dots not transmitted to server on streaming calls over Unix sockets

[spendres]

Problem description

When making server-streaming gRPC calls over Unix sockets, custom metadata headers containing dots in the header name (e.g., workload.spiffe.io) are not transmitted to the server, even though they appear correctly in client-side metadata objects.

Expected Behavior

The server should receive the workload.spiffe.io: true header and process the request normally, returning X.509 SVID credentials.

Actual Behavior

The server responds immediately with:

gRPC status: 12 (UNIMPLEMENTED)
Message: "unknown service spiffe.workload.SpiffeWorkloadAPI"

This is the error SPIRE returns when the workload.spiffe.io header is missing (a security measure to prevent unauthorized access).

Reproduction steps

1. Set up a SPIRE agent listening on a Unix socket (e.g., /run/spire/sockets/agent.sock)
2. Create a gRPC client using @grpc/grpc-js:

import * as grpc from "@grpc/grpc-js";
import * as protoLoader from "@grpc/proto-loader";

const packageDefinition = protoLoader.loadSync("workload.proto", {
  keepCase: true,
  longs: String,
  enums: String,
  defaults: true,
  oneofs: true,
});

const protoDescriptor = grpc.loadPackageDefinition(packageDefinition) as any;
const SpiffeWorkloadAPI = protoDescriptor.spiffe.workload.SpiffeWorkloadAPI;

// Connect to Unix socket
const client = new SpiffeWorkloadAPI(
  "unix:/run/spire/sockets/agent.sock",
  grpc.credentials.createInsecure()
);

// Create metadata with required header
const metadata = new grpc.Metadata();
metadata.set("workload.spiffe.io", "true");

console.log("Metadata:", metadata.getMap());
// Output: { 'workload.spiffe.io': [ 'true' ] }

// Make streaming call
const call = client.FetchX509SVID({}, metadata);

call.on("error", (err) => {
  console.error("Error:", err.code, err.details);
});

call.on("data", (response) => {
  console.log("Response:", response);
});

Also attempted using low-level makeServerStreamRequest:

const metadata = new grpc.Metadata();
metadata.set("workload.spiffe.io", "true");

const call = client.makeServerStreamRequest(
  "/spiffe.workload.SpiffeWorkloadAPI/FetchX509SVID",
  (arg: any) => Buffer.alloc(0),
  (buf: Buffer) => deserializeResponse(buf),
  {},
  metadata,
  {}
);

Environment

• OS: Linux Alpine 3.23 docker container
• Node: version 24.12.0
• Node installation method: docker image node24-alpine
• @grpc/grpc-js version: 1.12.x (also tested with 1.14.3)
• Transport: Unix domain socket

Additional context

With GRPC_TRACE=all GRPC_VERBOSITY=DEBUG, the client logs show:

Calling FetchX509SVID with metadata:
  Keys: { 'workload.spiffe.io': [ 'true' ] }

D ... | subchannel_call | [3] Received server headers:
    :status: 200
    content-type: application/grpc
    grpc-status: 12
    grpc-message: unknown service spiffe.workload.SpiffeWorkloadAPI

Note that the debug output does not show the outgoing headers being sent, and the server immediately rejects the request.

Additional Attempts

All of the following approaches resulted in the same error:

Using metadata.add() instead of metadata.set()
Using interceptors to inject metadata
Using createFromMetadataGenerator call credentials (cannot combine with insecure channel credentials)
Using the generated client's method directly: client.FetchX509SVID({}, metadata)
Using low-level makeServerStreamRequest with explicit metadata parameter

Verification

The same SPIRE agent works correctly with:

Go clients using github.com/spiffe/go-spiffe/v2/workloadapi
The official spire-agent api fetch CLI command

This confirms the server is functioning correctly and the issue is specific to @grpc/grpc-js.

Hypothesis

The header name containing dots (workload.spiffe.io) may be getting filtered, normalized, or lost during transmission over Unix sockets. Alternatively, there may be an issue with how metadata is attached to server-streaming calls specifically.
Workaround
We implemented a Go sidecar that communicates with SPIRE and exposes credentials via a simple HTTP API, which the Node.js service consumes.

Proto Definition

protobufsyntax = "proto3";
package spiffe.workload;

message X509SVIDRequest {}

message X509SVIDResponse {
    repeated X509SVID svids = 1;
    map<string, bytes> federated_bundles = 3;
}

message X509SVID {
    string spiffe_id = 1;
    bytes x509_svid = 2;
    bytes x509_svid_key = 3;
    bytes bundle = 4;
    string hint = 5;
}

service SpiffeWorkloadAPI {
    rpc FetchX509SVID(X509SVIDRequest) returns (stream X509SVIDResponse);
}

Related

SPIFFE Workload API specification: https://github.com/spiffe/spiffe/blob/main/standards/SPIFFE_Workload_API.md
The workload.spiffe.io header requirement is documented in the SPIFFE Workload API spec as a security measure

[murgatroid99]

So far, I am unable to reproduce the behavior you describe in my own tests, with a Node client communicating with a Node server. Can you provide a minimal client and server pair that reproduces the problem?