System.InvalidOperationException: Invalid operation. The connection is closed

[davgia]

Describe the bug

I see errors in the microservice log like that every time the kubernetes probe hit the health actuator:

{
"Timestamp":"2026-08-21T09:59:28.0905597+00:00",
"Level":"Error","MessageTemplate":"{DbConnection} at {Host} is down!",
"TraceId":"8765302b12c15e1b97d25cda446c5b70",
"SpanId":"8a651d1a8168d356",
"Exception":"System.InvalidOperationException: Invalid operation. The connection is closed.\n   at Microsoft.Data.SqlClient.SqlCommand.EndExecuteReaderAsync(IAsyncResult asyncResult)\n   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)\n--- End of stack trace from previous location ---\n   at Steeltoe.Connectors.RelationalDatabaseHealthContributor.CheckHealthAsync(CancellationToken cancellationToken) in /_/src/Connectors/src/Connectors/RelationalDatabaseHealthContributor.cs:line 55"
,"Properties":{
"DbConnection":"SQL Server",
"Host":"[REDACTED]",
"SourceContext":"Steeltoe.Connectors.RelationalDatabaseHealthContributor",
"RequestId":"0HNNVCESL2QT7:00000007",
"RequestPath":"/actuator/health",
"ConnectionId":"0HNNVCESL2QT7",
"Scope":[" [export-helper-service,8765302b12c15e1b97d25cda446c5b70,8a651d1a8168d356,e5506861a3d62139,true] "]
,"ApplicationName":"example-service"
}
}

The strange thing is the following:

• kubernetes does not mark the service as unhealthy
• if I port forward and hit the actuator with curl, I receive:

# curl http://localhost:5002/actuator/health -v
* Host localhost:5002 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:5002...
* Established connection to localhost (::1 port 5002) from ::1 port 61055
* using HTTP/1.x
> GET /actuator/health HTTP/1.1
> Host: localhost:5002
> User-Agent: curl/8.21.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< Content-Type: application/vnd.spring-boot.actuator.v3+json
< Date: Fri, 21 Aug 2026 09:59:44 GMT
< Server: Kestrel
< Transfer-Encoding: chunked
<
{"status":"UP","components":{"config-server":{"status":"UNKNOWN","details":{"error":"No property sources found"}},"RabbitMQ":{"status":"UP","details":{"host":"rabbitmq","version":"4.2.5"}},"SQL Server":{"status":"UP","details":{"host":"test,1433","service":"alpha"}},"diskSpace":{"status":"UP","details":{"total":65200107520,"free":19472498688,"threshold":10485760,"path":"/app","exists":true}},"jasper-reports-service":{"status":"UP","description":"jasper-reports-service health check succeeded","details":{"status":"Up","host":"jasper-reports-service:8092"}},"livenessState":{"status":"UP"},"ping":{"status":"UP"},"readinessState":{"status":"UP"},"reportService-global":{"status":"UP","description":"reportService-global health check succeeded","details":{"status":"Up","host":"report-service"}}},"groups":["liveness","readiness"]}* 
Connection #0 to host localhost:5002 left intact

So I don't get it.

This is my code to inject sqlserver in the microservice:

public static IServiceCollection AddSqlServer(this IServiceCollection services, IConfiguration config, Action<ConnectorAddOptionsBuilder> addAction = null, bool validateConfiguration = true)
{
    //add connectors
    Steeltoe.Connectors.SqlServer.SqlServerServiceCollectionExtensions.AddSqlServer(services, config, addAction);
    //inject configuration
    services.Configure<MultiTenantSqlServerConfiguration>(config);
    //inject factory
    services.AddSingleton<ISqlServerHelperFactory, SqlServerHelperFactory>();

    if (validateConfiguration)
    {
        services.AddHostedService<SqlServerConfigurationValidatorService>();
    }

    return services;
}

This is the secret loaded by the service (with dummy credentials of course):

apiVersion: v1
stringData:
  steeltoe__client__rabbitmq__Default__connectionstring: amqp://test:test@rabbitmq
  steeltoe__client__sqlserver__alpha__connectionstring: Data Source=test,1433;Database=test;User Id=test;Password=test;MultipleActiveResultSets=True;Encrypt=true;TrustServerCertificate=true;
kind: Secret
metadata:
  name: export-helper-service
  namespace: alpha
type: Opaque

I have some custom classes, yes, but they're used only in my code, the contributor should not be influenced.

Steps to reproduce

Steps to reproduce the behavior:

1. create a microservice and add sqlserver capabilities
2. configure at least one connectionstring
3. everything seems up, but the log gives error

Expected behavior

A clear and concise description of what you expected to happen.

Environment (please complete the following information):

• Steeltoe Version [4.2.0]
• Platform: [Kubernetes 1.36.2)
• OS: [Linux]
• .NET Version [dotnet 10.0]

[TimHess]

It may help to know more about the probe setup you're using.

It does look like there's a very narrow race condition if you've got more than one probe hitting /actuator/health. The current implementation uses a single DbConnection that's opened and closed within a request, so if you're using the same endpoint for multiple probes one may hit CloseAsync at an inconvenient time for the other and produce that exception. The race condition should be fixed in the PR linked above.

K8s probes are also one of the reasons we offer health group support. You may want to consider using /actuator/health/liveness and /actuator/health/readiness instead of the root endpoint. As part of that consideration, make sure you want external connectivity to something like a db to be part of that liveness check at all (a db issue may cause endless useless restart-crash loops when the app itself is healthy and a restart won't help).

[davgia]

It may help to know more about the probe setup you're using.

It does look like there's a very narrow race condition if you've got more than one probe hitting /actuator/health. The current implementation uses a single DbConnection that's opened and closed within a request, so if you're using the same endpoint for multiple probes one may hit CloseAsync at an inconvenient time for the other and produce that exception. The race condition should be fixed in the PR linked above.

K8s probes are also one of the reasons we offer health group support. You may want to consider using /actuator/health/liveness and /actuator/health/readiness instead of the root endpoint. As part of that consideration, make sure you want external connectivity to something like a db to be part of that liveness check at all (a db issue may cause endless useless restart-crash loops when the app itself is healthy and a restart won't help).

Yes, I have omitted some information. Kubernetes probes use health groups, not /actuator/health. Kubernetes considers the pod healthy and ready, so the probe returns 200 for sure. It's strange I see the error in the log though.