Describe the bug
Hardcoded 5s timeout in ConnectionService too short for user-resources call
Guacamole login intermittently fails (generic error, or "requested connection does not exist") for users with broad role assignments. Authentication always succeeds; the failure happens when Guacamole calls the core API's user-resources endpoint to fetch the VM list.
This only occurs in our largest workspace (36 role assignments, 51 VMs) — other, smaller workspaces are unaffected.
Root cause: ConnectionService.java has a hardcoded 5-second client timeout (API_TIMEOUT_SECONDS) with no retry. Tracing a live failure showed the core API taking a ~6.3 second silent gap between fetching the resource list (51 items) and starting template lookups — no Cosmos, Graph, or other logged activity during the gap — which exceeds Guacamole's timeout. Cosmos itself is not the bottleneck: RU consumption was flat at 0% over 7 days, and every individual Cosmos call completed in ~20-30ms.
Separately, the template lookup loop issues one Cosmos query per resource rather than a single batched query, which adds latency that scales with resource count — a likely contributing factor, though it doesn't fully account for the 6s gap on its own.
Ruled out: authentication/RBAC, network/DNS/firewall, Cosmos throttling, Service Bus, Resource Processor, VM-specific faults, and the OAuth 4KB-cookie warning (present on both successful and failed attempts).
Steps tried
- Upgrade core app service to P1v3
Steps to reproduce
- In a workspace with a large number of VMs and/or role assignments (our case: 51 VMs, 36 role assignments), sign in as a user with TRE Administrator or cross-workspace roles
- Attempt to connect to a VM via Guacamole
- Compare Guacamole's AppServiceConsoleLogs (HttpTimeoutException in ConnectionService) against the core API's logs for the same timestamp — a multi-second gap with no logged activity appears between the resource-list fetch and the first template query
Azure TRE release version (e.g. v0.14.0 or main):
v0.27.0
Deployed Azure TRE components - click the (i) in the UI:
UI Version:
0.8.25
API Version:
0.25.13
Describe the bug
Hardcoded 5s timeout in ConnectionService too short for user-resources call
Guacamole login intermittently fails (generic error, or "requested connection does not exist") for users with broad role assignments. Authentication always succeeds; the failure happens when Guacamole calls the core API's
user-resourcesendpoint to fetch the VM list.This only occurs in our largest workspace (36 role assignments, 51 VMs) — other, smaller workspaces are unaffected.
Root cause:
ConnectionService.javahas a hardcoded 5-second client timeout (API_TIMEOUT_SECONDS) with no retry. Tracing a live failure showed the core API taking a ~6.3 second silent gap between fetching the resource list (51 items) and starting template lookups — no Cosmos, Graph, or other logged activity during the gap — which exceeds Guacamole's timeout. Cosmos itself is not the bottleneck: RU consumption was flat at 0% over 7 days, and every individual Cosmos call completed in ~20-30ms.Separately, the template lookup loop issues one Cosmos query per resource rather than a single batched query, which adds latency that scales with resource count — a likely contributing factor, though it doesn't fully account for the 6s gap on its own.
Ruled out: authentication/RBAC, network/DNS/firewall, Cosmos throttling, Service Bus, Resource Processor, VM-specific faults, and the OAuth 4KB-cookie warning (present on both successful and failed attempts).
Steps tried
Steps to reproduce
Azure TRE release version (e.g. v0.14.0 or main):
v0.27.0
Deployed Azure TRE components - click the (i) in the UI:
UI Version:
0.8.25
API Version:
0.25.13