prometheus improvements

[DaanHoogland]

problem

1. Give the exporter's HttpServer an explicit bounded executor (httpServer.setExecutor(Executors.newFixedThreadPool(2))) so one slow scrape can't serialize/queue all others.
2. Add a short TTL/in-flight guard around updateMetrics() (e.g., skip recompute if last run was < N seconds ago, or synchronize so concurrent scrapes share one in-progress computation) so scrape frequency can never multiply backend load.
3. Instrument: log/measure updateMetrics() wall-clock time so the reporter (and CI) can confirm which sub-metric collector is actually slow and verify the fix closes the growth.

additional comments:

1. Stale dynamic config (CONFIRMED) — capacity.calculate.workers is a runtime-dynamic setting, but the new shared executor only reads it once at first creation; live changes are silently ignored until a restart.
2. Swallowed capacity-recalculation abort (CONFIRMED per the extra verify pass) — shutdown racing an in-flight recalculation throws RejectedExecutionException, caught by the blanket catch(Throwable) in recalculateCapacity(), silently skipping storage/IP/VLAN updates for that cycle.
3. Unsynchronized race on _capacityExecutorService (CONFIRMED per the extra verify pass) — can leak a freshly-recreated pool that's never shut down again.
4. Shared fixed-size pool serializes previously-independent callers (PLAUSIBLE) — rolling-maintenance host-drain gating can now queue behind the hourly timer or API-triggered recalculations.
5. Pool no longer bounded to actual task count, so it can stay oversized/stale relative to fleet size (efficiency).
6. Bundling this executor-lifecycle rewrite into what the reported bug (Scrape duration using prometheus is increasing continuously until restart #13586) only needed a one-line fix for (altitude/scope creep).
7. Inconsistent lazy-vs-eager thread-pool lifecycle pattern within the same class (reuse/convention).
8. Minor: the synchronized getter is called per-loop-iteration instead of hoisted once (efficiency).

[PrashantBhanage]

@DaanHoogland , I'd like to work on this could you assign it to me?

[PrashantBhanage]

Opened #13678 with the three changes from the issue, scoped to the prometheus exporter plugin only. PTAL when you get a chance.

[DaanHoogland]

@PrashantBhanage , happy to assign you, but let’s make sure we don’t create code conflicts (see my question #13678 (comment))

[PrashantBhanage]

@PrashantBhanage , happy to assign you, but let’s make sure we don’t create code conflicts (see my question #13678 (comment))

@DaanHoogland I have switched the base branch of PR #13678 to your ghi13586-prometheusDrainage-20 branch, and GitHub confirms there are zero merge conflicts.

[RosiKyu]

@DaanHoogland could you please confirm that this is addressed with: #13696, instead of the originally linked closed PR: #13678 - so we can close this issue? Thank you!

[DaanHoogland]

@RosiKyu , #13650 needs to be merged for this.