Healthcheck uses many threads

[thjarvin]

Fixes AB#57775, Fixes AB#57799

[haphut]

Good start! This PR aims to fix AB#57799 instead of AB#57775 but we could take out both bugs now.

I think unless we update to Java 21 to get Executors.newVirtualThreadPerTaskExecutor, Executors.newCachedThreadPool is a smart choice. However, we should use only one of them or it gets computationally expensive.

And the problematic line for AB#57775 is httpServer.setExecutor(Executors.newSingleThreadExecutor());.

Recommendations:

1. Create the ExecutorService as a private member. Use it both for httpServer and the checks.

   1. Shut it down in close:

      public void close() {
          if (httpServer != null) {
              httpServer.stop(0);
          }
          if (healthCheckExecutor != null) {
              healthCheckExecutor.shutdown();
              try {
                  if (!healthCheckExecutor.awaitTermination(3, TimeUnit.SECONDS)) {
                      healthCheckExecutor.shutdownNow();
                  }
              } catch (InterruptedException ie) {
                  healthCheckExecutor.shutdownNow();
                  Thread.currentThread().interrupt();
              }
          }
      }

2. ArrayList is not thread-safe. It is accessed from checkHealth and, I assume, by another thread via addCheck, removeCheck and clearChecks. Switch to CopyOnWriteArrayList for checks.

3. Optional: Checks could be traversed in order of completion to get to failure faster. Try ExecutorCompletionService for that.

4. As not all futures necessarily finish before finding failure, I would cancel the remaining futures:

   for (Future<Boolean> f : futures) {
       if (!f.isDone()) {
           f.cancel(true);
       }
   }

[haphut]

Oops, pushed suggestions early. I wanted to add at least a single test.

Great work regardless!

[haphut]

LGTM