arthas | Alibaba Java Diagnostic Tool Arthas/Alibaba Java诊断利器Arthas

The stack trace you have shown is 'situation normal': That is a threadpool executor thread that is ready to do work, but the queue of work is empty. In this case, 'waiting' means: I'm waiting for a job to do, not: "I have stuff to do, but cannot do because I am waiting for stuff to be finished first".

Now, 3000 threads is itself somewhat of a concern; each thread has its own stack space. How large that is depends on your -Xss parameter, but they tend to go from 64k to 1MB. If it's 1MB, that's... 3GB of stack space, that's... suboptimal. This number (how many threads you have waiting around for a job to accept) should also not be growing much after a VM has heated up.

If all/most of those WAITING threads have a similar trace, then there are really only two options:

• You made an executor and keep asking it, over time, to add more and more threads. I doubt this, but it's possible.
• You keep making executors. Don't do this.

The idea behind an executor is that you make only one, or at least very very few of these.

If you MUST create them as part of your running app (vs. the normal thing, of creating jobs and feeding them to the singleton executor), then be aware they are effectively resources: if you don't 'close' them, your process will require more and more resources, until eventually the VM will crash when it runs out.

To close them, you invoke shutdown() which is asking nicely, and shutdownNow() which is more aggressive and will any not-yet-picked-up jobs permanently undone.

So, to recap:

• You are making new executors during normal processing in your app. Search for new ScheduledThreadPoolExecutor in your codebase and inspect the situation. Add some logging if you must to see this in action.
• Then, most likely, you want to fix this and stop making new executors in the first place - just make one, once, and feed jobs into this one executor.
• If truly it makes sense to make them, use some guardian construct to ensure that you also clean them up when you're done using them. You can search for how to do so safely; it's a bit complicated, as you need to decide what to do with any jobs in the queue that are not yet done. If that's not an issue, it's easy: .shutdown() will get the job done.