ExecuterService.newFixedThreadPool의 메모리 릭 위험

hreadPoolExecutor는 memory leak 이슈가 있습니다.

https://blog.tier1app.com/2014/09/30/memory-leak-in-java-executor/

 

newFixedThreadPool 공식자료

공식 자료에 의하면 newFixedThreadPool은 shutdown 하지 않으면 pool내 스레드는 존재 한다고 합니다.

제가 사용하는 서비스에서 shutdown을 하지 않아 pool내 스레드가 parking상태로 남아있게 되고, 이 스레드가 쌓이게 되면서 메모리 누수가 발생하여 OutOfMemory 장애가 발생한 적이 있습니다. 

 

catalina.out 의 java.lang.OutOfMemoryError: unable to create new native thread 에러 와 hs_err_pid file 이 생성되었습니다.
tomcat옵션에 outofmemoryerror가 발생 했을 경우 heapdump file을 생성하는 옵션을 줬지만,

heapdump file을 생성하기 이전에 tomcat이 죽고, JVM crash 로 생성되는 java core dump log가 떨어지는 현상이 발생했습니다. (참고 : https://d2.naver.com/helloworld/1134732 )

 

hs_err.pid file

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
#   JVM is running with Unscaled Compressed Oops mode in which the Java heap is
#     placed in the first 4GB address space. The Java Heap base address is the
#     maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
#     to set the Java Heap base and to place the Java Heap above 4GB virtual address.
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2749), pid=34109, tid=0x00007f599cca0700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build 1.8.0_202-b08)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
 
---------------  T H R E A D  ---------------
 
Current thread (0x00007f6208254800):  JavaThread "pool-1016-thread-29" [_thread_new, id=24395, stack(0x00007f599cba0000,0x00007f599cca1000)]
 
Stack: [0x00007f599cba0000,0x00007f599cca1000],  sp=0x00007f599cc9fad0,  free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xad33a5]  VMError::report_and_die()+0x2e5
V  [libjvm.so+0x4e04c7]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x67
V  [libjvm.so+0x90fe86]  os::pd_commit_memory(char*, unsigned long, bool)+0xf6
V  [libjvm.so+0x90774f]  os::commit_memory(char*, unsigned long, bool)+0x1f
V  [libjvm.so+0x9113c8]  os::pd_create_stack_guard_pages(char*, unsigned long)+0x48
V  [libjvm.so+0xa7bc0e]  JavaThread::run()+0x1ee
V  [libjvm.so+0x90d8c2]  java_start(Thread*)+0x102
C  [libpthread.so.0+0x7dd5]  start_thread+0xc5
 
 
---------------  P R O C E S S  ---------------
 
Java Threads: ( => current thread )
  0x00007f620825a800 JavaThread "pool-1016-thread-32" [_thread_blocked, id=24398, stack(0x00007f599c89d000,0x00007f599c99e000)]
  0x00007f6208258800 JavaThread "pool-1016-thread-31" [_thread_blocked, id=24397, stack(0x00007f599c99e000,0x00007f599ca9f000)]
  0x00007f6208256800 JavaThread "pool-1016-thread-30" [_thread_new, id=24396, stack(0x00007f599ca9f000,0x00007f599cba0000)]
=>0x00007f6208254800 JavaThread "pool-1016-thread-29" [_thread_new, id=24395, stack(0x00007f599cba0000,0x00007f599cca1000)]
  0x00007f6208252800 JavaThread "pool-1016-thread-28" [_thread_new, id=24394, stack(0x00007f599cca1000,0x00007f599cda2000)]
  0x00007f6208250800 JavaThread "pool-1016-thread-27" [_thread_blocked, id=24393, stack(0x00007f599cda2000,0x00007f599cea3000)]
  0x00007f620824e000 JavaThread "pool-1016-thread-26" [_thread_new, id=24392, stack(0x00007f599cea3000,0x00007f599cfa4000)]
  0x00007f620824c000 JavaThread "pool-1016-thread-25" [_thread_blocked, id=24391, stack(0x00007f599cfa4000,0x00007f599d0a5000)]

catalina.out 

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home1/irteam/hs_err_pid34109.log
Jan 23, 2020 10:43:24 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [appServlet] in context with path [] threw exception [Handler processing failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread] with root cause
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:717)
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
        at com.nbp.infraeve.hbase.HBaseAdapter.read(HBaseAdapter.java:205)
        at com.nbp.infraeve.service.HbaseReadServiceImpl.hbaseReadPerformData(HbaseReadServiceImpl.java:47)
        at com.nbp.infraeve.service.InfraServiceImpl.selectHostPerformResouceInfo(InfraServiceImpl.java:1016)
        at com.nbp.infraeve.service.InfraServiceImpl$$FastClassBySpringCGLIB$$40702289.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:629)
        at com.nbp.infraeve.service.InfraServiceImpl$$EnhancerBySpringCGLIB$$e3c18c1b.selectHostPerformResouceInfo(<generated>)
        at com.nbp.infraeve.controller.HomeController.server_group_resource(HomeController.java:3172)
        at sun.reflect.GeneratedMethodAccessor550.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:953)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:855)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:829)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:479)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1471)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

 

문제 (장애)의 원인 

ExecutorService pool = Executors.newFixedThreadPool(POOL_SIZE);

사용되고 있는 Executors.newFixedThreadPool method는 ThreadPoolExecutor 객체를 생성하고, ThreadPoolExecutor 생성자 내부에는 Executors.defaultThreadFactory() 를 매개변수로 전달합니다.

/**
 * Creates a thread pool that reuses a fixed number of threads
 * operating off a shared unbounded queue.  At any point, at most
 * {@code nThreads} threads will be active processing tasks.
 * If additional tasks are submitted when all threads are active,
 * they will wait in the queue until a thread is available.
 * If any thread terminates due to a failure during execution
 * prior to shutdown, a new one will take its place if needed to
 * execute subsequent tasks.  The threads in the pool will exist
 * until it is explicitly {@link ExecutorService#shutdown shutdown}.
 *
 * @param nThreads the number of threads in the pool
 * @return the newly created thread pool
 * @throws IllegalArgumentException if {@code nThreads <= 0}
 */
public static ExecutorService newFixedThreadPool(int nThreads) {
    return new ThreadPoolExecutor(nThreads, nThreads,
                                  0L, TimeUnit.MILLISECONDS,
                                  new LinkedBlockingQueue<Runnable>());
}

 

/**
 * Creates a new {@code ThreadPoolExecutor} with the given initial
 * parameters and default thread factory and rejected execution handler.
 * It may be more convenient to use one of the {@link Executors} factory
 * methods instead of this general purpose constructor.
 *
 * @param corePoolSize the number of threads to keep in the pool, even
 *        if they are idle, unless {@code allowCoreThreadTimeOut} is set
 * @param maximumPoolSize the maximum number of threads to allow in the
 *        pool
 * @param keepAliveTime when the number of threads is greater than
 *        the core, this is the maximum time that excess idle threads
 *        will wait for new tasks before terminating.
 * @param unit the time unit for the {@code keepAliveTime} argument
 * @param workQueue the queue to use for holding tasks before they are
 *        executed.  This queue will hold only the {@code Runnable}
 *        tasks submitted by the {@code execute} method.
 * @throws IllegalArgumentException if one of the following holds:<br>
 *         {@code corePoolSize < 0}<br>
 *         {@code keepAliveTime < 0}<br>
 *         {@code maximumPoolSize <= 0}<br>
 *         {@code maximumPoolSize < corePoolSize}
 * @throws NullPointerException if {@code workQueue} is null
 */
public ThreadPoolExecutor(int corePoolSize,
                          int maximumPoolSize,
                          long keepAliveTime,
                          TimeUnit unit,
                          BlockingQueue<Runnable> workQueue) {
    this(corePoolSize, maximumPoolSize, keepAliveTime, unit, workQueue,
         Executors.defaultThreadFactory(), defaultHandler);
}


​

Executors.defaultThreadFactory() 는 기본적으로 비데몬 쓰레드로 생성합니다.

/**
 * The default thread factory
 */
static class DefaultThreadFactory implements ThreadFactory {
    private static final AtomicInteger poolNumber = new AtomicInteger(1);
    private final ThreadGroup group;
    private final AtomicInteger threadNumber = new AtomicInteger(1);
    private final String namePrefix;

    DefaultThreadFactory() {
        SecurityManager s = System.getSecurityManager();
        group = (s != null) ? s.getThreadGroup() :
                              Thread.currentThread().getThreadGroup();
        namePrefix = "pool-" +
                      poolNumber.getAndIncrement() +
                     "-thread-";
    }

    public Thread newThread(Runnable r) {
        Thread t = new Thread(group, r,
                              namePrefix + threadNumber.getAndIncrement(),
                              0);
        if (t.isDaemon())
            t.setDaemon(false);
        if (t.getPriority() != Thread.NORM_PRIORITY)
            t.setPriority(Thread.NORM_PRIORITY);
        return t;
    }
}

if (t.isDaemon()) t.setDaemon(false);

• • • 스레드풀의 스레드는 기본적으로 데몬 스레드가 아니기 때문에, main 스레드가 종료되더라도 스레드풀의 스레드는 작업을 처리하기 위해 계속 실행되므로 애플리케이션은 종료되지 않음

 

https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Executors.html#newFixedThreadPool(int)

Executors (Java Platform SE 7 )

 

기존 프로젝트에서는 newFixedThreadPool을  shutdown 하는 부분이 구현되어있지 않아서 발생한 이슈였으며, thread의 종료후 shutdown을 시켜주는 함수를 호출하면서 해결되었습니다. 
VisualVM을 사용하여 모니터링 하였으며 Thread생성을 급증 시키고, Thread가 종료되었을 때 반납이 되었는지 확인하는 작업으로 테스트를 진행하였습니다. 
https://seonnn.tistory.com/20?category=907341