тупик c3p0 на RefurbishCheckinResourceTask, getConnection ()

У меня есть веб-приложение, развернутое на tomcat6, есть функция импорта сообщений, которая импортирует большое количество сообщений, сохраняя их в базе данных Oracle 11g.

Я получаю deadlock изредка / непредсказуемо после некоторого запуска импорта.

След:

7 août 2014 13:04:38 com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
ATTENTION: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@637d7dd6 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
7 août 2014 13:04:38 com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
ATTENTION: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@637d7dd6 -- APPARENT DEADLOCK!!! Complete Status: 
    Managed Threads: 8
    Active Threads: 8
    Active Tasks: 
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@55e4ef4e (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@45460ed4 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#3)
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@49da1926 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#7)
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@14bc659d (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@78597c06 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#4)
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@6667f88e (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#5)
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@2a5d8dbb (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#6)
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@441364ad (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
    Pending Tasks: 
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@7ca35537
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@1370b3e5
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@9d7dc72
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@14bfd04d
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@3d0479df
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@27310c3a
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@519c7f0
Pool thread stack traces:
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:562)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#3,5,main]
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:562)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#7,5,main]
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:562)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:562)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#4,5,main]
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:562)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#5,5,main]
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:562)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#6,5,main]
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:562)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:562)


7 août 2014 13:05:13 com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
ATTENTION: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@637d7dd6 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
7 août 2014 13:05:13 com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
ATTENTION: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@637d7dd6 -- APPARENT DEADLOCK!!! Complete Status: 
    Managed Threads: 8
    Active Threads: 8
    Active Tasks: 
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@62ac4df7 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@e8fdc6f (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@4140546 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#3)
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@70a25e6b (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#7)
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@33e4fc92 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@4a6064a4 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#4)
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@466c6e60 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#6)
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@27e87047 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#5)
    Pending Tasks: 
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@bdba1dc
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@7be684ff
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@b9b2923
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@292c79d8
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@26f36906
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@114d8aa
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@7bf6e678
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@5c447da5
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@14950af6
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@3124219d
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@7a92974e
Pool thread stack traces:
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
        org.netbeans.lib.profiler.server.ProfilerRuntimeMemory.getAndSendCurrentStackTrace(ProfilerRuntimeMemory.java:221)
        org.netbeans.lib.profiler.server.ProfilerRuntimeObjLiveness.traceObjAlloc(ProfilerRuntimeObjLiveness.java:281)
        com.mchange.v2.c3p0.impl.NewProxyConnection.<init>(NewProxyConnection.java:32)
        com.mchange.v2.c3p0.impl.NewProxyConnection.<init>(NewProxyConnection.java:1376)
        com.mchange.v2.c3p0.impl.NewPooledConnection.getConnection(NewPooledConnection.java:151)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishResourceOnCheckin(C3P0PooledConnectionPool.java:285)
        com.mchange.v2.resourcepool.BasicResourcePool.attemptRefurbishResourceOnCheckin(BasicResourcePool.java:1615)
        com.mchange.v2.resourcepool.BasicResourcePool.access$200(BasicResourcePool.java:32)
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask.run(BasicResourcePool.java:1228)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:557)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
        org.netbeans.lib.profiler.server.ProfilerRuntimeMemory.getAndSendCurrentStackTrace(ProfilerRuntimeMemory.java:221)
        org.netbeans.lib.profiler.server.ProfilerRuntimeObjLiveness.traceObjAlloc(ProfilerRuntimeObjLiveness.java:281)
        com.mchange.v2.c3p0.impl.NewProxyConnection.<init>(NewProxyConnection.java:32)
        com.mchange.v2.c3p0.impl.NewProxyConnection.<init>(NewProxyConnection.java:1376)
        com.mchange.v2.c3p0.impl.NewPooledConnection.getConnection(NewPooledConnection.java:151)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishResourceOnCheckin(C3P0PooledConnectionPool.java:285)
        com.mchange.v2.resourcepool.BasicResourcePool.attemptRefurbishResourceOnCheckin(BasicResourcePool.java:1615)
        com.mchange.v2.resourcepool.BasicResourcePool.access$200(BasicResourcePool.java:32)
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask.run(BasicResourcePool.java:1228)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:557)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#3,5,main]
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask.run(BasicResourcePool.java:1229)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:557)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#7,5,main]
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask.run(BasicResourcePool.java:1229)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:557)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
        org.netbeans.lib.profiler.server.ProfilerRuntimeMemory.getAndSendCurrentStackTrace(ProfilerRuntimeMemory.java:221)
        org.netbeans.lib.profiler.server.ProfilerRuntimeObjLiveness.traceObjAlloc(ProfilerRuntimeObjLiveness.java:281)
        java.util.HashMap.addEntry(Unknown Source)
        java.util.HashMap.put(Unknown Source)
        java.util.HashSet.add(Unknown Source)
        com.mchange.v2.c3p0.util.ConnectionEventSupport.addConnectionEventListener(ConnectionEventSupport.java:39)
        com.mchange.v2.c3p0.impl.NewPooledConnection.addConnectionEventListener(NewPooledConnection.java:194)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishResourceOnCheckin(C3P0PooledConnectionPool.java:290)
        com.mchange.v2.resourcepool.BasicResourcePool.attemptRefurbishResourceOnCheckin(BasicResourcePool.java:1615)
        com.mchange.v2.resourcepool.BasicResourcePool.access$200(BasicResourcePool.java:32)
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask.run(BasicResourcePool.java:1228)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:557)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#4,5,main]
        org.netbeans.lib.profiler.server.ProfilerRuntimeMemory.getAndSendCurrentStackTrace(ProfilerRuntimeMemory.java:221)
        org.netbeans.lib.profiler.server.ProfilerRuntimeObjLiveness.traceObjAlloc(ProfilerRuntimeObjLiveness.java:281)
        java.util.HashMap.clone(Unknown Source)
        java.util.HashSet.clone(Unknown Source)
        com.mchange.v2.c3p0.util.ConnectionEventSupport.fireConnectionClosed(ConnectionEventSupport.java:49)
        com.mchange.v2.c3p0.impl.NewPooledConnection.fireConnectionClosed(NewPooledConnection.java:439)
        com.mchange.v2.c3p0.impl.NewPooledConnection.markClosedProxyConnection(NewPooledConnection.java:318)
        com.mchange.v2.c3p0.impl.NewProxyConnection.close(NewProxyConnection.java:1246)
        com.mchange.v1.db.sql.ConnectionUtils.attemptClose(ConnectionUtils.java:41)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishResourceOnCheckin(C3P0PooledConnectionPool.java:292)
        com.mchange.v2.resourcepool.BasicResourcePool.attemptRefurbishResourceOnCheckin(BasicResourcePool.java:1615)
        com.mchange.v2.resourcepool.BasicResourcePool.access$200(BasicResourcePool.java:32)
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask.run(BasicResourcePool.java:1228)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:557)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#6,5,main]
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask.run(BasicResourcePool.java:1229)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:557)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#5,5,main]
        com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask.run(BasicResourcePool.java:1229)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:557)

Похоже, что имеется много RefurbishCheckinResourceTask, и, согласно трассировке, c3p0 не выполняет getConnection. Я использую Spring, hibernate, c3p0.

Моя конфигурация c3p0:

<bean id="pmiDataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource"
    destroy-method="close">
    <property name="driverClass" value="oracle.jdbc.driver.OracleDriver" />
    <property name="jdbcUrl" value="jdbc:oracle:thin:********" />
    <property name="user" value="****" />
    <property name="password" value="****" />
    <property name="acquireIncrement" value="5" />
    <property name="numHelperThreads" value="6" />
    <property name="minPoolSize" value="10" />
    <property name="initialPoolSize" value="15" />
    <property name="maxPoolSize" value="25" />
</bean>

Мои свойства гибернации:

<property name="hibernateProperties">
    <value>
        hibernate.dialect=org.hibernate.dialect.Oracle10gDialect
        hibernate.auto_close_session=true
        hibernate.validator.autoregister_listeners=false
        hibernate.validator.apply_to_ddl=false
        hibernate.cache.use_second_level_cache=true
        hibernate.cache.region.factory_class=net.sf.ehcache.hibernate.EhCacheRegionFactory
        hibernate.cache.use_query_cache=false
        javax.persistence.validation.mode=none
        hibernate.generate_statistics=true
    </value>
</property>

Я проверил много форумов по этому поводу, один из самых похожих случаев - это. очевидный тупик c3p0 / создание аварийных потоков. ВНЕШНЯЯ БЛОКИРОВКА 0.9.1.2

Не могли бы вы помочь?


spring hibernate deadlock c3p0
person ROROROOROROR    schedule 08.08.2014    source источник
comment
numHelperThreads=6 вам действительно достаточно?   -  person Amogh    schedule 08.08.2014
comment
@Amogh Пробовал 8,10, поэтому проблема возникает чаще. Может стоит увеличить threadPool размер?   -  person ROROROOROROR    schedule 08.08.2014
comment
Да, вам нужно протестировать свое приложение в другом сценарии с разным объемом данных и соответственно изменениями в конфигах. Я думаю, на это нет однозначного ответа   -  person Amogh    schedule 08.08.2014

Ответы (1)


arrow_upward
0
arrow_downward

Это не проблема c3p0, на самом деле это моя вина.

Я использовал TaskExecutor для выполнения многопоточной внутренней задачи runnalbe. Поскольку эта процедура вызывает из внутреннего экземпляра метод внешнего класса, ей приходится иметь дело с проблемой синхронизации, которую я проигнорировал в первую очередь.

Когда внутренние экземпляры сталкиваются с какой-либо ошибкой из-за синхронизации, это блокирует возвращение некоторого соединения c3p0 к connection-pool.

Так что это оказалось проблемой outer-inner-class instance method вызова. Просто нужно быть осторожнее с проблемами синхронизации.

Иногда ошибка не отображается на машине разработки, но как только она публикуется на более мощной производственной машине с большим количеством ядер, ошибка может легко возникнуть.

person ROROROOROROR    schedule 14.08.2014