http://www.quickfixj.org/jira This file is an XML representation of an issue en-uk 4.3.1 615 25-03-2011 enterprise http://www.quickfixj.org/jira/browse/QFJ-645 QuickFIX/J Deadlock arises between the following threads: <br/> &nbsp; <br/> Thread1: <br/> &quot;GatewayStateTimer&quot; daemon prio=3 tid=0x04d65388 nid=0x2bef waiting for monitor entry [0x790ff000..0x790ff9f0] <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at quickfix.SessionState.isLogonSent(SessionState.java:162) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- waiting to lock &lt;0x9d12c020&gt; (a quickfix.Session) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at quickfix.Session.sentLogon(Session.java:690) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at quickfix.Session.isLoggedOn(Session.java:731) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at quickfix.Session.sendRaw(Session.java:2151) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at quickfix.Session.send(Session.java:2213) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at quickfix.Session.sendToTarget(Session.java:590) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at com.greatBank.application.fix.Engine.sendMessage(Engine.java:54) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at com.greatBank.application.fix.Engine.pingFIX(Engine.java:191) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at com.greatBank.application.fix.Engine.access$0(Engine.java:186) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at com.greatBank.application.fix.Engine$PingTimerTask.run(Engine.java:176) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at java.util.TimerThread.mainLoop(Timer.java:512) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at java.util.TimerThread.run(Timer.java:462) <br/> &nbsp; <br/> -&gt; Thread1 is sending a message: it first acquires the lock senderMsgSeqNumLock and then tries to acquire a lock on the session instance <br/> &nbsp; <br/> &nbsp; <br/> Thread2: <br/> &quot;QF/J Session dispatcher: FIX.4.2:SENDER1-&gt;TARGET1&quot; daemon prio=3 tid=0x014e5ac8 nid=0x2bfc waiting on condition [0x794ff000..0x794ff970] <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at sun.misc.Unsafe.park(Native Method) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:184) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:256) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at quickfix.SessionState.lockSenderMsgSeqNum(SessionState.java:322) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at quickfix.Session.sendRaw(Session.java:2117) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at quickfix.Session.generateLogout(Session.java:1293) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at quickfix.Session.generateLogout(Session.java:1284) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at quickfix.Session.reset(Session.java:756) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- locked &lt;0x9d12c020&gt; (a quickfix.Session) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at quickfix.Session.next(Session.java:836) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at quickfix.mina.ThreadPerSessionEventHandlingStrategy$MessageDispatchingThread.run(ThreadPerSessionEventHandlingStrategy.java:119) <br/> &nbsp; <br/> -&gt; Thread2: while processing an incoming message, the engine notices that it is in a new session time interval (StartTime/EndTime) and therefore a MsqSeqNum reset is started. <br/> The reset method acquires a lock on the session and, when generating a Logout, tries to acquire the lock senderMsgSeqNumLock, and then deadlocks with Thread1 QFJ-645

Deadlock between message sending and reset due to message reception on Session and senderMsgSeqNumLock

Bug Critical Resolved Fixed Christoph John marc sperisen deadlock Thu, 27 Oct 2011 11:42:47 +0200 Wed, 10 Oct 2012 18:46:30 +0200 Tue, 31 Jul 2012 09:36:24 +0200 1.5.0 1.5.1 1.5.3 Engine 2 <p>Hi <br/> I've also seen this issue with our application. </p> <p>My solution to this was to remove the synchronized keyword from the Session.reset function.</p> <p>I've also attached a unit test we have to check for the issue.</p> <p>Pete</p> <p>had this one too, but I haven't changed the library so I synchronized on the "Session" (=&gt; so locks are taken in the same order).</p> <p>Erratum on the preceding comment.<br/> Synchronizing "send" on the session created a huge performance hit when having heavy load.</p> <p>Peter's was solution was the best one from the beggining and my workarround is not that good ;o)</p> <p>I just put the code I used for the "reset" method :</p> <div class="code panel" style="border-style: solid;border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;border-bottom-style: solid;"><b>quickfix.Session</b></div><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">public</span> void reset() <span class="code-keyword">throws</span> IOException { <span class="code-keyword">if</span> (!resetting.compareAndSet(<span class="code-keyword">false</span>, <span class="code-keyword">true</span>)) { <span class="code-keyword">return</span>; } <span class="code-keyword">try</span> { <span class="code-keyword">if</span> (hasResponder()) { <span class="code-keyword">if</span> (application != <span class="code-keyword">null</span> &amp;&amp; application <span class="code-keyword">instanceof</span> ApplicationExtended) { ((ApplicationExtended) application).onBeforeSessionReset(sessionID); } generateLogout(); disconnect(<span class="code-quote">"Session reset"</span>, <span class="code-keyword">false</span>); } resetState(); } <span class="code-keyword">finally</span> { resetting.set(<span class="code-keyword">false</span>); } }</pre> </div></div> <p>Peter and John,<br/> thanks for your contributions. I have changed the reset() method accordingly and have added the UnitTest (I changed it slightly to work with a pausable Executor and made it detect the deadlock programmatically).</p> Duplicate QFJ-700