I am not aware of any special handling which could stop reconnecting from working only for http proxy... Could it be that the http connection is still considered "active" on the proxy? Could you please check that with "netstat" or some other tool which lists open sockets.
Could you please also create a stack dump when QFJ is in the state of "Pending connection not established..."

Unlike http proxy, i am observing socks proxy reconnecting, probably due to this:

if (proxyIoSession.getRequest() instanceof SocksProxyRequest || proxyIoSession.isReconnectionNeeded())

Unless i need to configure http proxy with something else to get it reconnected.
we are not seeing any outbound connection in this scenario, although the same tcp port remains open.
In the state of "Pending connection...", there are two QFJ threads:

"QFJ Message Processor" #35 daemon prio=5 os_prio=0 tid=0x000000002b144800 nid=0xb090 waiting on condition [0x000000002fb8e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000005d05d60d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  at quickfix.mina.QueueTrackers$1.poll(QueueTrackers.java:45)
  at quickfix.mina.SingleThreadedEventHandlingStrategy.getMessage(SingleThreadedEventHandlingStrategy.java:122)

"QFJ Timer" #34 daemon prio=5 os_prio=0 tid=0x000000002b144000 nid=0xa3a8 in Object.wait() [0x000000002f77f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.mina.core.future.DefaultIoFuture.await0(DefaultIoFuture.java:218)

• locked <0x00000005d0640f10> (a org.apache.mina.core.future.DefaultConnectFuture)
  at org.apache.mina.core.future.DefaultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:174)
  at quickfix.mina.initiator.IoSessionInitiator$ConnectTask.pollConnectFuture(IoSessionInitiator.java:232)
  at quickfix.mina.initiator.IoSessionInitiator$ConnectTask.run(IoSessionInitiator.java:207)
• locked <0x00000005d05fdd80> (a quickfix.mina.initiator.IoSessionInitiator$ConnectTask)

QFJ Timer doesnt stop on breakpoints in IoSessionInitiator.ConnectTask#connect(), SocketChannelImpl#connect() or Net#connect() anymore.

Let me know if anything else would be useful to check.

Unlike http proxy, i am observing socks proxy reconnecting, probably due to this:

if (proxyIoSession.getRequest() instanceof SocksProxyRequest || proxyIoSession.isReconnectionNeeded())
{ return conFuture; } 

Where is that code from? If that is from MINA then it could also be a MINA bug.

That is right. it comes from MINA core: org.apache.mina.proxy.ProxyConnector#connect0:181

So if I understood you right you could trigger the reconnection by setting isReconnectionNeeded to true in the debugger? If yes, then it should be a MINA bug, shouldn't it?

Correct, when i set the re-connection needed with a debugger it does the job. In this case, we close this JIRA, right?

It would be good if you could open a MINA issue and link to this issue. Let me know if I should open the issue against MINA. Thanks so far

I would appreciate if you would open the issue with them.

https://issues.apache.org/jira/browse/DIRMINA-1159

Thanks for the report. Under which cicrumstances does this cause an issue?

The maven enforcer plugin is being used to report on inconsistent component versions. By default the plugin treats all such inconsistencies as errors, so by default such inconsistencies would make the build fail. This behaviour is configurable so it is possible just to get a report of the issues without making the build fail. But I want such inconsistencies to be flagged as serious issues because some of these inconsistencies involve components that show up on Black Duck security scans. The software I am working on is subject to a corporate decree that the software gets through Black Duck without raising any flags. This job is made harder when the number of flags raised is larger due to the same component appearing multiple times at different versions due to these dependency convergence errors.

My aim is to report any such dependency convergence errors that I find via enforcer. If they are then all fixed (there are only a few) then we could set the flag such that any such errors are fatal, so as to detect any regressions as soon as they occur. Without such fixes I will have to write a script that processes the enforcer output and checks any issues against a list of exceptions. This quickfixj issue would have to be such as exception. It just makes ultimate the job of dealing with Black Duck scans harder.

Understood. If you would like you could also open a pull request on the quickfixj project on github. https://github.com/quickfix-j/quickfixj
I am also in favour of making the build fail on enforcer errors.

Good idea, maybe you could open a PR if you have the time.

Thanks

Do you only see this on FIXT.1.1 sessions or also on others?
And you are absolutely sure you are using version 1.6.4?

Hi Christoph, yes we are using version 1.6.4 from maven repository. Also, we are seeing this issue in Fix version 4.4 as well.

I am afraid I cannot help you unless you are able to reproduce this with a current QFJ version, i.e. 2.1 or 2.1.1.

Could it be that your system is either heavily loaded or your QFJ process is constantly garbage collecting or something like that? Between the sending of the Logon reply and the "resetting session" message are four seconds delay?!

What is your StartTime and EndTime set to? Can you post the entry from your log where it says: "Session SENDER->RECEIVER schedule is "...

I analysed a little bit the situation and I see the following potential solutions:

1. Generate a quickfixj-fields jar to be used as dependency by the other messages jars. This way the jar can be imported only once and split package issue is avoided.
   Advantage: avoid code duplication, avoid split package issue
2. Generate  quickfix.<version>.fields packages instead of quickfix.fields. This means that each quickfixj-messages-<version> jar will have its own fields package and the component should make sure to consume the fields from its corresponding package.
   Disadvantage: the common fields between FIX versions will be generated once for each messages version jar, so there is some code duplication. However, this means that the messages between versions are completely decoupled between FIX versions.
   I don't know if this solution is feasible, since the impact is quite big (this won't be backward compatible for client applications already using the quickfix.fields package). As a workaround for this a quickfix.fields package can be generated in the messages-all jar.

I would go for first solution to also keep the impact as low as possible and avoid code duplication. What do you think?
Please let me know if you have other preferences. I'll wait for your feedback before doing any actual changes.

Cheers,
Vlad

Hi Vlad,

I'm just an end-user. You discuss possible solutions, but can you explain outline the existing situation? I understand that modules cannot import the same package from multiple sources but it's not clear to me why the existing design exports the same packages/classes from multiple JARs. Can you please break that down? It would help us understand the proposed solutions and maybe even come up with others. Thank you.

Hi Gili,

I have to admit that I don't recall exactly why the module structure was done this way. Maybe it was the easiest way at that time when we migrated the modules from Ant to Maven or to OSGi.

Cheers,
Chris.

This is a tough one.

1) creating a separate module for fields

Seems like relatively easy solution, but I don't think there is much benefit of doing this one as the core structure problem remains. This might be only feasible to get JPMS working. BTW. Do you have a branch version (or module descriptors) of what you were trying to do?

2) each message module should has it's own version of fields

I agree with you. Even though this might be a duplication of fields I think this is the right way as this decouples fields for each version of FIX dictionaries.

• fields are different (some are missing, some are added)
• enum values for certain fields are different (mostly added as FIX version increases, but possibly they can be removed)

PROBLEM

quickfix-core code already depends on classes from quickfix.field package. Moreover, the code base is dependent (need to be available at run-time) on fields from different FIX versions all together e.g. SessionRejectReason (added in 4.2). They are somewhat "protected" by the code to use them only when specific FIX version is used - still they all need to be in class path at run-time.

This is the reason why code generator plugin generates fields in a very specific order as the code generator plugin does not override field classes that have been created. Currently this super set is assembled into each Maven module quickfixj-messages-fix**, so even if an application is dependent on quickfixj-messages-fix40 it will contain fields from different FIX versions such as SessionRejectReason, otherwise things will fail at run-time e.g.

quickfix.Message#parseBody
throw new FieldException(SessionRejectReason.TAG_SPECIFIED_OUT_OF_REQUIRED_ORDER, field.getTag());

I don't see an easy way of splitting this, but probably a way to go would be to abandon anaemic domain model and message modules would need to contain logic, so quickfix-core would delegate the logic depending on which version of FIX is being used. Also unit tests in quickfix-core are heavy dependent on fields and messages from different FIX versions.

I ran into same issue while trying to launch app from IDE. I was able to workaround it using the following maven configuration:

<dependency>
<groupId>org.quickfixj</groupId>
<artifactId>quickfixj-all</artifactId>
<version>2.1.1</version>
<exclusions>
<exclusion>
<groupId>org.quickfixj</groupId>
<artifactId>quickfixj-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.quickfixj</groupId>
<artifactId>quickfixj-messages-all</artifactId>
</exclusion>
<exclusion>
<groupId>org.quickfixj</groupId>
<artifactId>quickfixj-codegenerator</artifactId>
</exclusion>
<exclusion>
<groupId>org.quickfixj</groupId>
<artifactId>quickfixj-dictgenerator</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>org.apache.mina</groupId>
<artifactId>mina-core</artifactId>
<version>2.1.3</version>
</dependency>

Note that quickfixj-all/pom.xml uses maven-shade-plugin to copy everything from quickfixj-core, quickfixj-messages-all, quickfixj-codegenerator and quickfixj-dictgenerator to single fat jar. Thus, dependencies to these jars are not need. Maybe quickfixj-all/pom.xml should be updated to use optional dependencies instead? I.e.

<dependency>
<groupId>org.quickfixj</groupId>
<artifactId>quickfixj-core</artifactId>
<version>$

</version>
<optional>true</optional>
</dependency>

But in this case transitive dependencies should be explicitly specified in quickfixj-all/pom.xml. I.e.

<dependency>
<groupId>org.apache.mina</groupId>
<artifactId>mina-core</artifactId>
<version>2.1.3</version>
</dependency>

If quickfixj-all/pom.xml is updated in this way, the aforementioned workaround won't be needed.

Hi Philip Whitehouse,
funny, I also stumbled over this a few days ago while investigating https://github.com/quickfix-j/quickfixj/pull/224 / QFJ-981. Didn't get to investigate very far but IIRC the only settings that might differ are validation options

    checkFieldsOutOfOrder
    checkFieldsHaveValues
    checkUserDefinedFields
    checkUnorderedGroupFields
    allowUnknownMessageFields

so I thought I would just use the path plus the validation options as key for the cache in DefaultSessionFactory. Of course that means that there might be several DDs with different validation options. But on the other hand, I do not know if this really is a big downside. Of course depends how much sessions with the same dictionary and different validation options one might have in one VM.
Your proposed solution is of course a little more sophisticated since we would only have one instance of a DD, however as you said there might be API changes.

What do you think?
Thanks,
Chris.

The problem with multiple copies of the same DataDictionary is that they are a massive memory hog given the size of them (especially once you add all the service packs).

I agree it's just the validation options however. My VMs generally run lots of sessions with the same dictionary so I'm keen to avoid an unexpected RAM usage increase from fixing the bug.

The other side to it is a proxy layer we have allowing you to live-reload the dictionary file to correct issues. I might try to upstream this at the same time.

This is likely to be something I have time to proof-of-concept relatively soon as currently the bug provides run-time unreliability on what will actually be applied to validate a given message.
Which is fairly ugly in terms of potential impact.

OK, so did I understand you correctly that you are going to take a stab on this?
Or do you want me to do it?

Sorry, non-native english speaker here, so it's sometimes hard for me to get the nuances.

Thanks,
Chris.

13:35:17.885 QFJ Timer ERROR quickfix.SocketInitiator.run:356 - Error during timer processing
quickfix.RuntimeError: java.io.IOException: Stream Closed
at quickfix.SessionState.reset(SessionState.java:384)
at quickfix.Session.resetState(Session.java:2624)
at quickfix.Session.generateLogon(Session.java:2003)
at quickfix.Session.next(Session.java:1918)
at quickfix.mina.SessionConnector$SessionTimerTask.run(SessionConnector.java:350)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at quickfix.FileStore.close(FileStore.java:218)
at quickfix.FileStore.close(FileStore.java:209)
at quickfix.FileStore.closeAndDeleteFiles(FileStore.java:223)
at quickfix.FileStore.initialize(FileStore.java:101)
at quickfix.FileStore.reset(FileStore.java:405)
at quickfix.SessionState.reset(SessionState.java:382)
... 11 common frames omitted
Suppressed: java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
... 17 common frames omitted

Same issue

What file might be the problem? - java.io.FileOutputStream.writeBytes

Java 8

I am not sure I understand fully: when you want to limit the queue size and discard messages if the queue is full, then why do you do a resend request for that many messages anyway? You could then just not do any resends (e.g. reset to seqnum 1)?

Apart from that: if you are transmitting millions of messages in a few hours then I am not sure if FIX is the right choice of protocol. I assume that some kind of volatile data is transmitted (e.g. market data) which in turn leads to the question why you need resends at all if the data is outdated in the meantime anyway.

Or did I misunderstand something?

Thanks,
Chris.

Hi Chris - Thank you for the reply! Let me try to give more details.

1. The reason we want to persist these messages is for future analysis. Normally the system works perfectly fine, but we want to handle the rarely happened situation where there is an issue and the system is down in the middle of the day. When that happens, we do need to send resend requests to catch up the outdated data.

2. My understanding is that there are two queues involved:

• eventQueue in Session which is a bounded blocking queue
• messageQueue in SessionState which is a map and not bounded

Every message has to go through the eventQueue, but only messages with higher seqNum will be put into messageQueue. Let's assume the batch size is 20 and consider the following event sequence:

• initiator -> acceptor: expect 501, but seeing 10000, request resending of 501-520
• acceptor -> initiator: resend 501-520, done
• acceptor -> initiator: new messages 10001, 10002, 10003
• initiator -> acceptor: request resending of 521-540
• acceptor -> initiator: resend 521-540, done
• acceptor -> initiator: new messages 10004, 10005
  ...

In the given sequence, messages 501-540 will be handled correctly, but 10001-10005 will be added to messageQueue and will not be touched until all messages 501-10000 have been requested and processed.

My proposal is to limit the size of the messageQueue (the map) to prevent the unbounded growing. For example if we set the limit to 4 messages, then the message 10005 and later messages will not be added to the messageQueue. When the initiator finishes handling messages 501-10004, it will expect 10005, but since it's discarded, initiator will try to request messages from 10005 to 10024 and move on.

This scenario will happen only when both of the following conditions are satisfied:

• resend batch size is set.
• new messages are coming in when the acceptor resends the messages.

Hope this better explained the scenario.

Thanks,
Ryan

BTW this is not a blocking issue for us, just want to report what we've observed during the development.

Hi Ryan,

after thinking a little more about it I think it should work the way you said, i.e. if we limited the size of the queued messages QFJ will request them anyway as soon as it has dequeued all messages.

I have the following questions:
1. Why should this only be active when resend batch size is set? IMHO it could also happen that new messages are interleaved with resent messages. But of course that depends on how the counterparty has implemented their resend processing.
2. Why do you want to change the LinkedHashMap to TreeMap? Just because of the size limitation? Wouldn't a simple check if the allowed size has been reached (and then not queueing the message) be sufficient?
3. In the description you said "If the queue is full, then add the new message and remove the entry with highest sequence number." Why? I thought the idea was to not add the message?

Thanks,
Chris.

Of course it will lead to massive number of messages since messages will be re-requested multiple times if the counterparty already sent them and we re-request them because we did not put them to the queue.

Hi Chris,

Let me pull the example sequence from above and try to explain my understanding:

• initiator -> acceptor: expect 501, but seeing 10000, request resending of 501-520
• acceptor -> initiator: resend 501-520, done
• acceptor -> initiator: new messages 10001, 10002, 10003
• initiator -> acceptor: request resending of 521-540
• acceptor -> initiator: resend 521-540, done
• acceptor -> initiator: new messages 10004, 10005
• ...
  1. Totally agree that the limit can and should be applied for both batch and non-batch. Normally non-batch request won't cause this problem because of the acceptor logic. Let's take the same example, instead of using a batch of 20, if we send the resend request as 501-infinity, then messages 501-10000 will be added to the acceptor's send buffer. Now if new messages 10001-10005 appears on the acceptor side, they will only be appended to the end of the buffer, which means initiator will not see them until 501-10000 are received and processed. But if the acceptor takes another strategy of handling resend request, then non-batch may hit the same memory expanding problem. So I think you are right that we should limit the size of messageQueue in all cases.
  2 & 3. Generally speaking, we care more about the message sequence (MsgSeqNum) but not the insertion sequence. More specifically, let's assume the messageQueue is full with messages 10001-10004, and there is a temporary out of order message 542 coming in. Since we care more about 542 but not 10004 at this time, evicting 10004 would be the only option if we want to keep 542 and not expanding the queue size. By this way the TreeMap messageQueue will still work even after it's full.
  4. If the size limit is to be added, it should definitely be configurable so that user can choose between memory expanding and massive resend requests.

Thanks,
Ryan

Hi amichair,

maybe we should add the javadoc comment as you suggested and provide an additional method, say setFieldConverted(int key, Field<?> field) , so users can choose which method suits them best.

Cheers,
Chris.

Briefly had a look at this. What about making Field abstract (According to a comment in the class it is only non-abstract to have JNI compatibility. But that was like 10 years ago.) and adding an abstract convertToString() method that does the necessary conversion in the subclasses? Example for BooleanField:

    @Override
    public String convertToString() {
        return BooleanConverter.convert(getValue());
    }

Then we can call in FieldMap:

    public void setField(int key, Field<?> field) {
        setString(key, field.convertToString());
    }

That way we can ensure that we always have a conversion method for a field and don't need a lengthy if-else statement.

What do you think?

Encapsulating the converters inside the Field classes sounds like a good idea regardless of this issue or of Field being abstract

I have't delved into this one yet, but perhaps instead of convertToString, maybe toString itself should just do this properly?

Pull request created with implementation of the idea: https://github.com/quickfix-j/quickfixj/pull/226

Hi Philip Whitehouse,

I am not entirely sure but I might be that the return code is a relict from the C++ implementation.
... digging through the code ...

Here is the comment of the Session.send() method:

   /**
     * Send a message to a counterparty. Sequence numbers and information about the sender
     * and target identification will be added automatically (or overwritten if that
     * information already is present).
     *
     * The returned status flag is included for
     * compatibility with the JNI API but it's usefulness is questionable.
     * In QuickFIX/J, the message is transmitted using asynchronous network I/O so the boolean
     * only indicates the message was successfully queued for transmission. An error could still
     * occur before the message data is actually sent.
     *
     * @param message the message to send
     * @return a status flag indicating whether the write to the network layer was successful.
     */

I think I already analyzed this some time ago. What I found more special was that the message is actually sent first and then persisted which might leave a little gap when there is a crash. The .NET and C++ implementations do persist first and send afterwards.

Chris.

The comment about the return indicating a successful network write however is also broken however.

In the case of set throwing an IOException instead of returning false it will return false despite having already written to the network.

So it's definitely 'special'.

You could turn on synchronous writes if you want to make sure that the message was really sent. But as you said, the usage of the return code is somewhat inconsistent.

Linux

Of course, it would be great to see this Logout messages not only in the logs, but and in the quickfix.Application#fromAdmin as it usually happens - it allows us to handle this Logout message and set the expected sequence number (I already did this with some other counterparties).

I think the problem is that the session is disconnected at 2018-03-26 10:16:20.885 (see event log) but the Logout message is coming in at 20180326-14:16:20.886 (SendingTime on message in your network log). This means the session is disconnected 1ms before the counterparty sends the Logout message (given that both your and the counterparty's clock is in sync).

This can't be a general problem because this kind of disconnection (message seqnum too low) happens quite often with some counterparties and the messages show up in the log files. I don't know of any occurrences with current QuickFIX/J versions.

So unless we can reproduce this in a unit test we cannot do anything about it.

Sorry, but I can only repeat my statement from the other issues: please please please write to the mailing list before opening possible "bug" tickets. That way we can clarify if there really is a bug before issues are opened which in the end is saving your and our time.

Thanks,
Chris.

Please, take a look to this network dump of the incoming messages (on the our side):

10:16:19.846538 IP some_ip.port > our_ip.port: Flags [S.], seq 28365822, ack 2573233766, win 32000, options [mss 1380,nop,wscale 8], length 0
        0x0000:  000c 2909 731b 001f 9edf 3ebf 0800 4500  ..).s.....>...E.
        0x0010:  0030 0000 4000 3406 90c8 cfef 1bda ac14  [email protected].........
        0x0020:  1e22 6204 8259 01b0 d3fe 9960 6e66 7012  ."b..Y.....`nfp.
        0x0030:  7d00 8f83 0000 0204 0564 0103 0308       }........d....
10:16:20.885507 IP some_ip.port > our_ip.port: Flags [P.], seq 1:134, ack 103, win 4096, length 133
        0x0000:  000c 2909 731b 001f 9edf 3ebf 0800 4500  ..).s.....>...E.
        0x0010:  00ad 0000 4000 3406 904b cfef 1bda ac14  [email protected]......
        0x0020:  1e22 6204 8259 01b0 d3ff 9960 6ecc 5018  ."b..Y.....`n.P.
        0x0030:  1000 216b 0000 383d 4649 582e 342e 3201  ..!k..8=FIX.4.2.
        0x0040:  393d 3031 3039 0133 353d 3501 3334 3d38  9=0109.35=5.34=8
        0x0050:  3801 3439 3d4a 4c51 4401 3532 3d32 3031  8.49=JLQD.52=201
        0x0060:  3830 3332 362d 3134 3a31 363a 3230 2e38  80326-14:16:20.8
        0x0070:  3836 0135 363d 6678 7732 0137 3839 3d35  86.56=fxw2.789=5
        0x0080:  3101 3538 3d4d 7367 5365 714e 756d 2074  1.58=MsgSeqNum.t
        0x0090:  6f6f 206c 6f77 2c20 6578 7065 6374 696e  oo.low,.expectin
        0x00a0:  6720 3531 2062 7574 2072 6563 6569 7665  g.51.but.receive
        0x00b0:  6420 3101 3130 3d30 3735 01              d.1.10=075.
10:16:20.885543 IP some_ip.port > our_ip.port: Flags [R.], seq 134, ack 103, win 0, length 0
        0x0000:  000c 2909 731b 001f 9edf 3ebf 0800 4500  ..).s.....>...E.
        0x0010:  0028 0000 4000 3406 90d0 cfef 1bda ac14  .([email protected].........
        0x0020:  1e22 6204 8259 01b0 d484 9960 6ecc 5014  ."b..Y.....`n.P.
        0x0030:  0000 3711 0000 0000 0000 0000            ..7.........
10:16:22.887575 IP some_ip.port > our_ip.33372: Flags [S.], seq 4175399090, ack 3053759955, win 32000, options [mss 1380,nop,wscale 8], length 0
        0x0000:  000c 2909 731b 001f 9edf 3ebf 0800 4500  ..).s.....>...E.
        0x0010:  0030 0000 4000 3406 90c8 cfef 1bda ac14  [email protected].........
        0x0020:  1e22 6204 825c f8df 88b2 b604 add3 7012  ."b..\........p.
        0x0030:  7d00 878b 0000 0204 0564 0103 0308       }........d....
10:16:23.845393 IP some_ip.port > our_ip.33372: Flags [P.], seq 1:134, ack 103, win 4096, length 133
        0x0000:  000c 2909 731b 001f 9edf 3ebf 0800 4500  ..).s.....>...E.
        0x0010:  00ad 0000 4000 3406 904b cfef 1bda ac14  [email protected]......
        0x0020:  1e22 6204 825c f8df 88b3 b604 ae39 5018  ."b..\.......9P.
        0x0030:  1000 1b6f 0000 383d 4649 582e 342e 3201  ...o..8=FIX.4.2.
        0x0040:  393d 3031 3039 0133 353d 3501 3334 3d38  9=0109.35=5.34=8
        0x0050:  3901 3439 3d4a 4c51 4401 3532 3d32 3031  9.49=JLQD.52=201
        0x0060:  3830 3332 362d 3134 3a31 363a 3233 2e38  80326-14:16:23.8
        0x0070:  3436 0135 363d 6678 7732 0137 3839 3d35  46.56=fxw2.789=5
        0x0080:  3101 3538 3d4d 7367 5365 714e 756d 2074  1.58=MsgSeqNum.t
        0x0090:  6f6f 206c 6f77 2c20 6578 7065 6374 696e  oo.low,.expectin
        0x00a0:  6720 3531 2062 7574 2072 6563 6569 7665  g.51.but.receive
        0x00b0:  6420 3201 3130 3d30 3736 01              d.2.10=076.
10:16:23.845427 IP some_ip.port > our_ip.33372: Flags [R.], seq 134, ack 103, win 0, length 0
        0x0000:  000c 2909 731b 001f 9edf 3ebf 0800 4500  ..).s.....>...E.
        0x0010:  0028 0000 4000 3406 90d0 cfef 1bda ac14  .([email protected].........
        0x0020:  1e22 6204 825c f8df 8938 b604 ae39 5014  ."b..\...8...9P.
        0x0030:  0000 2f19 0000 0000 0000 0000            ../.........
...

As I see, there are constantly repeating 3 records:

1. Probably, a connect message.
2. Then, about one second later, the Logout message.
3. Then, immediately, one more message, probably disconnect.

I think that the time between our and the counterparty sides is not so good synchronized to rely on this (in fact, we have some problems with time synchronization on the used machine).
I guess, the problem may be in the too little delay between receiving the Logout and the disconnect messages. Also, it can be in the using of non-standard tag 789. Of course, it can be in something else, but it exists - that is the reason because I wrote this in the issues tracker, not to bother you, of course.

Thanks and my apologize if I'm using not the best procedure of issues notifying,
Nikolai.

Hi Nikolai,
there is no need to apologize. I just wanted to stress the fact that it is hard to fix issues that cannot be reproduced reliably. We would need some kind of reproducable unit test to look deeper into this issue. I must admit that I never heard of this problem before but of course this does not mean that it is not possible to happen. May be related to a couple of things.

Thanks,
Chris.

Linux

After further investigation of the problem, I guess that the problem is not in the speed of the connection, but in the fact that we set ResetSeqNumFlag=Y in the our Logon message but in the answering Logon message it is not presented (so, the priority of the problem can be lowered).

The code that is generating this error is in the method quickfix.Session#nextLogon:

        // Check for proper sequence reset response
        if (state.isResetSent() && !state.isResetReceived()) {
            disconnect("Received logon response before sending request", true);
        }

It can be a problem indeed, but the disconnect message seems totally misleading for me.
Also, I do not see any benefits to break down the connection in this case (we sent ResetSeqNumFlag but did not received it in the answer).

I don't know but what sense does it make to reset the sequence numbers only on one side? So I guess the check might have a point. Unless I'm missing something.

Just a real situation: our partner is ready to reset the sequence number on the one side only. It seems that using ResetSeqNumFlag for this purpose it is not nice by FIX standards, but also it is a fact that this situation is not related with the "Received logon response before sending request".
Also, it is not clear for me why dropping down the connection is better than working.

I think it does not make sense to open a bug ticket for every counterparty that does not conform to normal FIX standard.
If the counterparty wanted that you reset your sequence numbers I guess they should be sending you tag 141 in their Logon. At least IMHO.
Why are you sending 141? Does the counterparty require you to do so?

IMHO the best would be to direct such questions to the mailing list. Please post follow up questions and your answers there.
https://sourceforge.net/projects/quickfixj/lists/quickfixj-users

Thanks

John, I agree that it is no sence to open a bug ticket for a counetrparty that does not conform regular FIX standard (of course, I'll discuss this situation with them directly). But the ticket is about another thing and I'm really sorry if I'm describing it not clear.
The problem is that the engine is disconnecting with "Received logon response before sending request" when there is no such situation (logon response is coming after sending request).
Thank you for the patience.

I am sorry, but I cannot follow your description. Could you maybe describe the problem with some code examples or a unit test?
Thanks,
Chris.

Thanks Chris for looking into,i now try to be more specific.

Please look into DeafultSessionFactory.processFixtDataDictionaries, we find following piece of code, please note here that toApplVerID is used to create the instance of ApplVerID before the applicationDictionary is added to the datadictionaryProvider.

final DataDictionary dd = createDataDictionary(sessionID, settings, key,
                            beginStringQualifier);
dataDictionaryProvider.addApplicationDictionary(MessageUtils.toApplVerID(beginStringQualifier), dd);

and createDataDictionary method reads the settings

private DataDictionary createDataDictionary(SessionID sessionID, SessionSettings settings,
            String settingsKey, String beginString) throws ConfigError, FieldConvertError {
        final String path = getDictionaryPath(sessionID, settings, settingsKey, beginString);
        final DataDictionary dataDictionary = getDataDictionary(path);

        if (settings.isSetting(sessionID, Session.SETTING_VALIDATE_FIELDS_OUT_OF_ORDER)) {
            dataDictionary.setCheckFieldsOutOfOrder(settings.getBool(sessionID,
                    Session.SETTING_VALIDATE_FIELDS_OUT_OF_ORDER));
        }

        if (settings.isSetting(sessionID, Session.SETTING_VALIDATE_FIELDS_HAVE_VALUES)) {
            dataDictionary.setCheckFieldsHaveValues(settings.getBool(sessionID,
                    Session.SETTING_VALIDATE_FIELDS_HAVE_VALUES));
        }

        if (settings.isSetting(sessionID, Session.SETTING_VALIDATE_UNORDERED_GROUP_FIELDS)) {        	
            boolean chkUnorderedGrpFields = settings.getBool(sessionID,
                    Session.SETTING_VALIDATE_UNORDERED_GROUP_FIELDS);
            settings.getLogger().error("Msg SttString "+Session.SETTING_VALIDATE_UNORDERED_GROUP_FIELDS +" value "+ chkUnorderedGrpFields +" dataDictionary " +dataDictionary.toString() +" beginString "+beginString);
			dataDictionary.setCheckUnorderedGroupFields(chkUnorderedGrpFields);
        }

        if (settings.isSetting(sessionID, Session.SETTING_VALIDATE_USER_DEFINED_FIELDS)) {
            dataDictionary.setCheckUserDefinedFields(settings.getBool(sessionID,
                    Session.SETTING_VALIDATE_USER_DEFINED_FIELDS));
        }

        if (settings.isSetting(sessionID, Session.SETTING_ALLOW_UNKNOWN_MSG_FIELDS)) {
            dataDictionary.setAllowUnknownMessageFields(settings.getBool(sessionID,
                    Session.SETTING_ALLOW_UNKNOWN_MSG_FIELDS));
        }

        return dataDictionary;
    }

Please note that DefaultDataDictionaryProvider uses ApplVerID as the key of the map.

While there is a method called MessageUtils.parse(Session session, String messageString), which uses getApplVerID(session, messageString) to create the ApplVerId and uses the same to fetch the applicationDictionary from dataDictionaryProvider.

MessageUtils.parse(Session session, String messageString)
.....
final String beginString = getStringField(messageString, BeginString.FIELD);
		final String msgType = getMessageType(messageString);

		ApplVerID applVerID;

		if (FixVersions.BEGINSTRING_FIXT11.equals(beginString))
		{
			applVerID = getApplVerID(session, messageString);
		}
		else
		{
			applVerID = toApplVerID(beginString);
		}
		final MessageFactory messageFactory = session.getMessageFactory();

		final DataDictionaryProvider ddProvider = session.getDataDictionaryProvider();
		final DataDictionary sessionDataDictionary = ddProvider == null ? null : ddProvider.getSessionDataDictionary(beginString);
		final DataDictionary applicationDataDictionary = ddProvider == null ? null : ddProvider.getApplicationDataDictionary(applVerID);
....

So when an application message is received a new DataDictionary is created by DefaultDataDictionaryProvider, look at method

 public DataDictionary getApplicationDataDictionary(ApplVerID applVerID) 

so the problem is that this new DataDictionary created by DefaultdataDictionaryProvider does not copy the FIX configurations i.e. settings. Hence even though the application dev has set the settings they are not picked while parsing messages.

Please ask further clarifications.

Thanks,
Sachin

Hi,

I have changed https://github.com/quickfix-j/quickfixj/blob/master/quickfixj-core/src/test/java/quickfix/DefaultSessionFactoryTest.java (method testFixtDataDictionaryConfiguration) slightly to set "AllowUnknownMessageFields" to true.

 @Test
    public void testFixtDataDictionaryConfiguration() throws Exception {
        SessionID sessionID = new SessionID(FixVersions.BEGINSTRING_FIXT11, "SENDER", "TARGET");
        setUpDefaultSettings(sessionID);
        settings.setBool(sessionID, Session.SETTING_USE_DATA_DICTIONARY, true);
        settings.setString(sessionID, Session.SETTING_TRANSPORT_DATA_DICTIONARY, "FIXT11.xml");
        settings.setString(sessionID, Session.SETTING_DEFAULT_APPL_VER_ID, "FIX.4.2");
        settings.setString(sessionID, Session.SETTING_APP_DATA_DICTIONARY, "FIX42.xml");
        settings.setString(sessionID, Session.SETTING_APP_DATA_DICTIONARY + "." + FixVersions.BEGINSTRING_FIX40, "FIX40.xml");
        settings.setString(sessionID, Session.SETTING_ALLOW_UNKNOWN_MSG_FIELDS, "Y");   // added

        Session session = factory.create(sessionID, settings);

        DataDictionaryProvider provider = session.getDataDictionaryProvider();
        assertThat(provider.getSessionDataDictionary(sessionID.getBeginString()),
                is(notNullValue()));
        final DataDictionary applicationDataDictionary = provider.getApplicationDataDictionary(new ApplVerID(ApplVerID.FIX42));

        assertThat(applicationDataDictionary,
                is(notNullValue()));
        assertThat(provider.getApplicationDataDictionary(new ApplVerID(ApplVerID.FIX40)),
                is(notNullValue()));
        
        System.err.println("XXXX " + provider.getSessionDataDictionary(sessionID.getBeginString()).isAllowUnknownMessageFields() );
        System.err.println("XXXX " + applicationDataDictionary.isAllowUnknownMessageFields() );
        
        quickfix.fixt11.Logon logon = new quickfix.fixt11.Logon(new EncryptMethod(EncryptMethod.NONE_OTHER), new HeartBtInt(30),
                new DefaultApplVerID(ApplVerID.FIX42));
        MessageUtils.parse(session, logon.toString());

This gives the following as output:

XXXX true
XXXX true

So the setting is picked up by both dictionaries.

Then I stepped into MessageUtils.parse(session, logon.toString()) with the debugger and could also see that the setting is true on the applicationDataDictionary.

So maybe I am taking a different code path than you? BTW I tested it on master, i.e. with version 1.7.0-SNAPSHOT.

Cheers,
Chris.

Hi Chris,

I understand in your use case a new dictionary is never created, but there are some configurations in which a scenario occurs which creates a new dictionary by DefaultDataDictionaryProvider, then the settings will not be copied.
so to solve the problem we have 2 options

1) I correct the Fix configurations so that a new dictionary is never created.
2) Somehow set the session settings on each datadictionary created by DefaultDataDictionaryProvider

I see you are suggesting option 1.

--Regards,
Sachin

Hi,

when there are such configurations then please give an example configuration. Otherwise I don't know how to reproduce the problem.

Thanks,
Chris.

Attached is the zip src files with change

Add fix for correctly reading the settings in all the cases, i.e. whenever DataDictionary object is created from SessionFactory or MessageUtils.parse
    a) Update interface DataDictionaryProvider.java add DataDictionary getApplicationDataDictionary(SessionID sessionId, ApplVerID applVerID);
    b) Update DefaultDataDictionaryProvider
        1) add field SessionSettings
        2) add another constructor with SessionSettings, update old constructors to initialize field settings
        3) add implementation of new method getApplicationDataDictionary(SessionID sessionId, ApplVerID applVerID); which also readSettings\
        4) add public static method fillSettings
        5) add local method readSettings which calls static method fillsettings which is extracted from DefaultSessionFactory
    c) Update DefaultSessionFactory
        1) Update method create use new Constructor added for DefaultDataDictionaryProvider, pass settings.
        2) Update method createDataDictionary remove filling of settings locally and call DefaultDataDictionaryProvider.fillSettings instead
    d) Update MessageUtils, by updating method parse(Session session, String messageString) throws InvalidMessage so that appDataDictionary is fetched from ddProvider by also passing SessionID
    e) Update Session. next(Message message, boolean isProcessingQueuedMessages) throws FieldNotFound, RejectLogon, IncorrectDataFormat,
            IncorrectTagValue, UnsupportedMessageType, IOException, InvalidMessage, so that getApplicationDataDictionary is called with sessionID param as well.

only improvement the changes i uploaded have is that whereever the DataDictionary object is created, the existing settings of  current session are always applied on DataDictionary created.

Thanks for the file. Are you maybe able to generate a patch or even better open a pull request on github?
Thanks,
Chris.

Linux

You'll need to implement the HA functionality by yourself. You could use a database, synced file system or clustered in-memory cache to store the session information. But the whole failover decision has to be implemented by your application.

Since JIRA is the bug tracker you could ask on the mailing list whether anyone has already implemented something similar: https://lists.sourceforge.net/lists/listinfo/quickfixj-users
If you need production or implementation support please check: http://www.quickfixj.org/support/

Thanks,
Chris.

Hmm, I can only quote the comment on the send method:

The returned status flag is included for compatibility with the JNI API but it's usefulness is questionable.
In QuickFIX/J, the message is transmitted using asynchronous network I/O so the boolean
only indicates the message was successfully queued for transmission. An error could still
occur before the message data is actually sent.

If the connection is broken, then the next incoming message on re-logon will trigger another ResendRequest anyway.
What do you think?

This is currently maintained by marketcetera: http://repo.marketcetera.org/maven/quickfixj/quickfixj-core/

If you have any questions regarding this please write to the mailing list: https://lists.sourceforge.net/lists/listinfo/quickfixj-users

Fedora 23 x86_64, Windows 7 64 bit SP1

Could you please attach some logs? Thanks

FIX4.2

Sorry, this is a little hard to follow. I don't know about the command "fix PNPBBBN". Is this some custom stuff that you are executing? Actually, I would expect QFJ to send a Logout if it received too low a sequence number. The last log line is half an hour later than the log line before. What happened in between?

Actually, there are some unit tests around the chunked resend requests behaviour. But of course, this does not mean that there could be no bugs inside that portion of the code. But I cannot really follow your example. Do you have a unit test or some more concise steps which can be followed?

Sorry the log is a bit hard to read.
1. "fix PNPBBBN" is an admin command I created which calls session.setNextTargetMsgSeqNum() so I can manipulate the seq no to trigger the scenario.
2. The log started from 15:28:37 to 15:29:38. You can ignore the UTC time in the FIX message.
3. A simpler example is
1) both sides seq no started from 1.
2) for some reasons client didn't receive some messages
3) the next message client received was seq no 20
4) chunk size was set to 10. so client sent a resent request being=2 end =11
5) client received seq reset with GapFill=Y
6) client should send the next resend request begin=2 end=20 but it didn't

Hope this helps.

Sorry there was a typo in 6) client should send the next resend request begin=12 end=20 but it didn't

If you look at the incoming sequence reset you see the field 36/NewSeqNo set to 449. So that is why there are no more messages resent. This should actually also be a message in your event log. Something along the lines of: "received sequence reset to 449."

Hmm i am reading the FIX doc.
The message in all situations specifies NewSeqNo <36> to reset as thevalue of the next sequence numberimmediately following the messages and/or sequence numbers being skipped.
It seems <36> just indicates the next seq no I will receive. Does it mean the whole gap has been filled? In order to send the next resend request what <36> should be?

Actually I was connecting to CME drop copy and encountered this issue.

If you send a resend request and the counterparty sends SequenceReset with NewSeqNo = 449 then QFJ will continue with that incoming seqnum. That means that the gap has been filled up to that seqnum, yes. Probably there were only session messages in between, or messages that CME does not want to resend.

NB: I think this belongs more onto the mailing list or with CME support than here. It's no bug after all.

Sorry I still don't quite understand the logic here.
Let's say the seq gap is 1-20 and chunk size is set to 10. Application messages are in 11-20 but NOT in 1-10. After QFJ sends the 1st resend request from 1 to 10 what NewSeqNo should be in SequenceReset?
I don't think it will be 11 because seq no shouldn't decrease.
If it is 21 QFJ will continue with 21 and lose all messages from 11 to 20.

If my understanding is not correct can you please give me an example how chunked resend should work? Appreciate your help.

If you don't think it is a bug I will add code in my Application class to override the logic. I just want to point out that current logic does NOT work for CME iLink and Drop Copy 4.0 which place a 2500 limit on resend request. QFJ would lose all messages after the first batch. It makes more sense to assume the corresponding chunk gap has been filled based on the sequence reset, not the whole gap.

Sorry, I totally forgot about this one. What logic do you want to override? Actually, you do not need to override any of the session logic.

It seems <36> just indicates the next seq no I will receive. Does it mean the whole gap has been filled? In order to send the next resend request what <36> should be?

The NewSeqNo on a sequence reset is the next-to-be sequence number of the other communication side.

Let's say the seq gap is 1-20 and chunk size is set to 10. Application messages are in 11-20 but NOT in 1-10. After QFJ sends the 1st resend request from 1 to 10 what NewSeqNo should be in SequenceReset?

NewSeqNo should be 11.

I don't think it will be 11 because seq no shouldn't decrease.
If it is 21 QFJ will continue with 21 and lose all messages from 11 to 20.

Why does the seq no decrease? It is as follows: you connect to the counterparty. Your expected target seq no is 1. The other side comes in with 20. Now you send a resend request from 1-10.
You receive a SeqReset with NewSeqNo 11. Now your expected target seq no is 11. This is an increment to 1 and not a decrement.
You issue another ResendRequest for 11-20 and receive the app messages, each will increment your expected target seq no by 1.

I know that there are several users which use QFJ to connect to CME. The last issue around chunked resend requests I remember was QFJ-751 which was fixed in QF/J 1.6.0.

Please write a mail to the mailing list if you have further questions. https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Or please attach a failing unit test so that I can reproduce the error.

Thanks.

I can confirm that I am seeing the same behavior when connecting to the CME's Drop Copy 4.0. It appears this issue arises due to the way in which the CME increments tag 36 of SequenceReset messages and the way QuickFIX/J handles the sending of ResendRequest messages.

The following code is an excerpt from the Session class in the quickfix package which I believe is where the logic breaks down. It appears that QuickFIX/J will attempt to send chunks of ResendRequests depending upon when it receives incoming SequenceReset messages.The code below requires the value of the newSequence variable to be less than the EndSeqNo value defined in the range of messages that the client is supposed to fetch. This case fails and some ResendRequest messages will not be sent. Would it be safe to remove this logic from the if statement allowing subsequent RequestsRequests to be sent to the counterparty despite the fact that the newSequence value will be past the range of messages defined in the range variable? If so, the code would also need to be modified to not leverage the value of the newSequence variable when sending the ResendRequest to define the range of sequence numbers to send in the subsequent ResendRequest.

newSequence < range.getEndSeqNo()

private void nextSequenceReset(Message sequenceReset) throws IOException, RejectLogon,
FieldNotFound, IncorrectDataFormat, IncorrectTagValue, UnsupportedMessageType {
boolean isGapFill = false;
if (sequenceReset.isSetField(GapFillFlag.FIELD))

if (!verify(sequenceReset, isGapFill, isGapFill))

if (validateSequenceNumbers && sequenceReset.isSetField(NewSeqNo.FIELD)) {
final int newSequence = sequenceReset.getInt(NewSeqNo.FIELD);

getLog().onEvent(
"Received SequenceReset FROM: " + getExpectedTargetNum() + " TO: "
+ newSequence);
if (newSequence > getExpectedTargetNum()) {
state.setNextTargetMsgSeqNum(newSequence);
final ResendRange range = state.getResendRange();
if (range.isChunkedResendRequest()) {
if (newSequence >= range.getCurrentEndSeqNo()
&& newSequence < range.getEndSeqNo())

}
// QFJ-728: newSequence will be the seqnum of the next message so we
// delete all older messages from the queue since they are effectively skipped.
state.dequeueMessagesUpTo(newSequence);
} else if (newSequence < getExpectedTargetNum()) {

getLog().onErrorEvent(
"Invalid SequenceReset: newSequence=" + newSequence + " < expected="
+ getExpectedTargetNum());
if (resetOrDisconnectIfRequired(sequenceReset))

generateReject(sequenceReset, SessionRejectReason.VALUE_IS_INCORRECT,
NewSeqNo.FIELD);
}
}
}

John, I received your email. Do you have a non-personal email address? My corporate email doesn't allow sending to gmail, hotmail, etc.

I very briefly read through http://www.cmegroup.com/confluence/display/EPICSANDBOX/Drop+Copy+Session+Layer+-+Resend+Request and I do not see anything that is contradictory to how QFJ interprets the NewSeqNo tag. However, it seems that CME is implementing something very custom. A test case would definitely help here.
One thing I noticed though, is that they mention "duplicate resend requests". Maybe you can try if the configuration SendRedundantResendRequests=Y helps in that case?

Chrstoph,

What format would you prefer to see a test case in? I could provide the FIX messaging log so that it might be easier to see how the CME is increasing the sequence numbers in the SequenceReset messages they send in the middle of sending ResendRequest chunks past the end of the original ResendRequest range such that QuickFIX/J defines after logging on which causes some of the ResendRequest chunks to not be sent to the exchange.

As I mentioned in my earlier post, I believe the issue is in the code snippet I posted above where upon receiving SequenceReset messages QuickFIX/J will determine if a new ResendRequest chunk message needs to get sent out based on the following criteria in the nextSequenceReset method of the Session class by checking if newSequence < range.getEndSeqNo(). Since this fails, some ResendRequests fail to ever get sent out.

Say the defined ResendRequest range is 1-10000.
1) I send a ResendRequest for messages 1-2500. I receive these messages and then receive a SequenceReset to 2520 for example.
2) I then send a ResendRequest for messages 2501-5000. I receive these messages and then receive a SequenceReset to 5050.
3) I send a ResendRequest for messages 5001-7500. I receive these messages and then receive a SequenceReset of 11000.
At this point the newSequence < range.getEndSeqNo() test fails and QuickFIX/J never sends the final ResendRequest for messages 7501-10000. The FIX connection continues to receive real-time trades etc as per normal.

I would assume this to be custom logic that the CME is implementing on their end. I think the difference comes down to the fact that QuickFIX/J appears to rely more on the newSequence number that is received on an incoming SequenceReset message to dictate whether further ResendRequest chunks will be sent out and the CME may send out a SequenceReset message taking you past the originally defined EndSeqNo range.

I am not a FIX expert so I can't say if this solution would work but it seems that if the logic for sending chunks of ResendRequest messages is changed such that each of the ResendRequest chunk messages is defined immediately after the need for a ResendRequest arises and put into a queue (or some other object) QuickFIX/J could then pop the each ResendRequest message off the queue once it receives all of the messages for a given ResendRequest from the exchange and continue onto the next one and decouple itself in that way from the newSequenceNumber. This might ensure that any of the chunks of ResendRequest messages that need to get sent out don't get skipped. This might have negative implications as it might not be the standard logic for communicating via FIX however so it might not be a feasible solution however.

It would be really great if this functionality could be modified in QuickFIX/J as this is an extremely useful engine however for users attempting to use QuickFIX/J with CME's Drop Copy 4 this case will cause problems in a recovery scenario.

Hi John,

yes, a message log would be a good starting point. From this I should hopefully be able to construct an automated test case.

Thanks, Chris.

Hi John, Xiaojun Zhang: is there still the need to support this? Do you have any message logs?
Thanks, Chris.

Hi Christoph John,

I can confirm that there is a need to support this functionality. To be clear, I was experiencing this issue when working with the 1.6.0 release of QuickFIX/J and have not tested with more recent releases. I don't have any message logs available currently aside from what Xiaojun posted earlier in this issue.

Thanks

Hi John, OK, if this functionality was implemented could you test it against CME to verify if it works? Do they have some sort of certification test?

Hey John
I know it's been two years, but did you pass the certification in the end?

My case is

there are 2 fix session both are acceptor,have separate file store. but the 2 session's fix-version, SenderCompId, TargetCompID are same,only the port number is different,I didn't use property file to config the session but config the 2 sessions in code.

receive message worked well!
but if I use Session.lookupSession(sessionID).send(message);
the 2 session's sessionID is same, so Session.lookupSession(sessionID) always return the session2.

Can you use session qualifier field here? "SessionQualifier Additional qualifier to disambiguate otherwise identical sessions"

Also it will solve JMX problem if you have it. Both sessions will try to expose same MBean otherwise

A SessionQualifier can only be used with an initiator.

Apache Tomcat 8 on Windows 10, Spring 4 web application

Hmm, looks more like a false positive to me. But to be honest I never tried running QFJ in a managed environment or appserver.
http://stackoverflow.com/questions/28105803/tomcat8-memory-leak says that the ThreadLocals need to be removed but I think this is not done because it is expected that QFJ is ran in a stand-alone environment.

Would it be possible to call a method or something from outside of quickfix to manually shutdown all spawned threads when exiting my webapp?

From what I understand from the output it does not have to do anything with threads but ThreadLocal variables that are still held. But AFAIK there is no built-in mechanism to clean them up.

Java

QFJ currently only supports FIX5.0 up to SP2. The message PartyDetailsListRequest is from EP105 (extentsion pack) so it is not yet supported by QFJ.

Thanks! Do you know when the EP105 will be supported by QFJ?

No, sorry. It will probably wait until someone volunteers to update the data dictionary accordingly.

For immediate use I apply this patch file to be safe. However this should be re-visited with a more elegant fix.

Thanks for the patch!
Edit: did you encounter dirty read problems?

No I have not seen any problems but after discovery in code review I was uncomfortable running it without a fix.

I have not seen any negative consequences to applying the patch. So it was a defensive move.

N/a its in the java code

Not sure how best to address this - should the application DataDictionary be created when the Transport one is?

Or should the settings be made available to the DefaultDataDictionaryProvider class?

Please note that ProcessQ Thread is application thread which read objects from application Queue and not the the Quick Fix internal queue .

When this happens, do you have to restart your application or does the problem correct itself? Are you starting/stopping your Initiator/Acceptor at certain times? This sounds a little like QFJ-825.
Would it be possible to create a stack dump with the program "jstack" once the error appears?

Sorry rephrasing the above JIRA:

• We noticed 2 threads logging the string --> "quickfixj.msg.incoming" at the same time. One of the threads to log this was SocketConnectorIoProcessor-0.0, Surprisingly the second thread which logs this is the ProcesQ thread
• The ProcessQ thread is our own application thread so we are not sure how this could happen?
• This does not seem to be a logging issue, we are using log4j2 as our logger. I say that because of the 2 consecutive messages received one after the other, Quickfix/J complains when the second message comes in:

  2015-04-28 14:16:03,857 [QFJ Message Processor] INFO  quickfixj.msg.outgoing - FIX.4.2:MDAQ_DEV2_SANDPIT_MD->ANZD_SANDPIT_MD: 8=FIX.4.2^A9=141^A35=5^A34=11955^A49=MDAQ_DEV2_SANDPIT_MD^A52=20150428-06:16:03.857^A56=ANZD_SANDPIT_MD^A58=MsgSeqNum too low, expecting 158758 but received 158757^A10=030^A
  

All the log files below:

2015-04-28 14:16:03,856 [ProcessQ] INFO quickfixj.msg.incoming - FIX.4.2:MDAQ_DEV2_SANDPIT_MD->ANZD_SANDPIT_MD: 8=FIX.4.2^A9=1500^A35=X^A34=158757^A49=ANZD_SANDPIT_MD^A52=20150428-06:16:03.814^A56=MDAQ_DEV2_SANDPIT_MD^A262=9^A268=26^A279=2^A269=0^A278=7usB0^A55=USD/CHF^A15=USD^A279=2^A269=1^A278=7usA0^A55=USD/CHF^A15=USD^A279=2^A269=0^A278=7usB1^A55=USD/CHF^A15=USD^A279=2^A269=1^A278=7usA1^A55=USD/CHF^A15=USD^A279=2^A269=0^A278=7usB2^A55=USD/CHF^A15=USD^A279=2^A269=1^A278=7usA2^A55=USD/CHF^A15=USD^A279=2^A269=0^A278=7usB3^A55=USD/CHF^A15=USD^A279=2^A269=1^A278=7usA3^A55=USD/CHF^A15=USD^A279=2^A269=0^A278=7usB4^A55=USD/CHF^A15=USD^A279=2^A269=1^A278=7usA4^A55=USD/CHF^A15=USD^A279=2^A269=0^A278=7usB5^A55=USD/CHF^A15=USD^A279=2^A269=1^A278=7usA5^A55=USD/CHF^A15=USD^A279=2^A269=0^A278=7usB6^A55=USD/CHF^A15=USD^A279=2^A269=1^A278=7usA6^A55=USD/CHF^A15=USD^A279=0^A269=0^A278=7utB0^A55=USD/CHF^A270=0.95388^A15=USD^A271=1000000^A346=1^A279=0^A269=1^A278=7utA0^A55=USD/CHF^A270=0.95413^A15=USD^A271=1000000^A346=1^A279=0^A269=0^A278=7utB1^A55=USD/CHF^A270=0.95385^A15=USD^A271=2000000^A346=1^A279=0^A269=1^A278=7utA1^A55=USD/CHF^A270=0.95416^A15=USD^A271=2000000^A346=1^A279=0^A269=0^A278=7utB2^A55=USD/CHF^A270=0.95382^A15=USD^A271=2000000^A346=1^A279=0^A269=1^A278=7utA2^A55=USD/CHF^A270=0.95419^A15=USD^A271=2000000^A346=1^A279=0^A269=0^A278=7utB3^A55=USD/CHF^A270=0.95380^A15=USD^A271=5000000^A346=1^A279=0^A269=1^A278=7utA3^A55=USD/CHF^A270=0.95421^A15=USD^A271=5000000^A346=1^A279=0^A269=0^A278=7utB4^A55=USD/CHF^A270=0.95377^A15=USD^A271=10000000^A346=1^A279=0^A269=1^A278=7utA4^A55=USD/CHF^A270=0.95424^A15=USD^A271=10000000^A346=1^A279=0^A269=0^A278=7utB5^A55=USD/CHF^A270=0.95374^A15=USD^A271=10000000^A346=1^A279=0^A269=1^A278=7utA5^A55=USD/CHF^A270=0.95427^A15=USD^A271=10000000^A346=1^A10=041^A


2015-04-28 14:16:03,856 [SocketConnectorIoProcessor-0.0] INFO quickfixj.msg.incoming - FIX.4.2:MDAQ_DEV2_SANDPIT_MD->ANZD_SANDPIT_MD: 8=FIX.4.2^A9=1500^A35=X^A34=158757^A49=ANZD_SANDPIT_MD^A52=20150428-06:16:03.814^A56=MDAQ_DEV2_SANDPIT_MD^A262=9^A268=26^A279=2^A269=0^A278=7usB0^A55=USD/CHF^A15=USD^A279=2^A269=1^A278=7usA0^A55=USD/CHF^A15=USD^A279=2^A269=0^A278=7usB1^A55=USD/CHF^A15=USD^A279=2^A269=1^A278=7usA1^A55=USD/CHF^A15=USD^A279=2^A269=0^A278=7usB2^A55=USD/CHF^A15=USD^A279=2^A269=1^A278=7usA2^A55=USD/CHF^A15=USD^A279=2^A269=0^A278=7usB3^A55=USD/CHF^A15=USD^A279=2^A269=1^A278=7usA3^A55=USD/CHF^A15=USD^A279=2^A269=0^A278=7usB4^A55=USD/CHF^A15=USD^A279=2^A269=1^A278=7usA4^A55=USD/CHF^A15=USD^A279=2^A269=0^A278=7usB5^A55=USD/CHF^A15=USD^A279=2^A269=1^A278=7usA5^A55=USD/CHF^A15=USD^A279=2^A269=0^A278=7usB6^A55=USD/CHF^A15=USD^A279=2^A269=1^A278=7usA6^A55=USD/CHF^A15=USD^A279=0^A269=0^A278=7utB0^A55=USD/CHF^A270=0.95388^A15=USD^A271=1000000^A346=1^A279=0^A269=1^A278=7utA0^A55=USD/CHF^A270=0.95413^A15=USD^A271=1000000^A346=1^A279=0^A269=0^A278=7utB1^A55=USD/CHF^A270=0.95385^A15=USD^A271=2000000^A346=1^A279=0^A269=1
^A278=7utA1^A55=USD/CHF^A270=0.95416^A15=USD^A271=2000000^A346=1^A279=0^A269=0^A278=7utB2^A55=USD/CHF^A270=0.95382^A15=USD^A271=2000000^A346=1^A279=0^A269=1^A278=7utA2^A55=USD/CHF^A270=0.95419^A15=USD^A271=2000000^A346=1^A279=0^A269=0^A278=7utB3^A55=USD/CHF^A270=0.95380^A15=USD^A271=5000000^A346=1^A279=0^A269=1^A278=7utA3^A55=USD/CHF^A270=0.95421^A15=USD^A271=5000000^A346=1^A279=0^A269=0^A278=7utB4^A55=USD/CHF^A270=0.95377^A15=USD^A271=10000000^A346=1^A279=0^A269=1^A278=7utA4^A55=USD/CHF^A270=0.95424^A15=USD^A271=10000000^A346=1^A279=0^A269=0^A278=7utB5^A55=USD/CHF^A270=0.95374^A15=USD^A271=10000000^A346=1^A279=0^A269=1^A278=7utA5^A55=USD/CHF^A270=0.95427^A15=USD^A271=10000000^A346=1^A10=041^A

2015-04-28 14:16:03,857 [QFJ Message Processor] INFO  quickfixj.msg.outgoing - FIX.4.2:MDAQ_DEV2_SANDPIT_MD->ANZD_SANDPIT_MD: 8=FIX.4.2^A9=141^A35=5^A34=11955^A49=MDAQ_DEV2_SANDPIT_MD^A52=20150428-06:16:03.857^A56=ANZD_SANDPIT_MD^A58=MsgSeqNum too low, expecting 158758 but received 158757^A10=030^A

Could you please tell us:

• If this is a bug in 1.5.3 version of Quickfix/J?
• We only notice this happening 4 times a day during peak load, 2000 market data messages a second
• We did not seem to encounter this in 1.5.2

Could you please answer my questions from the last comment?
As already said, the only thing I could imagine is that you are hitting QFJ-825. But to be sure I would need a stack dump.
Are you able to try without log4j2? It's probably not the problem, but just to be sure... Are you using asynchronous logging?

Please find my answer here:

• No we are not stopping are starting Initiator/Acceptor at certain times.
• Also When this happened, we didn't restart the application as mention above due to this we get the sequence error as a result of that log-out has sent to target session after a while it connect back again as part of login retry attempts.

• This is not happening daily basis, we are trying to reproduce the problem at our end if succeeds we will share the JStack's stack dump with you. However No luck so far.
• We are using synchronous logging also this is unlikely to have log4j2 issue as its result in sequence error.

OK, without the stack dump I cannot really tell anything. Just let me know once you have it.
More questions: are you using a ThreadedSocketAcceptor/Initiator? Or the default SocketAcceptor/Initiator? is there only one session that you have configured? Could you please post your session config?

And I guess you are certain that the counterparty did not just send the same message twice?

1) We are using default Socket Initiator.
2) Have just one session .
3) Session config is as here
DataDictionary=config/FIX42.xml
ResetOnLogon=Y
ScreenLogShowOutgoing=N
FileLogHeartbeats=N
ResetOnDisconnect=Y
ScreenLogEvents=N
ConnectionType=initiator
SocketKeyStore=config/SummitInvt.pem
TargetCompID=ANZD_SANDPIT_MD
SenderCompID=MDAQ_DEV2_SANDPIT_MD
SocketConnectHost=1.x.x.x
ReconnectInterval=10
ScreenLogShowIncoming=N
SocketConnectPort=xxxx
SocketUseSSL=Y
FileStorePath=var/fix/fxlp
PersistMessages=N
EndTime=17:00:00
SocketKeyStorePassword=*******
TimeZone=America/New_York
StartTime=17:10:00
HeartBtInt=20
UseDataDictionary=Y
ResetOnLogout=Y
Username=TODO
e-mail=
SessionID= FIX.4.2:MDAQ_DEV2_SANDPIT_MD->ANZD_SANDPIT_MD

4)and yes there is no replay of same message from counter party .

Hmm, now that I see that you are using SSL I am asking myself if it could be the same problem as in QFJ-745. However, there is not really much information to find the problem and the issue creator did not come back with answers.

Linux Server

Hi, some questions:

• can you reproduce the problem or did it appear only once?
• did you change something in the session settings before the problem appeared?
• how do you use the FileLog? Are you creating it on your own?

It does not appear to be reproducible. We did have exceptionally high disk load when it happened.

We did not make any changes to the session settings

We are using the default FileLog without any modifications.

As additional piece of information, this is the code that we use to send FIX messages:
public synchronized boolean sendMessage(Message message) throws SessionNotFound

We also have multiple threads call this function.

I'll see if I try to reproduce it

Hi, any approximate ETA on this issue? it would help us a great deal if this is done. Thanks

Hi praneeth thonupunoori,
no-one is working on this currently I'm afraid. So we cannot give an ETA.
Chris.

So you propose to remove the logging and call Thread.currentThread().interrupt() instead?

Yes.

Alternatively, stop catching the InterruptedException and let it be thrown by the method. The impact would be:
On the SessionConnector class:

• "protected void waitForLogout()" would change to "protected void waitForLogout(...) InterruptedException"
• "protected void logoutAllSessions(...)" would change to "protected void logoutAllSessions(...) InterruptedException"
• "public void stop(...)" would change to "public void stop(...) InterruptedException"
  On the Connector interface:
• "void stop(...)" would change to "void stop(...) InterruptedException"

This would have repercussions on the Connector interface and break backward compatibility.
Re-asserting the interrupted state on the thread as you point out might be preferable as it would still provide a way for higher level code to detect the interruption and remain backward compatible.

OK, then we should change it to "Thread.currentThread().interrupt()".
What about the usages in SingleThreadedEventHandlingStrategy and ThreadPerSessionEventHandlingStrategy. Is it also feasible to change it there?

SingleThreadedEventHandlingStrategy

40: I think 'isStopped' should be volatile because it is read and set by 2 different threads

50: } catch (InterruptedException e)

catch (InterruptedException e)

No change. Can be ignored as this thread is never stopped. We don't even need a handle to it as its run() method just completes.

ThreadPerSessionEventHandlingStrategy

116: } catch (final InterruptedException e)

This is called from Mina's code so it should never happen. So I think it's reasonable to just log an error in the session's log.
Please note though that the same in SingleThreadedEventHandlingStrategy throws a RuntimeException instead. Maybe we should go with 1 alternative for both. Either log something or throw a RuntimeException.
Same as before, an improved handling would be for enqueue(...) (and onMessage(...) that calls it) not to catch it and instead throw it so not to dirupt anything inside Mina (or the Transport layer in general) that might want to know about it. But it's probably not worth it changing the interface at this time.

137:
} catch (final InterruptedException e) {
LogUtil.logThrowable(quickfixSession.getSessionID(),
"Message dispatcher interrupted", e);
stopping = true;
As far as I can tell the dispatching threads (MessageDispatchingThread) are never stopped with Thread.stop() so the catch below will never happen (as the code stands now).
One might as well leave the code as it is in case in future Thread.stop() is called on the MessageDispatchingThread object (e.g. via getDispatcher(...) that can expose it to the outside).
Still I would add Thread.currentThread().interrupt() to the code in the catch block to reassert the thread status.

One suggestion: in both SingleThreadedEventHandlingStrategy and ThreadPerSessionEventHandlingStrategy there are thread that are instantiated.
They are then notified to stop using a volatile variable. I would suggest replacing this mechanism with one more 'natural' to thread semantic.
Whereas the loop should be: while (!Thread.currentThread().isInterrupted())

Then when they need to be stopped one simply calls myThread.stop().
The while loop will exit and any clean up can be carried out before the run() method returns.
At this stage though, as the code works, it wouldn't add any benefits.

Hi Francesco,

thanks for your input.
During work on QFJ-790 I already changed the isStopped flag to volatile.

If it is OK with you I would put this issue to the next major release to resolve the remaining points.

Thanks,
Chris.

Mac OSX

test case.

It turned out that the setField( int, Field<?> ) method should rather not be used with non-String tags. Rather use setField( Field ), e.g.:
ioi.getHeader().setField(new XmlData(xml));
ioi.getHeader().setField(new XmlDataLen(xml.length()));

Of course, there should be means to prevent your original error. Either setField( int, Field<?> ) should convert the values (as setField( Field ) does) or the validate-method should not assume that all fields are Strings.

Attached a fixed version of the test.

This is related to the QF/J-specific behaviour to always accept a Logout message regardless of the sequence number. But it probably would not hurt to change this in order to be more in line with the FIX spec.

How do you register your connector? Can't you simply unregister it prior to re-registering it? Briefly looking at the code it seems there is no unregister method in class org.quickfixj.jmx.JmxExporter.
But shouldn't it work to unregister the connector by yourself by calling jmxExporter.getMBeanServer().unregisterMBean( objectName )?

I do call jmxExporter.getMBeanServer().unregisterMBean(objectName);.

I will expand on my problem. I need to be able to add and remove sessions at runtime. To achieve this a have a connector per session (there are other reasons that prevents me to use add/remove dynamic sessions, but I think the problem would still be present).
If I want a session to go away I just stop the connector and unregister the bean.
If I want a session to come back some time later I create new settings and new connector instance, but when the session id is the same the session does not show up.

It is because when I unregister the connector the sessionObjectNames map in SessionJmxExporter is still containing entry for my session id.

I think the corresponding entry in SessionJmxExporter.sessionObjectNames should be removed either in ConnectorAdmin.postDeregister or SessionAdmin.postDeregister.

What do you think?

OK, I also see in ConnectorAdmin.registerSessions() that the session is only registered if it is not already contained in the sessionObjectNames map. So, yes, your proposed solution to remove the entry in one of the postDeregister() methods should be OK.

I am using a customized DynamicAcceptorSessionProvider in 1.5.3 to create dynamic session with my own session configs instead of the static SessionSettings. Even though I call sessionConnector.removeDynamicSession(sessionID) before sessionConnector.addDynamicSession(session), the previous session is not destroyed. The behaviour includes but not limited to 1) the session jmx doesn't work (i.e. any operation has no effect), 2) the new session thread doesn't own the file handlers of the session store files, so that client can't connect with ResetSeqnum on when logon.

Will the proposed solution solve this issue? Has the solution been added to version 1.6.0? Has anyone been involved in fixing this issue?

This issue is currently only planned in to be fixed for 1.6.0 but has not been fixed yet. Could you possibly supply a test case?

I think the problem lies within the specification of the XmlMessage itself. In FIXimate (http://www.fixtradingcommunity.org/FIXimate/FIXimate3.0/) the MsgType n is only specified having a header and a trailer, but no body fields. When adding this information to the data dictionary and regenerating the code the MessageFactory returns a message but when adding fields and validating the message the error "No fields found" is thrown by the DataDictionary. So it seems that the XmlMessage cannot be completely supported.
Another point is that by definition of the FIXimate msgType n is an admin message but QuickFIX/J currently will not treat it as admin message (see MessageUtils.isAdminMessage()). I do not know if that will cause additional problems.

The same issue could be reproduced when the logon request is timing out.

There are 2 threads trying to do the same thing:
16:18:28.763 [QFJ Timer] INFO quickfix.FileStore - initialize() called from.
at quickfix.FileStore.initialize(FileStore.java:111) [quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.FileStore.reset(FileStore.java:465) [quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.SessionState.reset(SessionState.java:382) [quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.Session.resetState(Session.java:2503) [quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.Session.disconnect(Session.java:1968) [quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.Session.next(Session.java:1808) [quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.mina.SessionConnector$SessionTimerTask.run(SessionConnector.java:278) [quickfixj-core-1.6.2.jar:1.6.2]

And from:
16:18:28.788 [QFJ Message Processor] INFO quickfix.FileStore - initialize() called from.
at quickfix.FileStore.initialize(FileStore.java:111) [quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.FileStore.reset(FileStore.java:465) [quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.SessionState.reset(SessionState.java:382) [quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.Session.resetState(Session.java:2503) [quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.Session.disconnect(Session.java:1968) [quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.Session.next(Session.java:882) [quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.Session.next(Session.java:1109) [quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.mina.SingleThreadedEventHandlingStrategy$SessionMessageEvent.processMessage(SingleThreadedEventHandlingStrategy.java:144) [quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.mina.SingleThreadedEventHandlingStrategy.block(SingleThreadedEventHandlingStrategy.java:91) [quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.mina.SingleThreadedEventHandlingStrategy$1.run(SingleThreadedEventHandlingStrategy.java:125) [quickfixj-core-1.6.2.jar:1.6.2]

This causes unexpected behaviour and leaks File Descriptors.
Example:
ERROR [QFJ Timer] quickfix.SocketInitiator Error during timer processing
quickfix.RuntimeError: java.io.FileNotFoundException: [/..../Session].header (Too many open files)
at quickfix.SessionState.reset(SessionState.java:384) ~[quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.Session.resetState(Session.java:2499) ~[quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.Session.disconnect(Session.java:1967) ~[quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.Session.next(Session.java:1808) ~[quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.mina.SessionConnector$SessionTimerTask.run(SessionConnector.java:278) [quickfixj-core-1.6.2.jar:1.6.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_74]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
Caused by: java.io.FileNotFoundException: [/.../Session].header (Too many open files)
at java.io.FileOutputStream.open0(Native Method) ~[?:1.8.0_74]
at java.io.FileOutputStream.open(FileOutputStream.java:270) ~[?:1.8.0_74]
at java.io.FileOutputStream.<init>(FileOutputStream.java:213) ~[?:1.8.0_74]
at java.io.FileOutputStream.<init>(FileOutputStream.java:133) ~[?:1.8.0_74]
at quickfix.FileStore.initializeMessageIndex(FileStore.java:198) ~[quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.FileStore.initializeCache(FileStore.java:121) ~[quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.FileStore.initialize(FileStore.java:116) ~[quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.FileStore.reset(FileStore.java:442) ~[quickfixj-core-1.6.2.jar:1.6.2]
at quickfix.SessionState.reset(SessionState.java:382) ~[quickfixj-core-1.6.2.jar:1.6.2]
... 11 more

CentOS release 5.9 (Final)
Linux dev99 2.6.18-348.6.1.el5 #1 SMP Tue May 21 15:29:55 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux

java version "1.6.0_32"
Java(TM) SE Runtime Environment (build 1.6.0_32-b05)
Java HotSpot(TM) 64-Bit Server VM (build 20.7-b02, mixed mode)

Please excuse me, could you please elaborate on what exactly the problem is? Thank you.

Problem in concurrent parsing and modification of incoming MsgSeqNum counter by 2 threads:
1. "QFJ Message Processor" thread
2. "SocketConnectorIoProcessor" thread

As result SocketConnectorIoProcessor "lose" messages, report error and request to resend

Do you have some more information on this? Message logs maybe?
The stack trace only lists one thread. Do you also have a stack trace of the other involved threads?

Windows 7, using JBoss and camel-quickfix

Would it be OK for you to attach the patch to this ticket? I do not have the necessary rights to grant you SVN access but could integrate the patch shortly.

Hi, Christoph. Look if it helps. I've attached the patch as you requested.

Best regards.

bug fix

Dear Ryan,

we appreciate your idea and would like to consider it.
It is true the QF/J is quite monolithic and would greatly benefit from being split into more components.
E.g.

• separate IO layer in order to allow to replace MINA by Netty, etc (QFJ-598)

But currently there is no roadmap what should be done.

But it would be good if you could start a page in QF/J Confluence to outline your thoughts
and perhaps attached some code.

Thanks, Jörg

Hi Jorg,

Thanks for your feedback. I will certainly attach a patch once I've extracted out the pieces I need to.

Following that, I will look at creating a Confluence page as well (quite a few of the Confluence pages seem quite old as well I've noticed). Do you have a preference of which hierarchy you would like it placed in?

Ryan

A new FIX session is defined by resetting the numbers to 1. Or in other words: You cannot start a new session
without setting numbers back to 1.

The ResetOnLogon forces to reset the sequence numbers even if we are in the normal [StartTime, EndTime] frame.

So "Time of day that this FIX session becomes activated" should get "Time of day when the new FIX session starts".

Actually, the new FIX session starts some where in between the previous days EndTime and the todays StartTime.
And the engine really resets the sequence numbers if it detects that it is out of session time (e.g. after last days EndTime).

Thanks for the clarification, Jörg. You can go ahead and close this ticket.

Clone of QFJ-392, since I appear unable to re-open it.
Could someone give me the fix version where this was resolved? I too would like a qualifier for Acceptors.

I see now also QFJ-207, which is closed with a 'Wont Fix'. Please disregard if that is still the case.

I also have this error. After second unnecessary LogOut, when I am sending LogOn, in field 34 is bad value (to high).

Scenario:
Me (initiator): Logout t34=204
Counterparty (acceptor): Logout t34=315
Me: Logout* t34=205 (counterparty don't receive this message, but my outgoing_seqnum is incremented)
Me: Logon t34=206, t789=316
Counterparty: t34=316, t789=205 (205 was second Logout*)
Me: SequenceReset t34=205, 36=207 <-- this message sould not be sent. This is side effect of Logout* (t34=205)

Very sorry for my English

I've resolved problem by add "synchronized" modifier to methods:
quickfix.Session::next()
quickfix.Session::next(Message)
ugly, but looks work properly.

I could reproduce this issue. It seems to happen if there is a very low latency in the network connection between client and server (e.g. on the same host).

Two things happen very quickly:

• Incoming Logout is processed before the sentLogout flag in the session state is set / published.
• Shortly after that the socket is closed: "Disconnecting: IO Session closed"

QFJ tries to answer the Logout it considers as request and fails:

• event : No responder, not sending message: 8=FIX.4.4|9=53|35=5|34=...

But this message is kept in the message store and leads to a ResendRequest at the next logon.

IHMO, syncing on next() is too brute-force. On the other hand, correcting the already complicated state handling is tricky.

Lowering priority since this is a race condition in a very specific situation and does not affect general functionality.

Windows 7 Ultimate, Java 6 u29
QuickFix J 1.5.1

I think this would be a good improvement. BTW, the link to the forum thread is wrong. It leads to a discussion about log rotation.

Windows 7 Professional x64 (latest updates)
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

java 1.6 on Windows

Did you try posting your question to the mailing list? This doesn't look like it should be a bug report.
http://quickfixj.org/support/

CompIDs are part of the SessionID, if you change them it is not the same Session anymore.
About the IP/port, you can specify multiple ip/port for the session and switch to them.

Regards,

Laurent DANESI

uname -a: Linux 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:56:28 EST 20 06 x86_64 x86_64 x86_64 GNU/Linux
Java VM: BEA JRockit(R) R27.6.0-50_o-100423-1.5.0_15-20080626-2104-linux-x86_64,BEA Systems, Inc.