[QFJ-765] CheckSum error if the declared number of groups does not match the actual number of groups Created: 12/Dec/13  Updated: 07/Jan/14  Resolved: 07/Jan/14

Status: Closed
Project: QuickFIX/J
Component/s: Engine
Affects Version/s: 1.5.3
Fix Version/s: None

Type: Bug Priority: Default
Reporter: Judit Kapinya Assignee: Christoph John
Resolution: Not a bug Votes: 0
Labels: None


 Description   

The scenario:

The initiator parses the following message with quickfix.MessageUtils.parse(), and sends it to the acceptor.

8=FIX.4.4|9=238|35=AE|31=1.35|32=1000|48=EURUSD|22=6|55=EURUSD|64=20131106|194=1.35|15011=1.33|15012=1.33|461=RCSXXX|487=2|571=1105200000001234|829=0|572=M1006050412587|552=1|54=1|37=SYS1:4545556002|11=SYS2:1006050412587|526=SSP:BDID001|453=2|448=a123456|447=D|452=3|802=2|523=5730607|803=10|523=5730606|803=15|1=123456-789|

It is not a valid FIX message, as we have 453<NoPartyIDs>=2, but there is only one group in the message. We would expect the acceptor to return a validation error, but instead the following happens:

1) An error message is logged:

2013-12-07 15:47:30,919 [SocketAcceptorIoProcessor-0.0] ERROR [] q.mina.acceptor.AcceptorIoHandler - Invalid message: Expected CheckSum=29, Received CheckSum=28 in 8=FIX.4.4?9=367?35=AE?34=82?49=SIDE1?52=20131207-14:47:30.618?56=SIDE2?22=6?31=1.35?32=1000?48=EURUSD?55=EURUSD?64=20131106?194=1.35?461=RCSXXX?487=2?571=1105200000001234?572=M1006050412587?829=0?15011=1.33?15012=1.33?552=1?54=1?37=SYS1:4545556002?11=SYS2:1006050412587?526=SSP:BDID001?453=2?448=a123456?447=D?452=3?802=2?523=5730607?803=10?523=5730606?803=15?1=123456-789?10=028?

2) The acceptor does not send any response and does not increase the message sequence count, either.

The suspect:

The acceptor validates the fix message with the checksum(String s) method of Message class. It calculates 29. But the checksum calculated on the initiator side is 28. The initiator calculates the checksum with a different method, which is called by the toString() method of Message class. This is the method which does the calculation (in FieldMap class) - my comments added:

int calculateTotal() {

int result = 0;
// JK: fields variable contains tags which are not group tags
for (final Field<?> field2 : fields.values()) {
final Field<?> field = field2;
if (field.getField() == CheckSum.FIELD || isGroupField(field.getField()))

{ continue; }

result += field.getTotal();
}

// JK: groups variable contains the group tags, but only the tag number (the key is Integer, not Field - why?)
final Iterator<Map.Entry<Integer, List<Group>>> iterator = groups.entrySet().iterator();
while (iterator.hasNext()) {
final Map.Entry<Integer, List<Group>> entry = iterator.next();
final List<Group> groupList = entry.getValue();
if (!groupList.isEmpty()) {
final IntField groupField = new IntField((entry.getKey()).intValue());
// JK: at this point we have 453=0, as this is a new field, no value yet
groupField.setValue(groupList.size());
// JK: now it is 453=1, because we have only one group
result += groupField.getTotal();
for (int i = 0; i < groupList.size(); i++)

{ final Group group = groupList.get(i); result += group.calculateTotal(); }

}
}

return result;
}

So the checksum will be the checksum of a different message actually, where 453=1.



 Comments   
Comment by Christoph John [ 16/Dec/13 ]

It is correct that the target message seqnum is not incremented on a checksum error. Since FIX takes an optimistic approach it is expected that the checksum is incorrect due to a transient error and that the subsequent resend will transmit a correct message.

Are you able to provide the original message that went over the wire? Otherwise it is hard to create a unit test. Trying with the message from point 1) above would give me a checksum of 214. I guess you have altered it in some way.

Comment by Judit Kapinya [ 02/Jan/14 ]

First of all happy new year

Yes, I altered the message - removed sensitive information. Now I've changed my settings, and here's an unaltered message from the initiator's log:

<20140102-14:46:32, FIX.4.4:SIDE1->SIDE2, outgoing> (8=FIX.4.4?9=357?35=AE?34=11?49=SIDE1?52=20140102-14:46:32.928?56=SIDE2?22=6?31=1.35?32=1000?48=EURUSD?55=EURUSD?64=20131106?194=1.35?461=RCSXXX?487=2?571=1105200000001234?572=M1006050412587?829=0?15011=1.33?15012=1.33?552=1?54=1?37=SYS1:4545556002?11=SYS2:1006050412587?526=SSP:BDID001?453=2?448=a123456?447=D?452=3?802=2?523=5730607?803=10?523=5730606?803=15?1=123456-789?10=220?)

And this is the checksum error on the acceptor side:

2014-01-02 15:46:33,009 [SocketAcceptorIoProcessor-0.0] ERROR [] q.mina.acceptor.AcceptorIoHandler - Invalid message: Expected CheckSum=221, Received CheckSum=220 in 8=FIX.4.4?9=357?35=AE?34=11?49=SIDE1?52=20140102-14:46:32.928?56=SIDE2?22=6?31=1.35?32=1000?48=EURUSD?55=EURUSD?64=20131106?194=1.35?461=RCSXXX?487=2?571=1105200000001234?572=M1006050412587?829=0?15011=1.33?15012=1.33?552=1?54=1?37=SYS1:4545556002?11=SYS2:1006050412587?526=SSP:BDID001?453=2?448=a123456?447=D?452=3?802=2?523=5730607?803=10?523=5730606?803=15?1=123456-789?10=220?

I hope this helps.

Comment by Christoph John [ 03/Jan/14 ]

Happy new year to you, too.

Sorry, but I still cannot exactly reproduce the error since there are some required fields missing in the message (570/PreviouslyReported, 75/TradeDate, 60/TransactTime) plus there are two user-defined fields that are not in the default FIX4.4 data dictionary (fields 15011 and 15012). But even when altering my data dictionary to accept your message I get a different checksum (namely 214, the same as in my former test).
Would it be possible for you to transmit a message that follows the default FIX4.4 dictionary?

Comment by Christoph John [ 03/Jan/14 ]

Correction, I get 204 as checksum. But should not matter.

Comment by Judit Kapinya [ 06/Jan/14 ]

I modified the FIX message to have the mandatory fields and removed the custom ones:

Initiator:
<20140106-15:26:35, FIX.4.4:SIDE2->SIDE1, outgoing> (8=FIX.4.4?9=373?35=AE?34=8?49=SIDE2?52=20140106-15:26:35.369?56=SIDE1?22=6?31=1.35?32=1000?48=EURUSD?55=EURUSD?60=20140106-23:00:00?64=20131106?75=
20140106?194=1.35?461=RCSXXX?487=2?570=Y?571=1105200000001234?572=M1006050412587?829=0?552=1?54=1?37=SYS1:4545556002?11=SYS2:1006050412587?526=SSP:BDID001?453=2?448=a123456?447=D?452=3?802=2?523=57306
07?803=10?523=5730606?803=15?1=123456-789?10=016?)

Acceptor:
2014-01-06 16:26:35,419 [SocketAcceptorIoProcessor-0.0] ERROR [] q.mina.acceptor.AcceptorIoHandler - Invalid message: Expected CheckSum=17, Received CheckSum=16 in 8=FIX.4.4?9=373?35=AE?34=8?49=SIDE2?
52=20140106-15:26:35.369?56=SIDE1?22=6?31=1.35?32=1000?48=EURUSD?55=EURUSD?60=20140106-23:00:00?64=20131106?75=20140106?194=1.35?461=RCSXXX?487=2?570=Y?571=1105200000001234?572=M1006050412587?829=0?55
2=1?54=1?37=SYS1:4545556002?11=SYS2:1006050412587?526=SSP:BDID001?453=2?448=a123456?447=D?452=3?802=2?523=5730607?803=10?523=5730606?803=15?1=123456-789?10=016?

The checksum can be different, as SendingTime/52 will be appended by the engine and will be different every time.

Comment by Christoph John [ 06/Jan/14 ]

Thanks, with that message I am able to reproduce the checksum problem. But it leaves me thinking about how the message has been constructed. If you constructed it with QFJ, e.g. message.addGroup() and so on and then the checksum calculation is wrong then that would be an error. But I actually did not manage to add repeating groups to the message in a way that would leave the group count tag in a wrong state. How did you achieve that?

Comment by Judit Kapinya [ 07/Jan/14 ]

The message was not constructed programmatically. It was created manually as part of a test harness, and in this particular scenario we would expect the engine to return a validation error message (as in case of out of order groups, for instance), but instead this checksum error occurs. This is not a typical scenario or not even highly probable to happen, but as the incoming FIX message is not a valid one, it would be nice to get a validation error. The checksum error can be quite misleading, as the calculated checksum at the client side is actually correct, but calculated for an invalid message. I hope this makes sense

Comment by Christoph John [ 07/Jan/14 ]

OK, I understand. But you will get a checksum error in this case because the validation of the checksum is done earlier (at message construction) than the group validation (when checking against the data dictionary). This can be seen at the end of method Message.parseGroup():
// For later validation that the group size matches the parsed group count
parent.setGroupCount(groupCountTag, declaredGroupCount);

QF/J just iterates over the group as often as the group delimiter is found and sets the declaredGroupCount for later validation.
If I alter the checksum on your message to the correct value I get "Incorrect NumInGroup count for repeating group", so this would be what you expect.

Comment by Judit Kapinya [ 07/Jan/14 ]

Ok, thanks for the explanation. So such a case should never happen if the message is constructed programmatically. We won't worry about such scenarios then.

Comment by Christoph John [ 07/Jan/14 ]

You're welcome. Feel free to ask at the mailing list if further questions arise. https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Thank you

Generated at Tue May 07 11:15:55 UTC 2024 using JIRA 7.5.2#75007-sha1:9f5725bb824792b3230a5d8716f0c13e296a3cae.