To show bug:
1) Create FIXT1.1 / FIX.5.0 session with custom TransportDataDictionary and ApplicationDataDictionary
2) Start session, send session level logins
3) Try sending Application-level message to QFix/J session with CstmApplVerID (field 1129) set in the header to some value (I chose '5.4' for the value)
4) QFix/J emits exception trying to find DataDictionary FIX50_5_4, where '5_4' reflects value of the CstmApplVerID field

Why it happens:
QFix/J indexes DataDictionaries by a AppVersionKey. This key uses ApplVerID and CstmApplVerID fields to generate a unique reference to a given DataDictionary. I think it is a correct interpretation of the FIX spec to say that it should only use ApplVerID, because CstmApplVerID (1) is not a required field, and thus may or may not consistently appear in message headers, (2) does not have a required default value in a configuration file (e.g. DefaultApplVerID) (3) is defined in the FIX spec as "Used to support bilaterally agreed custom functionality", in no way affecting the DataDictionary or implying that a specific one should be used.

The fix:
I fixed this by removing all references to CstmApplVerID in the Engine, which allows CstmApplVerID to behave like a normal header field, leaving the implementation of any functionality based on its value up to the user. Most of the revolved around DefaultDataDictionaryProvider.java.