RC2 implementation in SFL

["Jacoby, Jeffrey" <jjacoby@xxxxxxxxxxxxxxx>]

Greetings,

I've just starting working with the SFL and I believe there 
is a problem in the RC2 message decryption implementation 
in the RSA CTIL (and possibly in the crypto++ CTIL).  I've 
discussed this with the BSAFE Crypto-C manager and he agrees
the implementaion does not correctly follow the meaning of
effective key bits as specified in RFC 2268.

The problem occurs when the effective key bit parameter 
is different from the size of the key value used to create
the B_KEY_OBJECT (used for content encryption).  RC2 allows you 
to set a B_KEY_OBJECT's KI_Item information with one number 
of key bits, e.g. 128, while specifying a different ekb value
for the A_RC2_CBC_PARAMS.effectiveBits value, e.g. 40.  

This is NOT the same as setting the KI_Item information 
with just the first 40 bits from the 128 bit key and using 
an ekb of 40.  

In the algorithm library file sm_rsa.cpp (v1.3 of the 
library, but I think v1.5 is the same in this respect), 
in the SMTI_Decrypt() method is this code (I've inserted 
comments to indicate where I believe the problem is):

   // JJ: by this point, the effective key bits value
   // JJ: has been extracted, and the BSAFE algorithm 
   // JJ: information is set

   // Create a Key Object
   if ((status = B_CreateKeyObject(&rc2Key)) != 0)
      SME_THROW(status, "B_CreateKeyObject failed", NULL);

   // Set the key data from the incoming Mek
   rc2KeyItem.len = m_rc2Params.effectiveKeyBits / 8;
//JJ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
//JJ:  This is the error.  It's possible and allowed that
//JJ:  the rc2KeyItem.len may NOT be the same as ekb/8.  It
//JJ:  very well could be greater.
   SME(rc2KeyItem.data = (unsigned char *)calloc(1,rc2KeyItem.len));
   unsigned int tmpLen=rc2KeyItem.len;
   if (tmpLen > pMEK->Length())
      tmpLen = pMEK->Length();   // ONLY get what is available.
//JJ:  We really need to get *ALL* that's available!
   memcpy(rc2KeyItem.data, pMEK->Access(), tmpLen);

   if ((status = (long)B_SetKeyInfo(rc2Key, KI_Item,
         (POINTER)&rc2KeyItem)) != 0)

The result of this is that B_DecryptFinal() may fail during
envelopedData content decryption.  The solution is to set the
B_KEY_OBJECT's KI_Item information with ALL the content encryption 
key bits/bytes extracted from the message, not just the first
m_rc2Params.effectiveKeyBits .  

While I am fairly sure I am correct on this point, it is by no 
means guaranteed :).  Please let me know if you have any comments.

Sincerely,

Jeff Jacoby, Sr. QA Engineer

RSA Security, SMDC
2755 Campus Drive
San Mateo, CA 94403

phone: (650) 295-7569         
email: jjacoby@xxxxxxxxxxxxxxx