I am at my wits end with this problem. I am trying to implement an encrypted channel using RSA(4096) to deliver an AES 256 bit key using java (JRE7). The first message I send is the generated 256 AES key encrypted with the RSA public key. Upon decryption, I am getting the following error message: javax.crypto.BadPaddingException: Blocktype mismatch: 0
My decryption code:
private byte[] decryptMessage(Key key, byte[] data){
Cipher cipher;
System.out.println("Decrypting message...");
System.out.println("key: " + key.toString());
System.out.println("data: " + data);
System.out.println("data length: " + data.length);
if(key instanceof PublicKey){
//RSA public key
System.out.println("Found PublicKey");
try {
cipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");
cipher.init(Cipher.DECRYPT_MODE, key);
return cipher.doFinal(data);
} catch (NoSuchAlgorithmException | NoSuchPaddingException | InvalidKeyException | IllegalBlockSizeException | BadPaddingException e) {
// TODO Auto-generated catch block
System.out.println("Failed to decrypt message");
e.printStackTrace();
}
}else if(key instanceof PrivateKey){
//RSA private key
System.out.println("Found PrivateKey");
try {
cipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");
cipher.init(Cipher.DECRYPT_MODE, key);
byte[] results = cipher.doFinal(data);
System.out.println("Decrypted results: " + results);
return results;
} catch (NoSuchAlgorithmException | NoSuchPaddingException | InvalidKeyException | IllegalBlockSizeException | BadPaddingException e) {
// TODO Auto-generated catch block
System.out.println("Failed to decrypt message");
e.printStackTrace();
}
}else if(key instanceof SecretKey){
//AES secret key
try {
cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
cipher.init(Cipher.DECRYPT_MODE, key);
return cipher.doFinal(data);
} catch (NoSuchAlgorithmException | NoSuchPaddingException | InvalidKeyException | IllegalBlockSizeException | BadPaddingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}else{
System.out.println("Invalid key type");
}
return null;
}
My encryption code:
private byte[] encryptMessage(Key key, byte[] data){
Cipher cipher;
if(key instanceof PrivateKey || key instanceof PublicKey){
//RSA key
try {
cipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");
cipher.init(Cipher.ENCRYPT_MODE, key);
return cipher.doFinal(data);
} catch (NoSuchAlgorithmException | NoSuchPaddingException | InvalidKeyException | IllegalBlockSizeException | BadPaddingException e) {
// TODO Auto-generated catch block
System.out.println("Failed to encrypt message");
e.printStackTrace();
}
}else if(key instanceof SecretKey){
//AES secret key
try {
cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
cipher.init(Cipher.ENCRYPT_MODE, key);
return cipher.doFinal(data);
} catch (NoSuchAlgorithmException | NoSuchPaddingException | InvalidKeyException | IllegalBlockSizeException | BadPaddingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}else{
System.out.println("Invalid key type");
}
return null;
}
I googled and came to some similar stories(most on this website), but they were all problems with default Padding between to different devices, computer to android for example). I am getting the error when sending messages from the same computer and I have padding and ECB both set up. I went through the source code of com.sun.crypto.provider.RSACipher and sun.security.rsa.RSAPadding and found the error is being thrown in the sun.security.rsa.RSAPadding.unpadV15 method due to the second bit not being set properly. I am also using the new NIO2 classes in Java7 specifically the AsychronousSocketChannel and CompletionHandler objects so I don't know if they are causing threading errors though I highly doubt it. I found the source code at http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/com/sun/crypto/provider/RSACipher.java and http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/sun/security/rsa/RSAPadding.java#RSAPadding.padV15%28byte[]%29 but again they didn't help me much. I have tried dropping the key size and changing the encoding/padding options of the cipher objects, but it always errors out even with padding disabled. Though it may not matter, but since I am using JRE7, there is no stable support for bouncycastle so I am using the default security provider.
Thanks in advance