OpenSSL3.x PKCS7_encrypt API with FIPS provider error:1C8000A8:Provider routines::invalid padding…

OpenSSL3.x PKCS7_encrypt API with FIPS provider error:1C8000A8:Provider routines::invalid padding mode

Migrating from PKCS7 API to CMS API

📌 Background

Being a newbie to OpenSSL, encountered one issue while working on a SCEP (Simple Certificate Enrollment Protocol) implementation. Hence sharing the problem and solution for the same. I was using the PKCS7_encrypt() API to create PKCS7 enveloped data. SCEP supports RSA-based encryption, I was using AES-256-CBC as the symmetric cipher. After upgrading to OpenSSL 3.4, I started encountering the following error:

error:1C8000A8:Provider routines::invalid padding mode

⚠️ Root Cause

The issue was tied to FIPS (Federal Information Processing Standards) compliance. In OpenSSL 3.x, RSA encryption using PKCS#1 v1.5 padding is no longer FIPS-approved (weak padding, read the details here — *https://medium.com/asecuritysite-when-bob-met-alice/so-how-does-padding-work-in-rsa-6b34a123ca1f)*. This became problematic because PKCS7_encrypt() uses RSA with PKCS#1 v1.5 padding by default, and there's no direct API to change the padding scheme within the PKCS7 interface.

✅Solution

To comply with FIPS mode, RSA encryption must use OAEP (Optimal Asymmetric Encryption Padding) instead. However, since PKCS7_encrypt() doesn't allow changing the RSA padding mode, I migrated the implementation to use CMS (Cryptographic Message Syntax) APIs, which offer more flexibility and are a modern alternative to the older PKCS7 APIs.

// Step 1: Create the CMS structure (partial mode to allow customization)
CMS_ContentInfo* cmsenc = CMS_encrypt(
    NULL,               // No recipients added yet
    encbio,             // BIO containing the data to encrypt
    scepReq->scep->conf->enc_alg, // Encryption algorithm (e.g., AES-256-CBC FIPS approved)
    CMS_BINARY | CMS_PARTIAL      // Flags: binary data, partial init
);

if (cmsenc == NULL) {
    std::cout << "Failed to create CMS structure for encryption" << std::endl;
    return 0;
}

// Step 2: Add recipients with FIPS-compliant padding (RSA-OAEP)
for (int i = 0; i < sk_X509_num(recipients); ++i) {
    X509* cert = sk_X509_value(recipients, i);
    CMS_RecipientInfo* ri = CMS_add1_recipient(cmsenc, cert, NULL, NULL, CMS_KEY_PARAM | CMS_BINARY);

    if (ri == NULL) {
        std::cout << "Unable to add recipient" << std::endl;
        return 0;
    }

    // Access the EVP_PKEY_CTX to modify the padding
    EVP_PKEY_CTX* pctx = CMS_RecipientInfo_get0_pkey_ctx(ri);
    if (pctx == NULL) {
        std::cout << "Error retrieving key context" << std::endl;
        return 0;
    }

    // Set RSA padding to OAEP (FIPS-approved)
    if (EVP_PKEY_CTX_set_rsa_padding(pctx, RSA_PKCS1_OAEP_PADDING) == 0) {
        std::cout << "Error setting RSA padding mode to OAEP" << std::endl;
        return 0;
    }
}

// Step 3: Finalize the CMS structure
if (CMS_final(cmsenc, encbio, NULL, CMS_BINARY) <= 0) {
    std::cout << "Error finalizing CMS encrypted data" << std::endl;
    return 0;
}

🧩Final Notes

• FIPS mode disallows legacy padding schemes like PKCS#1 v1.5, which breaks older APIs PKCS7_encrypt.
• CMS APIs provide a more flexible alternative that allows setting parameters like padding.
• RSA-OAEP is the FIPS-compliant replacement and works seamlessly when configured through the CMS API.
• With the deprecation of certain features in PKCS7 and enhanced capabilities in CMS, migrating to CMS APIs is a strategic move for long-term maintainability and standards compliance.

At the end CMS API can be used in NON FIPS mode, and works perfectly fine.