TLS Encryption and Compliance

Many compliance frameworks require the use of encryption to protect sensitive data transmitted across the Internet, and Transport Layer Security (TLS) is often used to meet these encryption requirements. Unfortunately, I often encounter clients that do not understand how TLS works and, as a result, have deployed insecure TLS configurations that cause compliance failures.

This post will provide a high-level overview of TLS and the steps that should be taken to implement it in a secure manner that will meet just about any compliance requirements. A deep dive will then be taken to help understand how TLS works and why these configuration steps are important.

This post does not cover the specific requirements of any framework. Readers with PCI DSS requirements may want to review Steve Maxwell’s previous posts on Strength Training with Transport Cryptology Part 1 and Part 2.

Overview

At a high level, TLS can be thought of as a protocol for two (2) endpoints to:

1. Verify the identities of one (1) or both endpoints via the use of special files called certificates that contain cryptographic information
2. Conduct a secure negotiation for the purpose of identifying an appropriate collection of cryptographic algorithms, known as a cipher suite, that both endpoints support
3. Use the agreed-upon cipher suite to protect communications from unauthorized interception, impersonation, and alteration

TLS problems that can weaken security and cause compliance issues include:

• Flaws can be found in the TLS protocol itself that could allow an attacker to compromise the encryption, which requires some combination of:
  • Disabling older versions of the protocol
  • Installing vendor patches that address the flaws in later versions of the protocol
  • Implementing configuration settings to work around the flaws on a per-installation basis
• Weaknesses can also be found in the cryptographic algorithms used by the TLS protocol, which requires disabling the cipher suites that use these weak algorithms, even in the latest versions of the TLS protocol.
• Mismanagement of cryptographic certificates and keys can allow attackers to impersonate endpoints and/or decrypt sensitive data so they can read or alter it.
• Programmers may incorrectly implement the TLS protocol and cryptographic algorithms or otherwise introduce exploitable bugs in the software that must be addressed by vendor patches.

Common Problems and Solutions

Common compliance problems related to TLS encryption and their solutions (which are each covered in more detail below) include:

The sections below help explain what each of these mean, why they are problems, and how they can be addressed.

TLS Explained

A TLS implementation consists of what can be considered five (5) discrete parts, each of which is covered below:

• The protocol: A set of standardized procedures for negotiating which cryptographic algorithms to use
• Cipher suites: A set of standardized cryptographic algorithms for rendering information unreadable and verifying that information hasn’t been altered
• Certificates: A standardized file that can be used to cryptographically validate that a device is what it claims to be
• Key management: The handling of the cryptographic keys used during the encryption and decryption processes
• Software implementations: Software that implements the negotiation protocols, cipher suites, certificate generation, and key management processes

Negotiation Protocol

There is a common misunderstanding that TLS (short for Transport Layer Security) and its predecessor SSL (Secure Sockets Layer) are themselves a type of encryption. In reality, these are protocols for two (2) devices to securely negotiate how they will use encryption to secure their communications. A reference to the version of TLS or SSL—e.g., TLS 1.3 or SSL 3.0—refers to the version of the protocol that determines the procedures and rules for the negotiation process. Each version of the TLS and SSL protocol contains a standardized list of supported standard cryptographic algorithms that the devices can choose from.

The SSL and TLS protocols have been updated occasionally since the first SSL release in 1995. For our purposes, the change from SSL to TLS can be considered a change in name only, and the SSL versions can be thought of as previous versions of TLS. Most services that support TLS, such as web servers, mail servers, VPNs, VoIP, etc., can support multiple versions of the TLS protocol simultaneously. This allows new versions of TLS to be rolled out without immediately disabling the previous versions and breaking connectivity with other devices that do not yet support the latest version.

An update to TLS typically:

• Addresses weaknesses in the protocol itself that could allow an attacker to affect the negotiation process
• Adds new cipher suites to allow the use of newer and more secure cryptographic algorithms
• Removes old cipher suites with cryptographic algorithms that are no longer considered secure

The security of this negotiation protocol is important: If an attacker is able to intercept and alter the negotiation between the two (2) endpoints (a man-in-the-middle (MitM) attack) the attacker may be able to convince the endpoints to use weak encryption (or no encryption at all), thereby allowing the attacker to intercept, read, and potentially alter the communication itself. It’s important that organizations disable older, weak versions of TLS and SSL so that a MitM cannot trick the endpoints into falling back to an insecure version of the protocol that would allow this kind of attack.

As of 2026:

• TLS version 1.3, published in 2018, is the latest version and is widely supported.
• TLS version 1.2, published in 2008, is still considered secure if properly configured and is widely supported.
• All previous versions of TLS (1.0 and 1.1), as well as all versions of SSL, are considered insecure and deprecated and should no longer be used or enabled.

At least one (1) matching TLS version must be enabled on both endpoints to establish a TLS connection, but the full list of enabled TLS versions does not need to match.

Cipher Suites

What is a Cipher Suite?

Modern encryption is a complex process, as different types of cryptographic algorithms are useful for different purposes (e.g., protecting a message from eavesdropping vs. verifying a message hasn’t been altered). Combining these different types of algorithms into what is known as a cipher suite allows a much more effective and efficient encryption solution that can leverage the relative strengths of each algorithm while avoiding the weaknesses.

Many cryptographic algorithms can be used with different key lengths, where a longer key is stronger but less efficient than a shorter key for the same algorithm. Similarly, some cryptographic algorithms may be used in different modes that each have pros and cons depending on how they are used.

A TLS cipher suite will specify the algorithms with key lengths and modes where appropriate. A single algorithm may show up in many different cipher suites, each with a different combination of modes, key lengths, and other companion algorithms used for different purposes. More on cipher suite naming below.

Cipher Suites in TLS

Each version of the TLS protocol contains a standardized list of cipher suites. When a TLS session is initiated, the endpoints will communicate with each other to identify which cipher suites are supported by each endpoint and then agree on which of these supported cipher suites to use.

Cipher suites can be individually disabled in endpoint configurations and thereby removed from the available options during the TLS negotiation process, regardless of which versions of the TLS protocol are enabled. At least one (1) matching cipher suite must be enabled on both endpoints to establish a TLS connection, but the full list of enabled cipher suites does not need to match.

This functionality exists in case weaknesses are found in a specific cryptographic algorithm, in the way a cryptographic algorithm is used, or if cryptographic key length is determined to be too weak for continued use. It’s much easier to advise users to disable a single cipher suite in the TLS configuration than to roll out a whole new version of TLS each time a weakness is found in an algorithm.

Organizations should routinely monitor for security alerts about encryption flaws and disable cipher suites that use insecure algorithms. The definitive source of information on recommended cipher suites is the IANA TLS Parameters list. When reviewing the “Recommended” column in this list:

• “Y” (for “Yes”) indicates cipher suites that should be used.
• “D” (for “Deprecated”) indicates cipher suites that should not be used.
• “N” (for “No”) indicates cipher suites that have not reached consensus, have limited applicability, or are intended only for specific use cases (these are usually not used).

Cipher suites that are deprecated or not recommended usually have a link in the Reference column indicating why they have this status.

The DTLS-OK column indicates whether the cipher suite can be used with TLS over datagram protocols (e.g., UDP), as opposed to more common TLS connections over TCP.

Some compliance frameworks reference NIST SP 800-52 for guidance on acceptable cipher suites (and TLS protocol versions), but this document is updated infrequently, so it is not of much practical use beyond defining the minimum requirements for U.S. government agencies.

Ciphersuite.info is another resource often referenced by vulnerability scan tools to identify recommended, secure, weak, and insecure cipher suites. The site also has search capabilities that make it easy to list cipher suites in TLS v1.2 and TLS v1.3. However, this is an independent project that may fall behind the IANA recommendations. A quick check performed as this blog was being written shows some algorithms marked as “Secure” on Ciphersuite.info are marked as not recommended on the IANA list.

Cipher Suite Names

The name of a cipher suite identifies each of the cryptographic algorithms and key lengths it uses for each cryptographic purpose. These are presented here because it is important for system administrators to be able to interpret cipher suite names to recognize which cipher suites should be disabled if a particular algorithm or key length is found to be weak.

A standardized cipher suite name format exists, but unfortunately, it is not universally adhered to (some software might use a dash instead of an underscore and/or omit the “TLS” and “WITH” text shown below). Cipher suite names in tools should be similar to the formats shown below, where the text in [brackets] is a stand-in for the algorithm names and key lengths.

Each cipher suite is also assigned a unique code that may be displayed by some tools. These codes are a pair of 256-bit hexadecimal numbers and are typically displayed as shown in the examples below.

Certificates

What is a Certificate?

TLS certificates are used to verify the identity of one (1) or both ends of a TLS connection at the start of the negotiation process. Without the correct use of certificates, an attacker could impersonate the endpoints and perform a MitM attack whereby the attacker intercepts, decrypts, and reads or potentially alters traffic from endpoints before re-encrypting the traffic and retransmitting it to the intended recipient.

A TLS certificate is a specially formatted file containing the public part of a two (2) part encryption key that can be shared to enable TLS communication. The other private part of the key is stored on one (1) or more devices that the certificate represents and must be kept secret. Additional data in the certificate can identify the certificate owner, the domains for which the public key is considered valid, an expiration date, and other details.

A device uses TLS to verify the identity of a remote endpoint by requesting a certificate file from the remote endpoint, checking the trust status of the certificate signatures as described below, and exchanging messages to verify that the remote endpoint possesses the private key associated with the public key found in the certificate file.

In some communications only the identity of one (1) endpoint will be verified, for example a web browser connecting to an online banking website will use a certificate to verify the bank’s web server isn’t being impersonated while the bank’s web server will verify the user’s identity via a password and multi-factor authentication rather than a certificate. In other communications the identity of both endpoints matter and will be verified, e.g., both endpoints of a site-to-site VPN will typically verify each other’s identities using certificates to ensure neither endpoint is being impersonated.

Certificate Signing and Trust

An attacker can generate their own keys and certificate that claim to be for any domain in an attempt to impersonate endpoints, thereby launching a MitM attack. To solve this problem, certificates must be cryptographically signed by an authority that attests the certificate is legitimate, and TLS implementations must verify these signatures to determine if the certificate can be trusted. There are three (3) ways this is accomplished, each of which has its own use cases:

Using the wrong type of certificate signature often causes compliance issues.

It is fairly common to encounter self-signed certificates in contexts where they are inappropriate. It is also somewhat common to encounter self-signed certificates in contexts where they could be appropriate, but the organization that issued the certificate did not properly configure the endpoints to exclusively trust specific authorized self-signed certificates, thereby rendering TLS ineffective. Organizations should use public or private CAs whenever possible and carefully configure systems to only trust authorized certificates when self-signed certificates are used.

Using a private CA requires strong internal process and technology controls to protect the private CA and its keys. An attacker that compromises the CA, including an insider threat with access to the CA, can issue rogue certificates that can be used to impersonate other systems. Organizations that are not mature enough to manage a private CA should use a public CA instead.

Certificate Pinning

Most software that implements TLS will, by default, attempt to verify a certificate’s signature using a set of public CA certificates preloaded into the software by the vendor (and potentially additional certificates associated with a private CA configured by an individual organization) and then accept any certificate that passes the signature verification test.

An alternative to this method of operation is to “pin” a certificate. This is where TLS software is configured to only accept a specific certificate for a specific host and reject all other certificates.

Certificate pinning is strongly recommended when using a self-signed certificate in a production environment. This is because self-signed certificates will never pass the default signature verification test, as they are not signed by any public or private signing authorities. Configuring a TLS instance to allow certificates that fail signature verification so self-signed certificates can be used without pinning would allow an attacker to impersonate any system.

Certificate pinning can also be useful with certificates signed by a public or private CA in high-security applications. Pinning a specific certificate prevents the TLS software from accepting rogue certificates for a specific host, even if an otherwise valid public or private CA is compromised.

Certificate Generation

Certificates both contain components of and are generated by cryptographic algorithms. Using insecure or weak cryptographic algorithms or key lengths when generating certificates may allow certificates to be forged and TLS connections to be compromised.

Organizations should take care to use strong algorithms and key lengths when generating keys to be used in a certificate and, if using a private CA, sign the certificates using strong signature algorithms. As of publication, ECC keys are gaining favor over the long-used RSA keys due to equivalent security with much shorter key lengths. 256-bit ECC keys offer a moderate level of security roughly equivalent to 2,048-bit RSA keys; SHA-256 is typically used for the signature.

Certificate Expiration

Organizations often create compliance issues by using expired certificates. Certificates have expiration dates so that attackers cannot use an otherwise legitimate certificate forever if the corresponding private key is compromised. Allowing the acceptance of expired certificates renders TLS protection ineffective over the long term.

Organizations should always renew certificates prior to expiration and configure systems to reject expired certificates.

Certificate Domains

Organizations also create compliance issues by failing to match the domain of an endpoint to the domain listed in the certificate. The domain information in the certificate is used to validate that the endpoint is within the expected domain. Allowing certificates with mismatched domains can also render TLS protection ineffective.

A domain mismatch often occurs when connecting to an IP address rather than a domain name or when an endpoint has more than one (1) domain name assigned (e.g., a public-facing name for external users and a private-facing name for internal users). A single host can have certificates that cover multiple domain names and IP addresses to allow many types of connections without domain mismatches.

Domains in certificates can be as broad or narrow as necessary, such as “*.trustedsec.com” or “www.trustedsec.com”. Best practice is to define the domain in a certificate as narrowly as possible to limit the potential damage if the private key associated with the certificate is compromised. A compromised private key can be used to impersonate any system with the domain of the associated certificate.

Organizations should generate certificates with narrow domain settings for all appropriate domain names and IP addresses and configure systems to reject certificates with mismatched domain names.

Key Management

What is Key Management?

The best encryption in the world can be rendered useless if the encryption and decryption keys are improperly generated or compromised due to poor handling. Key management is the process of securely generating, storing, and distributing cryptographic keys so that encryption remains effective.

Specialized key management software or HSMs can be used to streamline the key management process by securely implementing and automating the core cryptographic functions that are referenced throughout this section.

Management of cryptographic keys is a complex topic that could warrant its own blog post. A very deep dive into this topic can be found in NIST SP 800-57.

Key Generation

In addition to choosing strong algorithms and key lengths, organizations need to consider the source of data used to generate keys. Randomness is required to generate strong keys that an attacker can’t predict. The importance of random number generators cannot be overstated. Encryption backdoors have been created via supposed random number generators that create predictable keys, and these backdoors have been exploited by unknown actors.

Truly random data is extremely difficult to generate in software due to the deterministic nature of off-the-shelf computing hardware, so complex algorithms have been created to generate pseudorandom data that simulates randomness as close as possible.

Keys that are used in high-security applications, especially private CA master keys, should be generated using cryptographically secure pseudorandom number generators (CPRNGs) or (preferably) hardware random number generators. Encryption software may contain CPRNG capabilities. Key management software and HSMs will almost certainly contain CPRNG capabilities and could also support hardware key generation.

Key Storage

The private keys associated with a TLS certificate must be protected, because an attacker that compromises a private key can decrypt communications using the corresponding certificate or impersonate the any system within the domain covered by the certificate.

Private keys associated with private CAs are critical, as they can be used to generate fraudulent certificates that appear to be legitimately signed by the organization’s CA. This would allow an attacker to impersonate any system within the organization, and potentially other systems outside the organization. Private keys associated with public CAs are even more critical, as their compromise has Internet-wide impersonation consequences due to the ability to forge certificates for any domain, as occurred during the 2011 DigiNotar breach.

Protecting private keys is difficult to accomplish, however, because both TLS endpoints and CAs require private key access in order to perform their functions. This is another area where HSMs may help for high-security applications, as it is much more difficult for an attacker to extract a key from an HSM than it is to copy a key from a file.

Other methods to secure private keys include splitting the private key into parts that are stored separately and/or encrypting the private key itself with yet another key (which should be stored separately from and be at least as strong as the key being protected). These techniques are usually utilized with keys that are used infrequently, such as master keys for a private CA, and difficult to use on keys that are constantly active (e.g., the live keys on a service that uses TLS).

Storing the private key in as few locations as possible and allowing access to the fewest number of people possible is also recommended, as it reduces the number of opportunities for a key to be compromised.

Key Distribution

As with key storage, key distribution is critical because it might lead to unintentional key storage. For example, emailing a private key results in the key being stored for an indeterminate amount of time on the email server as well as any devices that receive the email. Similarly, keys distributed via removable media may be retrieved from the media—even after deletion—unless the media is properly sanitized.

Detailed procedures should be created to control how private keys are distributed to maintain the security of the keys during and after the process. This could include encrypting the private keys while they are being distributed.

Key Inventory

Organizations should maintain an inventory with the details of the various cryptographic keys and certificates they have generated. This allows organizations to retire keys and certificates when weaknesses are found in the algorithms used or if a key is compromised.

Similarly, organizations should maintain an inventory of the TLS software, key management software, and HSMs in use so that relevant vulnerabilities can be quickly identified and patched.

Software Packages

The TLS protocol, and the cryptographic algorithms it uses, are all defined in open public standards that can be implemented by anyone. Using the TLS in the real world requires the protocol and algorithm standards to be implemented in software.

As with all software, bugs are a possibility that could create vulnerabilities, either via incorrect implementation of the protocol or algorithms or through flaws that allow direct attacks on the software itself. Organizations must remain vigilant for vendor security updates and patches that affect their TLS encryption software and libraries.

Effectively managing TLS vulnerabilities requires understanding what TLS implementation software is using. Common scenarios include:

• TLS libraries embedded in operating systems such as the Security Support Provider Interface (SSPI) API in Windows and the Network Framework used across Apple’s various operating systems
• Open source libraries, such as GnuTLS, LibreSSL, and OpenSSL, which are often used by Linux, BSD, and other open source, Unix-style software
• Commercial TLS libraries
• Software packages may contain their own implementation of TLS rather than relying on well-known libraries

FIPS 140 Validation

What is commonly referred to as “FIPS Encryption” is actually a reference to software or hardware validated using FIPS 140, one (1) of many other Federal Information Processing Standards that cover a variety of topics. FIPS 140 establishes security requirements for cryptographic software and hardware. Vendors that have created cryptographic software and/or hardware can submit their products to independent labs for verification that the products meet all of the FIPS 140 requirements.

Note that it is the software/hardware implementation of the encryption algorithms that is validated, not the underlying cryptographic algorithms or the TLS protocol itself, as these are already standardized. FIPS 140 is essentially ensuring that the software/hardware has correctly implemented the standardized protocols and algorithms in a way that is resistant to attacks.

U.S. Federal agencies are required to use cryptographic products that have been validated against FIPS 140, but private organizations are generally free to use non-validated products. Organizations handling information on behalf of the U.S. government may be contractually required to use FIPS 140 validated encryption to protect certain information handled on behalf of the government. Private organizations that voluntarily choose to use FIPS 140 validated products can be assured that the software has been independently reviewed.

While an organization could be using software or hardware that is FIPS 140 validated, there may still be some configuration settings that must be applied to maintain compliance. This might be as simple as enabling a “FIPS 140 mode” or may require disabling specific features or algorithms that did not go through the FIPS 140 validation process. Organizations with FIPS 140 requirements should check vendor documentation for details.

FIPS 140-2 is currently being replaced with FIPS 140-3. FIPS 140-2 validations will be retired on September 21, 2026, but organizations subject to federal requirements can continue to purchase them unless told otherwise. Organizations with FIPS 140 validation requirements should consider selecting FIPS 140-3 validated solutions for future purchases when feasible.

A list of FIPS 140 validated encryption modules is available from NIST.