refcodes-forwardsecrecy: High encryption throughput

Siegfried Steiner

19 min read Updated on September 04, 2026 April 02, 2015

Categories

• refcodes

Tags

• REFCODES
• RefcodesForwardsecrecy
• ForwardSecrecy
• Cryptography
• BareMetal
• Serverless
• CloudComputing
• Java
• Maven

README

The REFCODES.ORG codes represent a group of artifacts consolidating parts of my work in the past years. Several topics are covered which I consider useful for you, programmers, developers and software engineers.

What is this repository for?

This artifact provides high-throughput authenticated symmetric encryption and protects rotating data-encryption keys with an asymmetric encryption approach. It seamlessly works together with the refcodes-logger toolkit, enabling you to log vast amounts of information in an effective and protected manner.

Despite its established name, the artifact implements key rotation with envelope encryption, not forward secrecy in the cryptographic protocol sense. Historical data-encryption keys remain recoverable with the corresponding long-term RSA private key. Compromise of that private key together with access to persisted cipher versions can therefore expose historical data. Applications requiring protocol-level forward secrecy must additionally use an ephemeral key-agreement protocol and erase session keys after use.

How do I get set up?

To get up and running, include the following dependency (without the three dots “…”) in your pom.xml:

<dependencies>
	...
	<dependency>
		<groupId>org.refcodes</groupId>
		<artifactId>refcodes-forwardsecrecy</artifactId>
		<version>${org.refcodes.version}</version>
	</dependency>
	...
</dependencies>

The artifact is available from Maven Central. The source code is hosted at Bitbucket, and the API documentation is available at javadoc.io.

Introduction

The refcodes-forwardsecrecy artifact provides a framework for easy encryption and decryption functionality with a high throughput.

The design strictly uses separation of concerns regarding encryption and decryption functionality. This framework is designed to perform encryption and decryption on huge amounts of data. Authenticated AES-GCM encryption uses the random 256-bit data-encryption key directly, while RSA-OAEP protects the rotating data-encryption keys. RSA-PSS signatures bind each request challenge to its namespace. RSA keys shorter than 2048 bits are rejected.

Rotating data-encryption keys limits the amount of data encrypted with one key. It must not be confused with Forward Secrecy, because compromise of a long-term RSA private key can still expose persisted historical keys.

Fancy output of a refcodes-forwardsecrecy driven service

Various REFCODES.ORG artifacts make use of the refcodes-forwardsecrecy framework, for example the refcodes-tabular-alt-forwardsecrecy artifact.

How do I get started?

To get started I list the requirements I had in mind when I was designing the refcodes-forwardsecrecy artifact:

• We have to store big-data in a secure though fast manner
• We also have to retrieve big-data in a secure though fast manner
• Decryption parts and encryption parts are to be strictly separated from each other to enable operation of physically separated encryption and decryption systems (servers)
• Encrypting parts are to generate their symmetric ciphers (for encryption and decryption) by themselves, volatile, in-memory and exclusively for their own (in-memory) use only, encryption must not persist symmetric ciphers
• Decryption parts are to use the (decrypted) symmetric ciphers in-memory, volatile and exclusively for their own (in-memory) use only. Conclusion: A pure asymmetric encryption approach is far too slow when decrypting the data. Symmetric keys must not be accessible (on a physical storage) on the encrypting systems as plain text

Summary

This artifact provides a toolkit for easy encryption and decryption functionality with a high throughput.

The design strictly uses separation of concerns regarding encryption and decryption functionality. This framework is designed to perform encryption and decryption on huge amounts of data. Therefore an authenticated symmetric cryptography approach is used for data encryption and combined with asymmetric cryptography for protecting data-encryption keys.

Having the requirements above in mind, the design approach behind the refcodes-forwardsecrecy toolkit may be summarized as follows:

• Each cipher belongs to a unique cipher UID; which is generated alongside with the cipher; though completely independently from the cipher. The cipher UID can be public (is considered to be public) without causing any security issues
• The tuple consisting of cipher and cipher UID is managed as a so-called cipher version
• The encryption parts use the public key of an asymmetric encryption approach to store the generated cipher versions in a data store (file-system, repository) to be used by decryption parts owning the according private key
• When encrypting data, the encryption parts encrypt the data with the cipher and prefix the cipher’s cipher UID to that data
• The cipher versions are stored in a data store (file-system, repository) for usage by an authorized decrypting parts owning the according private key
• The decryption parts can read the encrypted cipher versions from the data store (file-system, repository), decrypting it with their private key
• The decrypted cipher versions are only held in-memory by the decrypting parts
• The decryption parts retrieve the data to be decrypted and extract the cipher UID from that data which has been prefixed before by the encryption parts to that data
• Having the cipher UID the decryption parts can look up the according cipher and decrypt the data
• As many cipher versions may get generated and persisted, housekeeping should be applied to outdated cipher versions
• The only use case when decryption and encryption is to be performed on the same system is when updating encrypted data which has been encrypted with an outdated cipher (a cipher UID may contain the date of cipher creation or a cipher version may contain an additional valid date attribute)
• In such cases, only the outdated cipher versions must be accessible by such an update (encryption and decryption) part as it generates its own valid encryption cipher
• If cipher versions are unexpectedly disclosed, they become obsolete as soon as the corresponding encrypted data has been updated with a newly generated cipher version
• That fresh cipher version is stored using the public key of an asymmetric encryption algorithm as of step 3
• A shortcoming is on how to securely provide such an update (encryption and decryption) part just with the outdated cipher versions; whilst preventing access to the valid cipher versions (technically, as seen below when looking at the design, this is no problem: Using a separate data store for outdated ciphers secured by a separate key-pair just requires the outdated ciphers to be moved to the alternate data store)
• As so, the private key for decrypting the outdated cipher versions must be different to the one which is used for decrypting the valid cipher versions…

Discussion

This mechanism has the following obvious shortcomings: Having control over a decryption part makes all the data readable. Though when using asymmetric encryption, having access to a system with the private key has the same effect. As of performance reasons, the approach of managing symmetric ciphers separate from the data chunks to be processed and processing those symmetric ciphers with an asymmetric cryptography approach has been chosen:

The key-rotation infrastructure discussed here handles persistent data. Using a pure asymmetric encryption approach would not reduce the security hot spot. Furthermore, asymmetric cryptography is expensive for many small chunks of data, even when combined with symmetric cryptography:

Processing small chunks of data either directly with an asymmetric cryptography approach or combined with a symmetric cryptography approach with post processing of the symmetric cipher for that chunk with an asymmetric cryptography approach costs execution time for each data chunk processed respectively; which is to a great extend slower than processing using a symmetric approach. The assumption behind it might be that asymmetrically processing symmetric ciphers is less cost intensive than asymmetrically processing the data itself. With small chunks of data this assumption is not true. The vastly reduce execution time this framework processes the symmetric ciphers for the data chunks independent from the data chunks; separately handled by an asymmetric cryptography approach; also independent from the data chunks; preventing the costly execution time to process each data chunk with an asymmetric cryptography approach.

Setup

A usual setup ensures that the symmetric ciphers are generated only by the encryption part of the framework and are stored there only in volatile memory. Those symmetric ciphers are published in encrypted form to the decryption part of the framework using an asymmetric cryptography approach. The encryption part uses the public key of the decryption part to protect the symmetric ciphers. The decryption part may also support an asymmetric cryptography approach for providing the symmetric ciphers to multiple decryption services, each equipped with its own private key.

There are two common used types of refcodes-forwardsecrecy setup for your applications.

1. Basic setup: The basic setup encrypts the symmetric ciphers of the encryption part with the public key of the decryption part. Any decrypting participant located in the decryption part must have access to the same private key in order to make use of the encrypted ciphers.

2. Usual setup: The normal setup is similar to the basic setup with the addition that for using the ciphers on the decryption part, additional key pairs are required for each decryption participant located in the decryption part; each decrypting participant uses its own private key, a single private key must not be known by all decrypting participants.

Design

The framework design provides three layers each for the encryption part respectively the decryption part:

Provider: A provider can be an encryption provider or a decryption provider and is being used by the business logic to encrypt or decrypt texts. The business logic must not know anything about ciphers or cipher versions. For encryption or decryption a symmetric encryption approach is used

Service: The service can be an encryption service or a decryption service managing ciphers or cipher versions exclusivity in-memory for the symmetric encryption approach. For exchanging cipher versions with a data store (file-system, repository) between the encryption and the decryption systems (encryption servers and decryption servers), an asymmetric encryption approach may be used.

Server: A server can be an encryption server or a decryption server; responsible for securely storing (encryption server) or securely retrieving (decryption server) cipher versions. Hence a server is required for exchanging cipher versions between the data store (file-system, repository) and the according services; here an asymmetric encryption approach may be used for encrypting cipher versions (encryption server) and decrypting cipher versions (decryption server).

Please take a look at the drawing below for an overview on the key- and cipher-exchange process: Red regards the asymmetric key life-cycle, yellow regards the high-volume data to be encrypted and green regards the decryption process.

Overview of the refcodes-forwardsecrecy artifact’s internal functioning

For a detailed description see chapter “10. … AND OUT OF THE CLOUD: SECURITY” in my paper on Big data processing the lean way - a case study.

The complete asymmetric cipher version exchange can be handled inside the service layer so that the servers need not know anything about private or public keys, as in the basic setup. This has the shortcoming that all decryption services must have access to the same private key. It has the advantage that the server does not reveal any ciphers. A decryption wrapper can instead wrap the decryption server and add separate key-pair support for exchanging cipher versions with the decryption services, as in the normal setup.

As we support encryption and decryption, the triple of provider, service and server exists for the encryption part as well as for the decryption part.

At a closer look, we see the following types participating in the refcodes-forwardsecrecy tool-box:

Cipher version

A cipher version represents a cipher (used for symmetric encryption respectively decryption) related to a cypher’s version (UID).

Implementation specific specializations might add attributes to determine whether a cipher version is outdated and must be replaced by a new cipher version or other attributes helpful for housekeeping and management of cipher versions. The cipher versions are (indirectly) bound to a namespace.

Namespace

A namespace separates the various sets of cipher versions semantically from each other.

An application might be bound to a namespace or a tenant of an application might be bound to a namespace (depends on the requirements). In practice this is achieved by bounding encryption providers, which are the hot spots for the business logic, to a namespace; for the business logic the namespace is transparent (it is a matter of configuration).

Encryption provider

The encryption provider is bound to a namespace and is being used by the business logic to encrypt data.

It is merely responsible for retrieving a currently valid cipher for encrypting data. The encryption provider does not expose the cipher though it might store it in clear text in-memory only (advanced implementations might encrypt the in-memory cipher). The (internally) retrieved cipher is requested from and provided by an encryption service (on the same machine) which takes care of providing (and creating) a cipher with a cipher UID as well as publishing the according cipher version (via the encryption server to the decryption server). As the encryption provider does not persist any data, an in-memory cipher version will only be used as long as the encryption provider is up-and-running. In case of a restart, a new cipher for encryption is requested from the encryption service. Also the encryption provider can be forced to (create and to) use a next valid cipher on demand.

Encryption service

An encryption service is bound to a single namespace and creates unique cipher versions for that namespace.

The encryption service may make use of an encryption server persisting cipher versions per namespace. It could actually generate a dedicated cipher just once, so any unauthorized system having access to the ciphers gets a different cipher not used by any of the authorized participant. Never two participants will encrypt with the same cipher (taken the probability that two participants generate the same cipher is very low and nearly never to happen; in case it happens there is still no security risk). The key advantage is that if an intruder can also retrieve ciphers, those ciphers being retrieved are never used by other systems for encryption as a cipher version is bound to the requester.

To later determine which cipher to use when decrypting data, each cipher has a cipher UID assigned to it (a cipher UID and cipher make up a cipher version). Encrypted data is prefixed with this cipher UID, making it easy to select the cipher responsible for decryption. The cipher UID is public and generated independently of the cipher itself. Access to the cipher UID does not reveal the cipher.

There is not even a relation between cipher and cipher UID in terms if hash code. This means using brute force approaches with rainbow tables or whatsoever to reconstruct the cipher from the cipher UID is to fail.

Depending on the implementation, the encryption service makes use of a public key of an asymmetric encryption approach for encrypting the cipher versions; to be persisted by the encryption server.

Protecting only the cipher is sufficient; the cipher UID can be stored in plain text. Knowing a cipher UID does not reveal the corresponding cipher, which is generated independently.

Encryption server

The encryption server takes cipher versions generated by an encryption service for a provided namespace.

A cipher version provided to the encryption server (by an encryption service) is persisted so that the decryption server can access this cipher version. When persisting, a cipher version’s cipher UID is verified whether there is already a cipher version with the same cipher UID already persisted and where necessary rejected (such collisions can be avoided with good cipher UIDs).

Regarding the implementation of the encryption server, securely persisting can be done with the public key of an asymmetric encryption approach so that only the decryption service can get the plain text ciphers from the cipher versions. To avoid transmitting plain text cipher versions from the encryption service to the encryption server, the encryption service should already encrypt the cipher version with the according public key so that the encryption server always receives encrypted cipher versions.

The key-rotation infrastructure supports encryption servers which only need to persist and retrieve cipher versions. Encryption and decryption can be done in the corresponding service layers. For example, the encryption service uses a public key to protect the key material of a cipher version and passes it to the encryption server without requiring additional server-side encryption. A decryption service requests the protected cipher versions from the decryption server and recovers the key material with the corresponding private key.

By replacing the implementation of the encryption server, the way cipher versions are persisted can be changed easily.

Decryption server

The decryption server contains cipher versions assigned to a namespace.

Depending on the implementation, the decryption server might as well contain a number of public keys (for an asymmetric encryption approach) also assigned to the individual namespaces identifying the owners of the private keys with which it is secure to communicate.

The decryption server might access persisted cipher versions. Depending on the implementation, the cipher versions to be persisted must be encrypted with the decryption server’s public key. An encryption service having this public key then can do secure persisting.

Requesting the cipher versions from the decryption server might then be done by authenticating that the requester is entitled to request the cipher versions by verifying the signature of the requester’s namespace and challenge with the public keys held by the decryption server and by encrypting the cipher versions with that according public key. The decryption server itself might use an asymmetric encryption approach to decrypt persisted cipher versions persisted by the encryption server (and being encrypted by the encryption service).

A decryption server’s wrapper could be hooked on top the decryption server which uses the private key used for encrypting the ciphers by the encryption service to decrypt the ciphers and encrypts the ciphers again with a public key from a key pair of an according decryption service. The decryption service authenticates itself with a signature over its namespace and a fresh challenge generated from its according private key. The decryption server can validate the signature and use the trusted public key for encryption. By replacing the implementation of the decryption server, the way cipher versions are persisted can be changed easily.

Decryption service

A decryption service is bound to a single namespace and provides cipher versions required for decrypting text by the decryption provider

The decryption service may make use of a decryption server managing the cipher versions per namespace.

Depending on the implementation, the decryption service has a private key for an asymmetric encryption approach whose public counterpart is used by the encryption service. This private key is used to recover the ciphers from the retrieved cipher versions.

A decryption server’s wrapper may be hooked on top of the decryption server containing public keys known as being trusted and the private key for decrypting ciphers being encrypted by the encryption service. When cipher versions are being requested by a decryption service from the wrapped decryption server, the decryption service authorizes itself by signing the namespace and a fresh challenge with a signature passed to the decryption server. In case the signature is verified by the decryption server with one of its trusted public keys, then the public key in question is used by the decryption server for encrypting the cipher versions being transmitted to the decryption service.

Decryption provider

The decryption provider is bound to a namespace and is being used by the business logic to decrypt data.

The decryption provider provides decrypting functionality as encrypted data must be decrypted again by another service or system. This system must now be able to retrieve all known ciphers versions (by a decryption service) for determining the correct cipher for decrypting encrypted text (as encrypted text is prefixed by the cipher UID identifying the cipher to use for decryption).

Cipher version factory

Cipher version factories manufacture cipher versionsand make the creation of those replaceable.

As requirements might arise to use cipher versions with additional attributes or functionality; the cipher version factory can be replaced with a custom implementation instantiating cipher version (sub-)types with the additional required attributes or functionality. Additional attributes might be a validity date useful for housekeeping or management purposes.

In case you provide your custom cipher version factory implementation, make sure the cipher version (sub-)type you return fits with the cipher version (sub-)type of your custom cipher version generator. A good approach is to make your custom cipher version generator make use your custom cipher version factory (see the default implementations of the cipher version generator and cipher version factory).

Cipher version generator

Depending on the security demands and performance issues; the generator generating cipher versions can be replaced with a custom cipher version generator using its own approach generating ciphers and cipher UIDs (cipher versions). In case you provide your custom cipher version generator implementation, make sure the cipher version (sub-)type you return fits with the cipher version (sub-)type of your custom cipher version factory. A good approach is to make your custom cipher version generator make use your custom cipher version factory (see the default implementations of the cipher version generator and cipher version factory).

Snippets of interest

Below find some code snippets which demonstrate the various aspects of using the refcodes-forwardsecrecy artifact (and , if applicable, its offsprings). See also the example source codes of this artifact for further information on the usage of this artifact.

A first setup

First we prepare our local setup: Therefore for the decryption part we use the InMemoryDecryptionServer type:

InMemoryDecryptionServer theDecryptionServer = new InMemoryDecryptionServer();
DecryptionService theDecryptionService = new PublicKeyDecryptionService( "myNameSpace", "/path/to/private/key", theDecryptionServer, 500 );
DecryptionProvider theDecryptionProvider = new JceDecryptionProvider( theDecryptionService );

As we have a local setup here, we use the InMemoryDecryptionServer instance to setup the encryption part’s InMemoryEncryptionServer instance:

InMemoryEncryptionServer theEncryptionServer = new InMemoryEncryptionServer( theDecryptionServer );
EncryptionService theEncryptionService = new PublicKeyEncryptionService( "myNameSpace", "/path/to/public/key", theEncryptionServer );
EncryptionProvider theEncryptionProvider = new JceEncryptionProvider( theEncryptionService );

In a distributed environment, we would use the FileSystemEncryptionServer type for the encrypting part and the FileSystemDecryptionServer type for the decrypting part (or according custom implementation of the DecryptionServer type alongside the counterpart EncryptionServer type)!

Mass key rotation

Here again we use a local setup; in a distributed environment we would split the encryption part from the decryption part into separate services:

for ( ... ) {
	theEncryptionProvider.nextCipherVersion();
	String theEncryptedText = theEncryptionProvider.toEncrypted( theText );
	...
	String theDecryptedText = theDecryptionProvider.toDecrypted( theEncryptedText );
	...
}

In a real world setup, encryption logic (line 3) and decryption logic (line 5) would reside on different services!

Resources

The whole encryption process is illustrated in chapter “10. … AND OUT OF THE CLOUD: SECURITY” in my paper on Big data processing the lean way - a case study (Slides) at the Coding Serbia 2014 conference.

Contribution guidelines

• Report issues
• Finding bugs
• Helping fix bugs
• Making code and documentation better
• Enhancing the code

Who do I talk to?

• Siegfried Steiner (steiner@refcodes.org)

Licensing Philosophy

This project follows a dual-licensing model designed to balance openness, pragmatism and fair attribution.

You may choose between the LGPL v3.0 or later and the Apache License v2.0 when using this software.

The intention behind this model is simple:

• Enable use in both open-source and proprietary projects
• Keep the codebase approachable and reusable
• Ensure that improvements to the library itself remain available to the community
• Preserve clear attribution to the original author and the ecosystem

Under the LGPL v3.0+, you are free to use this library in any application. If you modify the library itself, those modifications must be made available under the same license and must retain proper attribution.

Alternatively, the Apache License v2.0 allows broad use, modification and distribution, including commercial usage, provided that copyright notices and the accompanying NOTICE file are preserved.

This dual-licensing approach intentionally avoids artificial barriers while discouraging closed, uncredited forks of the core library. Contributions, improvements and refinements are encouraged to flow back into the project, benefiting both the community and downstream users.

For licensing questions, alternative licensing arrangements or commercial inquiries, please contact the copyright holder.

Share Tweet LinkedIn Reddit