AWS KMS: MetaStore and MostRecentProvider Explained

I couldn't find suitable documentation on this and was hard for me when I had to implement a solution using MetaStore (MS) and MostRecentProvider (MRP). Here is the blog with my understanding. Hope this will be helpful.

MS and MRP, together will help improve performance of KMS. This is particularly useful if you have a usecase where lots of calls are made to DynamoDB for data retrieval and there are stringent NFRs to adhere with.

Before going ahead with MS and MRP, let me give a overview of how encryption/decryption happens using DirectKMSMaterialProvider. This is one of the EncryptionMaterialsProvider implementation which is given in one of AWS blogs: 

Lets say, I store customer data in DynamoDB and each row has 20 fields. Out of 20 fields, I'm encrypting 10 as they are PII (Personal Identifiable Information) data. How KMS works is

With DirectKMSMaterialProvider, to encrypt 1 record, there will be 10 calls to KMS as KMS will encrypt each field individually. This proves to be very costly operation as think of a scenario where we have thousands of calls made per second and there are hundreds or thousands of records to be retrieved in a single call within say 10 ms. Every call to dynamoDB is having an additional latency here which is encryption/decryption for data insertion/retrieval respectively. This defeats the purpose of using dynamoDB for performance reasons.

AWS has multiple EncryptoinMaterialsProvider implementations. The whole issue here is due to the way Envelope encryption happens in DirectKMSMaterialProvider of KMS. Before proceeding further, if you wish to hover on basic concepts of KMS, visit my blog here.

MetaStore:

Alright.. treat MS as some meta data. This is a simple collection of EncryptionMaterialProviders backed by an encrypted DynamoDB table. MS can be used to build key hierarchies or meta providers. Cool.. not good enough explanation?? May be because I just copied the above lines from MS documentation. Lets go a step further to understand what it is.

MS is used to store encrypted dataKeys. Simple. A collection of encrypted dataKeys. Where we store? We store in a DynamoDB table. If we see the constructor of MS:

public MetaStore(final AmazonDynamoDB ddb, final String tableName,
            final DynamoDBEncryptor encryptor) 

This takes AmazonDynamoDB that has connectionDetails, tableName which is the dynamoDB table that will hold encrypted dataKeys and DynamoDBEncryptor which is the EncryptionMaterialsProvider.

creating an instance of MS will initialize and create the dynamoDB table with the given tableName. This class has other APIs to manage MS.
MS will always be used in conjunction with MRP. I've used spring to create MS class. Will provide a snippet after explaining MRP that will help understand how these two classes work together.

MostRecentProvider:

As the name suggests, this class is used to encrypt the data with the most recent version of the key materials from MS. It has the intelligence to decrypt with whichever version it used to encrypt. Every dataKey in MS will have a version attached to it. MRP will always work using latest version. Will explain this in detail shortly.

For now, lets look at how the constructor of MetaRecentProvider looks like.

public MostRecentProvider(final ProviderStore keystore, final String materialName, final long ttlInMillis)

So, first argument is the MS we created, and then the materialName and ttlInMillis arguments. Let me give few words about last two parameters.

MaterialName: Collection of dataKeys in MS can be grouped under a materialName and can be used to protect certain table. Using this, we can have separate dataKeys for each table if needed. Or, have multiple levels of MS to protect data each with a materialName.

ttlInMillis: We know that data stored in dynamoDB table is encrypted dataKeys. Encrypted dataKeys are not useful for encrypt/decrypt unless they are decrypted. So, dataKeys are decrypted using CMK and are then cached in memory to avoid multiple calls to AWS KMS. How often the cache will be refreshed is configurable and ttlInMillis gives that value. Simple.

Now, we know how MS and MRP are linked. Let me explain bit more about some APIs to manage MS

For example: Lets say I implemented MS with a dynamoDB table named KeyStoreTable. There is a API in MS named newProvider. Application will call this API whenever a new dataKey is to be created. Note that the dataKey will be created under the given materialName.

public EncryptionMaterialsProvider newProvider(final String materialName)  

Code snipped showing how the MS and MRP are stitched using Spring Config.

@Bean                                                                                                    
public AWSCredentialsProvider awsCredentialsProvider(){
    return new InstanceProfileCredentialsProvider();
}

@Bean                                                                                                    
public AmazonDynamoDBClient amazonDynamoDBClient() {
    AmazonDynamoDBClient amazonDynamoDBClient = new AmazonDynamoDBClient(awsCredentialsProvider());
    return amazonDynamoDBClient;
}

@Bean
public AWSKMS awsKms(){
    AWSKMS awsKMS = new AWSKMSClient(awsCredentialsProvider());
    awsKMS.setEndpoint(kmsEndpoint);
    return awsKMS;
}

@Bean
public EncryptionMaterialsProvider directKMSMaterialsProvider(){
    return new DirectKmsMaterialProvider(awsKms(), kmsCustomerMasterKeyAlias);
}

@Bean
public DynamoDBEncryptor dynamoDBEncryptor(AttributeEncryptor attributeEncryptor){
    return attributeEncryptor.getEncryptor();
}

@Bean
public AttributeEncryptor attributeEncryptor(EncryptionMaterialsProvider encryptionMaterialsProvider){
    return new AttributeEncryptor(encryptionMaterialsProvider);
}

@Bean
public MetaStore metaStore(){
    //Attribute encryptor with DirectKMSMaterialsProvider
    AttributeEncryptor attributeEncryptor = new AttributeEncryptor(directKMSMaterialsProvider());
    return new MetaStore(amazonDynamoDBClient(), keyStoreTable, dynamoDBEncryptor(attributeEncryptor));
}

@Bean
public EncryptionMaterialsProvider mostRecentProvider(MetaStore providerStore, String materialName, String ttlInMills){
    return new MostRecentProvider(providerStore, materialName, Long.parseLong(ttlInMills));
}

@Bean
public DynamoDB dynamoDB() {
    return new DynamoDB(amazonDynamoDBClient());
}

@Bean
public  DynamoDBMapper dataMapper() {

    DynamoDBMapperConfig mapperConfig = cacheManagerConfig.prepareMapperConfigWithTableNameResolver(CACHE_MANAGER_PROFILE);
    EncryptionMaterialsProvider mostRecentProvider = mostRecentProvider(metaStore(), materialName, ttlInSecs);

    return new DynamoDBMapper(amazonDynamoDBClient(), mapperConfig, attributeEncryptor(mostRecentProvider));
}

Snippet is self explanatory. Hope this is helpful!! If any queries, do get back to me in comments or to my mailId: mail2vinay.bs@gmail.com

AWS KMS: Explained

AWS Key Management Service (AWS KMS) is a managed service used to create and control encryption keys used to encrypt data. AWS KMS can be integrated with other AWS services like EBS, S3, DynamoDB etc.

Key Concepts of AWS KMS at a glance

Customer Master Keys

Primary resource of AWS KMS is Customer Master Keys (CMK). CMKs are either customer-managed or AWS-Managed. CMKs can be used either to protect upto 4KB data directly. Best way is to use CMK to protect dataKeys, which in-turn are used to protect actual data. CMKs never leave AWS KMS unencrypted.

CMK can be rotated once a year if its customer-managed.

Data Keys

Data keys are used to protect actual data. In envelope encryption, data Keys will be encrypted by CMK. And the encrypted data keys are used to encrypt/decrypt actual data. AWS KMS offers APIs to create data keys. While AWS KMS APIs can be used to generate, encrypt and decrypt data keys, AWS KMS will not store, manage or track your data keys. It has to be done in the application.

Envelope Encryption

AWS KMS uses envelope encryption to protect data. Envelope encryption is a practice of encrypting plain text data with a unique data key, and then data key is encrypted using key encryption key KEK). There can be multiple levels of KEKs. That is, we can choose to encrypt KEK with another KEK. But ultimately, the KEK has to be encrypted by a master key. Master key is an unencrypted(plaintext) key with which you can decrypt one or more keys.

In KMS, master key is called Customer Master Key (CMK). 

Envelope Encryption offers following advantages.

1. It protects data keys.
2. Option to encrypt same data using multiple master keys.

Following image provides an overview of how envelope encryption works in AWS KMS.

Encryption Context

All AWS KMS cryptographic operations (encryption/decryption) accepts an optional set of key-value pairs that can contain additional contextual information about data. This set of key-value pairs is called encryption context. Encryption context used for encryption should be same for decryption of the data for the decryption to succeed. Encryption context is not secret. It can be logged and can be used for auditing and controlling access to AWS KMS API operation.


Refer to this link to understand how Envelope Encryption Works.

Source: http://docs.aws.amazon.com/kms/latest/developerguide/overview.html

 http://docs.aws.amazon.com/kms/latest/developerguide/concepts.html 

http://docs.aws.amazon.com/kms/latest/developerguide/workflow.html

Writing JAX-WS SoapHandler

SoapHandler(SMH) is a interceptor for JAXWS webservies. It will intercept in bound and out bound messages. It is a couterpart of GenericHandler for the JAXRPC webservices. There are two types of handlers for JAXWS webservices. One is LogicalHandler and other is SoapHandler. LogicalHandler is used to write some logic on in bound or out bound messages. Lets focus now on writing SoapHandler. Lets try to do it in simple steps. 1. Write a handler by implementing interface SoapHandler. SoapHandler is a Generic interface, which takes in any class which "IS A" SOAPMessageContext. If you want your custom features in SOAPMessageContext, then implement you own. Else, use SOAPMessageContext.

 
public class MySoapHandler implements SOAPHandler{

    @Override
    public boolean handleMessage(SOAPMessageContext context) {
 // TODO Auto-generated method stub
 return false;
    }

    @Override
    public boolean handleFault(SOAPMessageContext context) {
 // TODO Auto-generated method stub
 return false;
    }

    @Override
    public void close(MessageContext context) {
 // TODO Auto-generated method stub
 
    }

    @Override
    public Set getHeaders() {
 // TODO Auto-generated method stub
 return null;
    }

}

2. Once you implement the class, you will have to implement a number of methods. Prominent method being 2.1 handleMessage(SOAPMessageContext context) - This is the method used to get the in bound or out bound messages and process it. 2.2 handleFault(SOAPMessageContext context) - Whenever there is a Fault, this message is called and you can write logic to handle the fault. 3. I will log the inbound and out bound messages in handleMessgae and let the flow continue.

 
 
     @Override
    public boolean handleMessage(SOAPMessageContext context) {
 Boolean isOutbound = (Boolean) context.get(MessageContext.MESSAGE_OUTBOUND_PROPERTY);
 if (!isOutbound) {
     System.out.println("In Bound Message ... ");
 }
 else {
     System.out.println("Out Bound Message ... ");
 }

 try {
     context.getMessage().writeTo(System.out);
 }
 catch (SOAPException e) {
     e.printStackTrace();
 }
 catch (IOException e) {
     e.printStackTrace();
 }
 return true;
    }

4. Write a handlerConfig.xml file which will contain the fully qualified class name of the handler we just created. As the name of the file indicates, it is a chain of handlers.

    <handler-chains xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xsi:schemalocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/javaee_web_services_metadata_handler_2_0.xsd">  
      <handler-chain>  
           <handler>  
                <handler-name>SoapRequestValidator</handler-name>  
                <handler-class>com.example.jws.handler.MySoapHandler</handler-class>  
           </handler>  
 <!-- Can have multiple chains ... -->  
      </handler-chain>  
 </handler-chains>  

5. In the JAXWs webservice, include the handlerChain using the annotation:
@HandlerChain(file = "handlerConfig.xml")
Place the handlerConfig.xml in the same package as the webservice endpoint implementation file.
6. Finally, very important point is to include the handlerConfig.xml in the war which you build. If you don't include the handlerConfig file, then you don't get any error. But the handler will never be called. So, do remember to include the handlerConfig.xml in the war file.
Expect an example project on this shortly in my github account. Simple isn't it :)

Deploy to Tomcat 7 using Maven

Steps to deploy war file in Tomcat 7 through Maven:

1. tomcat-users.xml: This file is present at conf directory under tomcat folder (apache-tomcat-7.x/conf)

   Add role "manager-script". This is required in tomcat 7 if deployment is done through a script, like maven.
   Create a user by any name and give a password of your choice and assign this user with the "manage-script" role.
 
   Eg:

      <tomcat-users>

<role rolename="manager-script"/>
        <user username="admin" password="admin"  roles="manager-script"/>

      </tomcat-users>

For deploying to tomcat through eclipse, above roles will suffice. However, to manage through tomcat manager UI, additional roles needs to be added the use.

Note that for Tomcat 7 onwards, the roles required to use the manager application were changed from the single manager role to the following four roles. You will need to assign the role(s) required for the functionality you wish to access.

manager-gui - allows access to the HTML GUI and the status pages
manager-script - allows access to the text interface and the status pages
manager-jmx - allows access to the JMX proxy and the status pages
manager-status - allows access to the status pages only


 2. Setup the tomcat server in maven's settings.xml, under servers.
   
 <server>
<id>TomcatServer</id>
<username>admin</username>
<password>admin</password>
      </server>

 3. In the pom.xml, add the below plugin:

            <plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>tomcat-maven-plugin</artifactId>
<configuration>
   <!-- Manager path in tomcat 7 is different than other versions of tomcat. -->
<url>http://localhost:8080/manager/text</url>
<!-- This is the server ID, given in the settings.xml -->
<server>TomcatServer</server>
<path>/exampleApplication</path>
</configuration>
</plugin>

Note that tomcat 7 has a different manager path than other versions of tomcat.

  4. Use command: mvn tomcat:deploy to deploy war file and command: mvn tomcat:redeploy to redeploy the web application. 

Using same Embedded object more than once in the same entity

If more than once we have to use the same Embedded object, then we have to use @Attributeoverrides and @AttributeOverride annotations to give override the column names of the embedded objects. Otherwise, we get a error like this : "org.Hibernate.MappingException: Repeated column in mapping entry:" for all the columns of the Embedded objects that have been repeated.

For example, if Name is a Embedded object in Student Entity, then the mapping should be like this

 @Entity  
 public class Student implements Serializable {  
      @AttributeOverrides({  
                @AttributeOverride(name = "firstName", column = @Column(name = "FATHER_FIRST_NAME")),  
                @AttributeOverride(name = "surName", column = @Column(name = "FATHER_SUR_NAME")),  
                @AttributeOverride(name = "lastName", column = @Column(name = "FATHER_LAST_NAME")),  
                @AttributeOverride(name = "displayName", column = @Column(name = "FATHER_DISPLAY_NAME")) })  
      @Embedded  
      private Name father;  
      @AttributeOverrides({  
                @AttributeOverride(name = "firstName", column = @Column(name = "MOTHER_FIRST_NAME")),  
                @AttributeOverride(name = "surName", column = @Column(name = "MOTHER_SUR_NAME")),  
                @AttributeOverride(name = "lastName", column = @Column(name = "MOTHER_LAST_NAME")),  
                @AttributeOverride(name = "displayName", column = @Column(name = "MOTHER_DISPLAY_NAME")) })  
      @Embedded  
      private Name mother;  
  ...  
 }  

And Embeddable object'r mapping being:

 @Embeddable  
 public class Name implements Serializable {  
      /**  
       * Default generated Serialization ID  
       */  
      private static final long serialVersionUID = -971064645279229579L;  
      @Column(name = "FIRST_NAME")  
      private String firstName;  
      @Column(name = "SUR_NAME")  
      private String surName;  
      @Column(name = "LAST_NAME")  
      private String lastName;  
 ...   
 }  

Configuring log4j with Spring framework

• Make sure that the commons-logging is there in the classpath of the projet. Including spring-core will automatically include commins-logging with it. I tried for spring-core version 3.1.1.RELEASE
• First, define the log4j.xml with the desired log4j configuration for the project. Example is given below. Place the file along with log4j.dtd in the resources/config folder of the  project.

 <?xml version="1.0" encoding="UTF-8" ?>  
 <!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">  
 <log4j:configuration xmlns:log4j='http://jakarta.apache.org/log4j/'>  
      <!-- console logger -->  
      <appender name="stdout" class="org.apache.log4j.ConsoleAppender">  
           <layout class="org.apache.log4j.PatternLayout">  
                <param name="ConversionPattern"  
                     value="%-5p [%d{EEE MMM d HH:mm:ss z yyyy:}] [%t] [%X{ProcessId}] [%L] %c : %m%n" />  
           </layout>  
           <filter class="org.apache.log4j.varia.LevelRangeFilter">  
                <param name="LevelMin" value="DEBUG" />  
           </filter>  
      </appender>  
     <!-- Specify the path for the logFile -->  
      <appender name="FILE" class="org.apache.log4j.RollingFileAppender">  
           <param name="File" value="<path of the log file>" />  
           <param name="Threshold" value="DEBUG" />  
           <param name="MaxFileSize" value="10MB" />  
           <param name="MaxBackupIndex" value="10" />  
           <layout class="org.apache.log4j.PatternLayout">  
                <param name="ConversionPattern" value="%d{ISO8601}: [%t] %c, %p, %X{ProcessId}: %m%n" />  
           </layout>  
      </appender>  
      <!-- Place your application's package whose classes needs to be logger enabler at Debug level. -->  
      <logger name="<Your application package>" additivity="false">  
           <level value="DEBUG" />  
           <appender-ref ref="stdout" />  
           <appender-ref ref="FILE" />  
      </logger>  
      <logger name="org.springframework" additivity="false">  
           <level value="INFO" />  
           <appender-ref ref="stdout" />  
           <appender-ref ref="FILE" />  
      </logger>  
      <logger name="org.hibernate" additivity="false">  
           <level value="INFO" />  
           <appender-ref ref="stdout" />  
           <appender-ref ref="FILE" />  
      </logger>  
      <root>  
           <priority value="DEBUG" />  
           <appender-ref ref="stdout" />  
           <appender-ref ref="FILE" />  
      </root>  
 </log4j:configuration>  

• In the spring configuration, define a bean of type 'org.springframework.beans.factory.config.MethodInvokingFactoryBean' like this:

 

   <bean id="log4jInitialization"  
           class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">  
           <property name="targetClass" value="org.springframework.util.Log4jConfigurer" />  
           <property name="targetMethod" value="initLogging" />  
           <property name="arguments">  
                <list>  
                     <value>classpath:config/log4j.xml</value>  
                </list>  
           </property>  
      </bean>  

          The above bean is for the class org.springframework.beans.factory.config.MethodInvokingFactoryBean, which takes in 3 arguments. First the targetClass, which in this case is Spring's Log4jConfigurer class and the second argument will be the method "initLogging" of class Log4jConfigurer, which will be called with the third arguments passed in as the third argument.
We will pass the location of the log4j.xml.

During runtime, MethodInvokingFactoryBean will invoke the method "initLogging" of class Log4jConfigurer with the arguments passes to it through property "arguments"

• Class in which the logging should be used should have a declaration like this.   org.apache.log4j.Logger logger = org.apache.log4j.Logger.getLogger(clazz);

• Level of logging can be set in the log4j.xml.

Insertion Sort

Insertion sort, as the name says works by inserting the element in proper position of a sorted list. This works by first marking a element in the array as a markedElement. All the elements before the markedElement will be sorted. All the elememts to the right of the markedElement will be unsorted.
Sorting proceeds by comparing the markedElement with the sorted elements, inserting the markedElement in the correct position based on the comparision.

In Detail: For example, if we consider an array of N elements, to startwith, we will consider the second element as the markedElement. So, to the left of the markedElement is only one element, which we consider as sorted.(Remember, elements towards left of the markedElement is considered as sorted).
We proceed by comparing the markedElement with the largest element(rightmost element) in the sorted list. Upon each pass, the size of the sorted sub-list will increase.

Best case sorting is when the elements are already sorted. Then, the sort has a complexity of O(N). Worst case is when a array is sorted in the reverse order. Thats when the complexity will be quadratic O(N2).
However, insertion sort is considered as a quickest way to sort small data. It is considered quicker than quickSort for small data. It is not a good solution for sorting large data.



InsertionSort Program

package com.algoWork.sort;

import java.util.ArrayList;
import java.util.List;

public class InsertionSort<T extends Comparable<T>> {

    public void sort(List<T> arrayElements) {

        // Iterate over the arrayElements from second element
        for (int i = 1; i < arrayElements.size(); i++) {

            // consider the secondElement as the markedElement
            T markedElement = arrayElements.get(i);
            // Get the index of the markedElement
            int markedElementIndex = i;
            // Keep shifting the elements in the subList towards right until you
            // find the correct position to insert the markedElement. This
            // Comparison will be done till markedIndex is greater than 0 and
            // markedElement is less that the compared element in the subList.

            while (markedElementIndex > 0
                    && markedElement.compareTo(arrayElements
                            .get(markedElementIndex - 1)) < 0) {
                arrayElements.set(markedElementIndex,
                        arrayElements.get(markedElementIndex - 1));
                markedElementIndex--;
            }
            // At the end of the loop, correct position to insert the
            // markedElement in the subList will be found and insert the element
            // in that position.
            arrayElements.set(markedElementIndex, markedElement);
        }

    }

}

Complexity: Complexity of this sorting mechanism is O(N2). But this is twice as fast compared to Bubble sort and Selection sort.

Reference: Data Structures and Algorithms 2nd Edition Robert Lafore