Message Queuing Frameworks – Comparison Grid


We often come with requirements which are suited for integrating Messaging Frameworks in the Software Systems.
There are many messaging frameworks available in the market – some are open-source, some are paid-licensed, some provide great support, have good Community support.

In order to make an apt choice, we look out and explore different messaging frameworks based on our requirements.

This post compares few Popular Messaging Frameworks and aims to provide or equip you with enough information to make a decision on choosing the best framework as per your requirements.

COMPARISON GRID

RabbitMQ Apache Kafka AWS SQS
HA ☑ Requires some extra work and
may require 3rd party Plugins like Shovel and Federation
☑ Out of the Box (OOB) ☑ OOB
Scalable
Guaranteed Delivery ☑ Supports Consumer Acknowledgments ☑ Supports Consumer Acknowledgments ☑ Supports Consumer Acknowledgments
Durable ☑ Through Disk Nodes and Queues with extra configuration ☑ OOB ☑ Message Retention upto 14 days max and default being 4 days.
Exactly-Once Delivery ☑ Annotates a message with redelivered when
message was delivered earlier but consumer ack failed earlier.
Requires Idempotent behavior of a Consumer
☑ Dependent on Consumer behavior.

Consumer is made responsible to track Offsets (messages read so far) and store those offsets. Kafka started supporting storing offsets within Kafka itself.

It supports storing Offsets OOB through HIGH LEVEL CONSUMERS, however

Requires
Idempotent
behavior of a Consumer

☑ MessageDeDup ID and MessageGroupID attributes are used.

Requires
Idempotent
behavior of a Consumer.
FIFO supports Exactly-once while Standard Queues support Atleast-Once

Ease of Deployment ☑ For Distributed Topology,
Requires more effort and 3rd party Plugins
☑ Requires ZooKeeper ☑ Managed by AWS
Authentication Support ☑ OOB ☑ OOB ☑ OOB
Authorization aka Acl Support ☑ OOB ☑ OOB ☑ OOB
TLS Support ☑ OOB ☑ OOB ☑ OOB
Non-Bocking Producers ☑ Supports both Synchronous and Async
Performant ⭐ ⭐ Medium to High ⭐ ⭐ ⭐ ⭐ Very High ⭐ ⭐ ⭐ ⭐ Very High

FIFO: 300 tps
Standard Queues: Unlimited

Open Source  ☑  ☑
Load Balancing Across Consumers ☑ Can be done Through Consumer Groups ☑ Multiple Consumers can read from the same queue in an atomic way
Delay Queues NOT OOB NOT OOB ☑ OOB
Visibility Timeout Queues NOT OOB NOT OOB ☑ OOB
Message Dedup  ☑
Message Size Limits Upto 256 KB. AWS SDK supports storing large messages in S3 etc though.
No of Messages in a Queue ☑ No limits ☑ No Limits ☑ No Limits but

Standard Queues: 1,20,000 In-Flight Messages
FIFO: 20,000 In-Flight Messages

Details here and here

Messages are In-Flight after they have been received from the queue by a consuming component, but have not yet been deleted from the queue

Message Content Limits ☑ No Limits ☑ No Limits A message can include only XML, JSON, and unformatted text. The following Unicode characters are allowed: #x9 | #xA | #xD | #x20 to #xD7FF | #xE000 to #xFFFD | #x10000to #x10FFFF

Any characters not included in this list are rejected.

Disaster Recovery Not OOB Not OOB but simple. Replicas can be deployed across regions Not OOB and simple. Requires different Strategies to achieve it.

Advertisements
Posted in Common Technical Problems, Technical | Tagged , , , , , | Leave a comment

Pass Custom DateTime Zone in SQL Query Date Time Parameter | Hibernate


Using Hibernate and Struggling with querying DateTime Column in RDBMS (like MS-SQL) in specific timezone?
No matter what Timezone your DateTime object has, while issuing hibernate query,
do you observe that Time in Default Timezone of JVM is always getting passed and thus not giving you desired results?

If that’s the case, this article describes a process to achieve querying DateTime column with specific timezone.

WHY THIS HAPPENS?

It is because your Application Server and Database Server are running in Different TimeZones.

If your Application Server and Database Server are running in different TimeZones, we need to ensure that the Date Time query parameter values shall be sent as per DB Timezone to get desired results.

Let’s understand how does Hibernate\DB Driver forms a Sql Query in the next section.

HOW HIBERNATE CREATES A QUERY?

On an Application Server, DB Driver forms a Command before sending it to RDBMS. Database System then executes the query (may compile if needed) and return the results accordingly.

DB Driver instantiates a Command in the form of PreparedStatement object. Then, DBConnection is attached with the above Command Object on which this command will be executed. Since we want to query by certain parameters, DateTime in our case, DB Driver sets the query parameters on the command. 

PreparedStatement exposes few APIs to set different parameters depending upon the type of the parameter.
To pass DateTime information, various APIS being exposed are:

  • setDate
  • setTime
  • setTimestamp

All these functions allow passing Calendar object to be passed. Using this Calendar object, Driver constructs the SQL DateTime value.

If this Calendar object is not passed, Driver then uses the DEFAULT TIMEZONE of the JVM running the application. This is where things go wrong and desired results are not obtained.

How can we solve it then?

DIFFERENT APPROACHES

  1. Setting same timezone of the Application Server and of DB Server
  2. Setting timezone of the JVM as that of DB Server
  3. By extending the TimestampTypeDescriptor and AbstractSingleColumnStandardBasicType classes and attaching to the Driver

1st and 2nd Approaches are fine, however these can have side-effects.

1st can impact other applications which are running on the same system. Usually, 1 application runs on a single server in Production or LIVE environment, however, with this we are delimiting the deployment of other applications.

2nd approach is better than 1st one since it will not impact other applications, however, the caveat here is what if your application is talking to different DB Systems which are in different timezones. Or, what if you want to set TimeZone on only few selected Time Fields.

3rd approach is flexible. It allows you to represent different time fields in even different time zones.

AlRight. Can we have steps then to implement Approach #3

STEPS FOR 3rd Approach:

Provide Custom TimestampTypeDescriptor and AbstractSingleColumnStandardBasicType
  • Implement Descriptor class as given below:
    import java.sql.CallableStatement;
    import java.sql.PreparedStatement;
    import java.sql.ResultSet;
    import java.sql.SQLException;
    import java.sql.Timestamp;
    import java.util.Calendar;
    import java.util.TimeZone;
    
    import org.hibernate.type.descriptor.ValueBinder;
    import org.hibernate.type.descriptor.ValueExtractor;
    import org.hibernate.type.descriptor.WrapperOptions;
    import org.hibernate.type.descriptor.java.JavaTypeDescriptor;
    import org.hibernate.type.descriptor.sql.BasicBinder;
    import org.hibernate.type.descriptor.sql.BasicExtractor;
    import org.hibernate.type.descriptor.sql.TimestampTypeDescriptor;
    
    /**
     * Descriptor for {@link Types#TIMESTAMP TIMESTAMP} handling with zone.
     */
    public class CustomZonedTimestampDescriptor extends TimestampTypeDescriptor {
        public static final CustomZonedTimestampDescriptor PST_INSTANCE = new CustomZonedTimestampDescriptor();
    
        /**
         * Instantiate an object of CustomZonedTimestampDescriptor with Timezone set to "America/Los_Angeles"
         */
        public CustomZonedTimestampDescriptor() {
            this.calendar = Calendar.getInstance(TimeZone.getTimeZone("America/Los_Angeles"));
        }
    
        /**
         * Instantiate an object of CustomZonedTimestampDescriptor
         * @param zone Timezone to be used
         */
        public CustomZonedTimestampDescriptor(TimeZone zone) {
            this.calendar = Calendar.getInstance(zone);
        }
    
        /**
         * Get the binder (setting JDBC in-going parameter values) capable of handling values of the type described by the
         * passed descriptor.
         *
         * @param javaTypeDescriptor The descriptor describing the types of Java values to be bound
         *
         * @return The appropriate binder.
         */
        @Override
        public <X> ValueBinder<X> getBinder(final JavaTypeDescriptor<X> javaTypeDescriptor) {
            return new BasicBinder<X>( javaTypeDescriptor, this ) {
                @Override
                protected void doBind(PreparedStatement st, X value, int index, WrapperOptions options) throws
                        SQLException {
                    st.setTimestamp(index, javaTypeDescriptor.unwrap(value, Timestamp.class, options), calendar);
                }
            };
        }
    
        /**
         * Get the extractor (pulling out-going values from JDBC objects) capable of handling values of the type described
         * by the passed descriptor.
         *
         * @param javaTypeDescriptor The descriptor describing the types of Java values to be extracted
         *
         * @return The appropriate extractor
         */
        @Override
        public <X> ValueExtractor<X> getExtractor(final JavaTypeDescriptor<X> javaTypeDescriptor) {
            return new BasicExtractor<X>( javaTypeDescriptor, this ) {
                @Override
                protected X doExtract(ResultSet rs, String name, WrapperOptions options) throws SQLException {
                    return javaTypeDescriptor.wrap(rs.getTimestamp(name, calendar), options);
                }
    
                @Override
                protected X doExtract(CallableStatement statement, int index, WrapperOptions options) throws SQLException {
                    return javaTypeDescriptor.wrap(statement.getTimestamp(index, calendar), options);
                }
    
                @Override
                protected X doExtract(CallableStatement statement, String name, WrapperOptions options)
                        throws SQLException {
                    return javaTypeDescriptor.wrap(statement.getTimestamp(name, calendar), options);
                }
            };
        }
    
        private final Calendar calendar;
    }
    

    In the above code, Default constructor uses PST Timezone by default. For other TimeZones, simply use the Parameterized Constructor.

  • Implement Type class and use the above Descriptor class
    import com.expedia.www.air.commission.migration.db.descriptors.CustomZonedTimestampDescriptor;
    
    import java.util.Comparator;
    import java.util.Date;
    import java.util.TimeZone;
    
    import org.hibernate.dialect.Dialect;
    import org.hibernate.engine.spi.SessionImplementor;
    import org.hibernate.type.AbstractSingleColumnStandardBasicType;
    import org.hibernate.type.LiteralType;
    import org.hibernate.type.TimestampType;
    import org.hibernate.type.VersionType;
    import org.hibernate.type.descriptor.java.JdbcTimestampTypeDescriptor;
    
    /**
     * A type that maps between {@link java.sql.Types#TIMESTAMP TIMESTAMP} and {@link java.sql.Timestamp} with zone
     */
    public class CustomZonedTimestampType extends AbstractSingleColumnStandardBasicType<Date>
            implements VersionType<Date>, LiteralType<Date> {
        /**
         * Instantiate an object of CustomZonedTimestampType with Timezone set to "America/Los_Angeles"
         */
        public CustomZonedTimestampType() {
            this(CustomZonedTimestampDescriptor.PST_INSTANCE);
        }
    
        /**
         * Instantiate an object of CustomZonedTimestampType
         * @param zone Timezone to be used
         */
        public CustomZonedTimestampType(TimeZone zone) {
            super(new CustomZonedTimestampDescriptor(zone), JdbcTimestampTypeDescriptor.INSTANCE);
        }
    
        /**
         * Returns the abbreviated name of the type.
         * @return String the Hibernate type name
         */
        @Override
        public String getName() {
            return TimestampType.INSTANCE.getName();
        }
    
        /**
         * Convert the value into a string representation, suitable for embedding in an SQL statement as a
         * literal.
         * @param value The value to convert
         * @param dialect The SQL dialect
         * @return The value's string representation
         * @throws Exception Indicates an issue converting the value to literal string.
         */
        @Override
        public String objectToSQLString(Date value, Dialect dialect) throws Exception {
            return TimestampType.INSTANCE.objectToSQLString(value, dialect);
        }
    
        /**
         * Generate an initial version.
         * @param session The session from which this request originates.
         * @return an instance of the type
         */
        @Override
        public Date seed(SessionImplementor session) {
            return TimestampType.INSTANCE.seed(session);
        }
    
        /**
         * Increment the version.
         * @param current the current version
         * @param session The session from which this request originates.
         * @return an instance of the type
         */
        @Override
        public Date next(Date current, SessionImplementor session) {
            return TimestampType.INSTANCE.next(current, session);
        }
    
        /**
         * Get a comparator for version values.
         * @return The comparator to use to compare different version values.
         */
        @Override
        public Comparator<Date> getComparator() {
            return getJavaTypeDescriptor().getComparator();
        }
    }
    
  • Add an annotation @Type on the fields for which proper Timezone has to be used
    @Entity
    public class MyEntityClass implements Serializable {
        public static final String CUSTOMIZED_TIMESTAMP_TYPE = "com.db.types.CustomZonedTimestampType";
    
        public Date getUpdateDate() {
            return updateDate;
        }
    
        public void setUpdateDate(Date updateDate) {
            this.updateDate = updateDate;
        }
    
        @Type(type = CUSTOMIZED_TIMESTAMP_TYPE)
        private Date updateDate;
    }
    

As per the above code, MyEntityClass has a field named upDateDate for which we want to send the date and time in correct Timezone.

ADVANTAGES

  • Reliability: Expected Results from DB are obtained
  • No other application running on a same system is impacted
  • Most importantly, the above steps enable a code to be run on any system set in any TimeZone

 

 

Posted in Java, Technical | Tagged , , , , | Leave a comment

Continuous Data Sync Across Hetereogeneous Data Persistent Systems


INTRODUCTION

Often a need arises to migrate the data from one System to another system. These Persistent Data Systems, Source and Destination, could be entirely different, from different vendors.
It could be due to change in requirements or technology advancements.

Add to it the changes in above tier which is making use of Persistent System.
To make sure that everything works fine on new system, you may plan to start executing small %age of traffic on New System and calibrate\compare the results with Old Stack results.

For proper calibration and find out the differences in result set from Old and New Systems, the task in hand is to synchronize the Data across 2 systems, being stored differently in different systems.

If that’s the case, this article can help you achieve Data Synchronization across Heterogeneous Systems on an ongoing basis.

This article aims to present the concept to seamlessly move the data incrementally from your current data storage system to different data storage system, be it on Premise or on cloud.

TERMS USED

Batch: Means a collection of data records to be moved across
BatchState: Represents the Status of Batch Transfer, whether it is IN_PROGRESS, FAILED, COMPLETED
Metadata: Represents the batch details which will help in detecting the next batch of data to be synchronized

WHICH COMPONENTS ARE INVOLVED?

Data Source : Actual source containing the original data to be synchronized
Data Destination: This is the Persistent System where you want your data to be moved to
Syncer Component: Responsible to detect the incremental changes and synchronize
Transformer Component: Responsible to transform the source data structure into Destination DS. This will be required if you restructure the data.
Tracker System: Responsible to store the status and the details related to last batch of data being sync’ed

Below diagram depicts the Problem statement of Sync’ing the On-Premise RDBMS data to No-SQL, MongoDB Storage System in AWS Cloud.

Pictorial_HL

WHY INCREMENTALLY?

You may have a huge data in your storage system which you cannot move in a single operation. This could be due to resource constraints of memory, network etc which may hinder the data synchronization.

And what if this data is changed frequently by Biz users. Doing Full synchronization each time can prove to be costly.

How can we reduce this cost? How do we increase the chances of successful data synchronization?

How can we make this process resilient and resume from the point where it stopped or failed the last time?

How about splitting up the Data to be synchronized?

How about defining a batch of data, pull up the data of this batch only and then transfer this data batch?

In order to accomplish this, we need to store the details using which we can determine, how much data we have already sync’ed and what is the next batch of data that we need to sync.

HOW IT WORKS?

Before we go further into steps involved, lets understand the batch Metadata.

WHAT COULD BE BATCH METADATA?
{
"batchOffset" : 0,
"batchSize" : 20,
"migrationStartDateTime" : NumberLong(1486026714594),
"migrationEndDateTime" : NumberLong(1486026718245),
"rulesUpdateDateTimeBeginPickedUpForMigration" : null,
"rulesUpdateDateTimeEndPickedUpForMigration" : null,
"status" : "COMPLETED",
"remarks" : "",
"isIncrementalModeOn" : false
}

batchOffset is the marker. Based on the Status, you can compute from where to begin or resume the process. So, if last batch was successfully sync’ed, next batch to be sync’ed starts with batchOffset+batchSize, or, otherwise, batchOffset in case the last batch failed.

batchSize denotes the no of records you want to sync in a single operation and thus it also tells the amount of data.
It shall neither be too small (otherwise resulting in more roundtrips and more processing time) nor too big (otherwise requiring more resources – memory, network bandwidth etc)

status denotes the sync operation Status of the batch

isIncrementalModeOn denotes whether  sync process is just pulling up the incremental updates (including additions) or not. This does mean that source data had been completely synchronized once.

rulesUpdateDateTimeBeginPickedUpForMigration and rulesUpdateDateTimeEndPickedUpForMigration denote the time boundaries for incremental updates. These are useful in pulling up the incremental changes during this time period.

migrationStartDateTime and migrationEndDateTime are useful for tracking purposes to determine how much time did this batch sync take.

With this information, let’s see the sequence of events which happen to sync the batch of data.

The process is initiated or resumed with Syncer component.

  1. Syncer pulls up the last migrated batch details form the Tracker system.
  2. Using Batch Metadata, it identifies the next batch of data to be synchronized.
    It makes an entry into Tracker System to store the next batch metadata with IN_PROGRESS status.
  3. It then builds the query, pulls up the records as per next batch metadata from Source system. You can use any ORM, hibernate or jpa to get the data.
  4. It then delegates to Transformer to transform the source data structure to destination data structure.
  5. With transformed data, it identifies the data to be created and data to be updated and accordingly splits the data.
  6. It then sends data to Destination System.
  7. Depending upon the operation status, it marks the Batch either as COMPLETED or FAILED status.

And, these sequence of steps go on till there isn’t any more data to sync.

At this point, isIncrementalModeOn is saved as TRUE in the Tracker system and post this, SYNCER System can tweak the query to pull the data records for a time window.

 

This slideshow requires JavaScript.

BATCH PROCESS STATE

In case you want to have Primary, Secondary Sync Processes so as to guarantee the High Availability of the Sync Process, we need to maintain and detect the various states of a Sync Process. With this data, we can ensure that at a time, no 2 sync processes are running.

BatchProcessStates

BATCH STATES aka STATUS

Every individual batch of data goes through few states. Below diagram represents the various states, a batch goes through in a syncing process.

BatchStates

THINGS TO KEEP IN MIND

  • Idempotency and Duplicacy Prevention:

We are transferring a batch of records. Therefore, it may happen that batch gets Partially Succeeded, meaning that few records got sync’ed and rest failed due to any reasons. In such cases, if you retry posting the data, it may result into same data getting saved twice or more. To prevent this, query what data has to be inserted and what data has to be updated. You can make use of indexes or similar concept.

  • Timezone Differences:

Syncer system and Data Source System can be in different timezones or source data may be stored in a specific Timezone. So, if you are pulling up records based on time window, make sure that timezone information is converted into source system before querying.

  • Security:

For sensitive data, you can enable SSL/ TLS over transport layer. Also, You may want to have authentication and authorization enabled on both data ends: Source and Destination Storage Systems.

  • Hard Deletes:

Soft Deletes like making biz rule inactive or likewise will be taken care by Syncer process. What if tuple is hard deleted from a source storage. For Hard deletes, you may have to use Triggers to catch the deleted tuples.

  • Alert Mechanism to detect Stopped Sync Process:

Sync process can also fail due to any reason. Without any alerting mechanism, it may go unnoticed and these Heterogeneous Systems can go out of sync. To prevent such circumstances, log start, stop events into some sinks like Splunk and have Alerts on them.

WHAT QoS PARAMETERS ARE IMPLEMENTED?

  • Eventual Consistency
  • Guaranteed Sync
  • FaultTolerance
  • Idempotency
  • Also, Updates while Sync are not missed

HOSTING MECHANISM

There can be multiple ways to host a Syncer process. Depending upon the traffic your consuming application takes, you can

  • Either host the syncer process under the same application which relies on this data
  • Or, host it under a separate process and schedule it using AWS Lambda or AWS Batch

ALTERNATIVES

Amazon DMS also offers the ongoing data migration however it supports only selected Storage Systems. At the time of implementing this, Amazon DMS Offering does not have MSSQL –> MongoDB supported.

If you want to sync data to AWS RDS, Amazon DMS can be used.

Also, if you have huge data ranging in hundreds of TBs and a limited network bandwidth and wants to get this done quickly and only for once, AWS Snowball is another offering you can use.

Posted in Technical | Tagged , , , , , , , , , , , , , , , , | Leave a comment

Health Checks : Detection, Reporting, Configuration of Server Instance\Process Health Status


In this article, i will talk about the Running Instance Health, what can represent the Health, how can we detect the health and how can we use this health information to make the System resilient.

Health, basically, defines how well an instance is responding. Health can be:

  • UP
  • DOWN

REAL LIFE PROBLEM
Imagine you reach a Bank and found it being closed. Or, Imagine you are standing in a bank counter queue and waiting to be served. By the time your turn arrives, person sitting at a counter goes away. May be that person is not feeling well.

How would you feel in such a situation? Irritated? Frustrated?
What if you would have been told upfront about this situation? Your time would not have wasted. You would not have felt bad.

But what if someone else takes a job of that counter and start serving you.

Now, imagine a pool of servers hosting a site which allows you to upload a video, say http://www.Youtube.com. You are trying to upload a small video of yours on a site and every time you try to upload, you get some error after sometime and video could not be uploaded.

Basically, Software Applications like http://www.youtube.com run on machines – be it physical or virtual in order to get desired results. Executing these applications require machine’s local resources like memory, cpu, network, disk etc or other external dependencies to get things done.
These resources are limited and executing multiple tasks concurrently put a risk of contention and exhaustion.
It may happen that enough resources are not available for execution and thus the task execution will eventually fail.

In order to make the system Resilient, one of the things that can be done is Proactively determine the Health Status and  report it – to LoadBalancer or to Service Discoverers etc whenever asked, to prevent or deal with the failures.

Reporting a health Status with proper Http Status Codes like 200 for UP and 500 for DOWN can be quite useful.

WHAT CAN DEFINE INSTANCE\PROCESS HEALTH?
Below is a list of some common metrics that can be useful in detecting the health of an instance:

  • Pending Requests
    • Container Level
    • Message Level
  • Latency Overhead – Defined as the TP99 latency added by this application/layer
    • TP99 or TP95 or TP75 as per your Service SLAs
  • Resources
    • % Memory Utilization – Leading towards OOM
    • % CPU Utilization
      • Host Level
      • Process Level
    • Number of Threads
  • Any Business KPI
  • External Dependencies Failures optioanlly

Identifying a list of above criterias is important as well as choosing the correct Threshold or Saturation Values as well.
Too low values or high values can result into system unreliability.

WHY IS IT IMPORTANT?

System is usually expected to be highly available and reliable. High Availability can be achieved through Redundancy where in multiple server instances are running in parallel, processing the requests and thus the demand.

What if One or more instances are running out of resources and thus not able to meet the demand.

Detecting such a state at an appropriate time and taking an action can help in achieving High Availability and Reliability of the System.

It helps in making the system resilient against failures.

ACTIONS ON DETECTING UNHEALTHY

  • REPLENISH thru REBOOT: If you have limited servers pool capacity and cannot increase the capacity, the unhealthy machine has to be restarted\rebooted in order to get it back to healthy state.
  • REPLACE: If you have unlimited server capacity or using Cloud Computing Framework – AWS, Azure, Google Cloud etc, rather than rebooting the machine, you have an option of starting a new machine and killing and removing the old unhealthy machine from processing the requests.

Once an instance is detected unhealthy, instance shall be replenished or replaced.
Either that unhealthy instance shall be rebooted to get it to Healthy state or be replaced with a new server which is put behind LoadBalancer and old being removed from LoadBalancer.

OTHER CONSIDERATIONS

  • Do enable Connection Draining
  • Do configure Connection Draining timeout
  • Enable HealthCheck Response Caching
  • Scale before Declaring UnHealthy
  • Prefer Recent Trend before Declaring UnHealthy – configure unHealthy, healthy Thresholds

These settings prevent the In-Flight requests to be aborted prematurely.
Without these settings, data can be inconsistent state

  • Report Health with Proper Http Status Codes
    • 200 for UP
    • 500 for DOWN

CODE IMPLEMENTATION

Basically, what we need is to peek into current metrics and evaluate the Health as UP or DOWN

So, we need an HealthEvaluator, List of HealthCriteria, Some Operators and Health Definition.

public interface IHealthEvaluator {
    /**
     * Return an indication of health.
     * @return the health after consulting different metrics
     */
    Health health();
}
public final class CompositeMetricBasedHealthEvaluator implements IHealthEvaluator {
    /**
     * Instantiates an object of CompositeMetricBasedHealthEvaluator
     * @param healthCriteriaList List containing Metrics to be used for Health Evaluation
     * @param metricReadersList List containing Metric Readers
     */
    public CompositeMetricBasedHealthEvaluator(List<HealthCriteria<Number>> healthCriteriaList,
                                               List<MetricReader> metricReadersList) {
        this(healthCriteriaList, metricReadersList, null);
    }

    /**
     * Instantiates an object of CompositeMetricBasedHealthEvaluator
     * @param healthCriteriaList List containing Metrics to be used for Health Evaluation
     * @param metricReadersList List containing Metric Readers
     * @param metricsList List containing the Public Metrics
     */
    public CompositeMetricBasedHealthEvaluator(List<HealthCriteria<Number>> healthCriteriaList,
                                               List<MetricReader> metricReadersList,
                                               List<PublicMetrics> metricsList) {
        this.healthCriteriaList = CollectionUtils.isNotEmpty(healthCriteriaList)
                ? ListUtils.unmodifiableList(healthCriteriaList) : ListUtils.EMPTY_LIST;
        this.metricReaderList = metricReadersList;
        this.metricsList = metricsList;
    }

    /**
     * Return an indication of health.
     * @return the health after consulting different metrics
     */
    @Override
    public Health health() {
        Health.Builder curHealth = Health.up();
        Status status = Status.UP;
        for (HealthCriteria healthCriteria : this.healthCriteriaList) {
            String metricName = healthCriteria.getMetricName();
            if (StringUtils.isNotBlank(metricName)) {
                Metric metric = this.getFirstMatchingMetric(metricName);
                if (metric != null) {
                    status = evaluate(healthCriteria, metric);
                    curHealth.withDetail(metricName, String.format("Value:%s, Status:%s", metric.getValue(), status));
                } else {
                    curHealth.withDetail(metricName, Status.UNKNOWN);
                }
            }
        }

        curHealth.status(status);

        return curHealth.build();
    }

    private Metric getFirstMatchingMetric(String name) {
        Object metricProvider = this.selectedMetricProvider.get(name);

        if (metricProvider instanceof MetricReader) {
            return find((MetricReader) metricProvider, name);
        } else if (metricProvider instanceof PublicMetrics) {
            return find((PublicMetrics) metricProvider, name);
        }

        // Preference to use MetricReaders
        if (CollectionUtils.isNotEmpty(this.metricReaderList)) {
            for (MetricReader metricReader : this.metricReaderList) {
                Metric<?> metric = find(metricReader, name);
                    if (metric != null) {
                        this.selectedMetricProvider.put(name, metricReader);
                        return metric;
                    }
            }
        }

        if (CollectionUtils.isNotEmpty(this.metricsList)) {
            for (PublicMetrics publicMetrics : this.metricsList) {
                Metric<?> metric = find(publicMetrics, name);
                if (metric != null) {
                    this.selectedMetricProvider.put(name, publicMetrics);
                    break;
                }
            }
        }

        return null;
    }

    private static Status evaluate(HealthCriteria healthCriteria, Metric metric) {
        int result = compare(metric.getValue(), healthCriteria.getThresholdOrSaturationLevel());
        ComparisonOperator op = healthCriteria.getOperator();

        if ((ComparisonOperator.EQUAL.equals(op) && result != 0) ||
                (ComparisonOperator.LESS_THAN.equals(op) && result >= 0) ||
                (ComparisonOperator.LESS_THAN_EQUAL.equals(op) && result > 0) ||
                (ComparisonOperator.GREATER_THAN.equals(op) && result <= 0) ||
                (ComparisonOperator.GREATER_THAN_EQUAL.equals(op) && result < 0)) {
            return Status.DOWN;
        }

        return Status.UP;
    }

    private static Metric<?> find(MetricReader reader, String name) {
        try {
            return reader.findOne(name);
        } catch (RuntimeException ex) {
            // Ignore the Runtime exceptions
            return null;
        }
    }

    private static Metric<?> find(PublicMetrics source, String name) {
        return (Metric<?>) CollectionUtils.find(source.metrics(),
                (met) -> StringUtils.equalsIgnoreCase(((Metric) met).getName(), name));
    }

    private static int compare(Number n1, Number n2) {
        if (n1 != null && n2 != null) {
            return Double.compare(n1.doubleValue(), n2.doubleValue());
        }

        if (n1 != null) {
            return 1;
        }

        if (n2 != null) {
            return -1; // Even for -ive numbers
        }
        return 0;
    }

    private final List<HealthCriteria<Number>> healthCriteriaList;
    private final List<PublicMetrics> metricsList;
    private final List<MetricReader> metricReaderList;
    private final Map<String, Object> selectedMetricProvider = new HashMap<>();
}

HealthCriteria defines 3 things: what has to be checked, it’s expected value(or a range) and Operator. Value can be integer, float or decimal etc

public class HealthCriteria<TInput extends Number> {
    /**
     * Gets the Operator
     * @return Operator to be used for health evaluation
     */
    public ComparisonOperator getOperator() {
        return operator;
    }

    /**
     * Sets the Operator
     * @param operator Operator to be used for health evaluation
     */
    public void setOperator(ComparisonOperator operator) {
        this.operator = operator;
    }

    /**
     * Gets the Threshold or Saturation value against which health evaluation to be done
     * @return Threshold or Saturation value
     */
    public TInput getThresholdOrSaturationLevel() {
        return thresholdOrSaturationLevel;
    }

    /**
     * Sets the Threshold or Saturation value against which health evaluation to be done
     * @param thresholdOrSaturationLevel Threshold or Saturation value
     */
    public void setThresholdOrSaturationLevel(TInput thresholdOrSaturationLevel) {
        this.thresholdOrSaturationLevel = thresholdOrSaturationLevel;
    }

    /**
     * Gets the name of the metric to be used for health evaluation
     * @return Metric name
     */
    public String getMetricName() {
        return metricName;
    }

    /**
     * Sets the name of the metric to be used for health evaluation
     * @param metricName Metric name
     */
    public void setMetricName(String metricName) {
        this.metricName = metricName;
    }

    private String metricName;
    private TInput thresholdOrSaturationLevel;
    private ComparisonOperator operator;
}

@Configuration
@ConfigurationProperties("healthIndicator")
public class HealthCriteriaList {
    public List<HealthCriteria<Number>> getCriterias() {
        return criterias;
    }

    public void setCriterias(List<HealthCriteria<Number>> criterias) {
        this.criterias = criterias;
    }

    private List<HealthCriteria<Number>> criterias;
}

Some basic Operators that can be supported are:

public enum ComparisonOperator {
    EQUAL,
    LESS_THAN,
    LESS_THAN_EQUAL,
    GREATER_THAN ,
    GREATER_THAN_EQUAL;
}

Using the above code, you can evaluate the Health based on metrics and plug it into any application, be it SPRINGBOOT or DROPWIZARD or CXF etc

SPRINGBOOT ADAPTER like below can be used which can easily plug into and start evaluating the health based on metrics.

public final class MetricBasedSpringBootAdapter implements HealthIndicator {
    /**
     * Instantiates an object of MetricBasedSpringBootAdapter
     * @param healthEvaluator Reference to an instance of IHealthEvaluator impl
     */
    public MetricBasedSpringBootAdapter(IHealthEvaluator healthEvaluator) {
        Assert.notNull(healthEvaluator, "Underlying HealthEvaluator");
        this.underlyingHealthEvaluator = healthEvaluator;
    }

    /**
     * Return an indication of health.
     * @return the health for Server Instance after consulting different metrics
     */
    @Override
    public Health health() {
        return this.underlyingHealthEvaluator.health();
    }

    private final IHealthEvaluator underlyingHealthEvaluator;
}

HOW IT WORKS IN SPRINGBOOT?

Spring Boot includes a number of built-in endpoints.
One of the endpoints is the health endpoint which provides basic application health information.
By default, the health endpoint is mapped to /health

On invoking this endpoint, Health information is collected from all HealthIndicator beans defined in your
ApplicationContext and based on Health Status returned by these HealthIndicators, Aggregated Health Status is returned.

Spring Boot includes a number of auto-configured HealthIndicators and allows to write our own.

Since we keep track of certain metrics in our applications, we wanted an ability to evaluate Health based on certain
Metrics’ values. For e.g., if Number of Thread exceed ‘n’, Health shall be reported as DOWN

For this purpose, CompositeMetricBasedHealthEvaluator is implemented.
It relies on either MetricReaders or PublicMetrics to get the Metrics’s current values and evaluate the
Health accordingly.

It reports the Individual Health of all configured Health indicator Criterias and reports Health as DOWN If any of
them is Down.

For Unavailable Metric, Health cannot be determined and thus reported as UNKNOWN for that specific metric.

STEPS TO ENABLE IN SPRINGBOOT

* Enable Health Endpoint if not enabled already
* Configure custom endpoint name optionally and other parameters like Caching of results etc
* Configure MetricReader(s) and\or PublicMetric(s)
* Configure the HealthIndicator Metric Criterias
* Instantiate CompositeMetricBasedHealthEvaluator
* Inject the MetricReaders and\or PublicMetrics and Criterias configured above
* Instantiate and Inject MetricBasedSpringBootAdapter into Spring Application Context
* Inject CompositeMetricBasedHealthEvaluator while instantiating
* Disable\Enable Auto-Configured HealthIndicators

That’s all need to be done to enable Health Evaluation using Metrics.

HOW TO ENABLE HEALTH ENDPOINT?

One of the ways is to enable it through Application Configuration YAML file.
In your application.yml file, put the following configuration:

endpoints:
health:
enabled: true
time-to-live: 1000

With the above configuration, health point is enabled and also results will be cached for 1000ms.
Default time-to-live = 1000ms.

HOW TO CONFIGURE HEALTH INDICATOR METRIC CRITERIAS?

1) **VIA APPLICATION CONFIGURATION YAML file**

One of the ways is to configure it in Application Configuration YAML file itself.
In your application.yml file, put the following configuration:

healthIndicator:
criterias:
- metricName: threads
thresholdOrSaturationLevel: 100
operator: LESS_THAN
- metricName: anotherMetricNameGoesHere
thresholdOrSaturationLevel: 100.23
operator: ANY_COMPARISON_OPERATOR(EQUAL, LESS_THAN, LESS_THAN_EQUAL, GREATER_THAN, GREATER_THAN_EQUAL)

With the above configuration, 2 Criterias are defined and **HealthCriteriaList** object gets instantiated using
Configuration Annotation.

Here, Thread Criteria specifies that for Health to be **UP**, number of threads < 100.
If NumberOfThreads >= 100, Health will be reported as **DOWN**

Likewise, more criterias can be defined.

Note that
* **metricName** can contain ‘.’ character as well.
* **thresholdOrSaturationLevel** can have any Valid Number, be it Integer or Decimal Number
* **operator** can be any valid value from ComparisonOperator enum.

2) **Same Configuration can be done through code**

List<HealthCriteria<Number>> criterias = new ArrayList<>();

HealthCriteria<Number> criteria = new HealthCriteria<>();
final String expMetricName = "threads";
criteria.setMetricName(expMetricName);
criteria.setThresholdOrSaturationLevel(100);
criteria.setOperator(ComparisonOperator.LESS_THAN);

criterias.add(criteria);

HOW TO PLUGIN MetricBasedSpringBootAdapter?

MetricBasedSpringBootAdapter implements HealthIndicator interface. Thus, simply injecting it into
Spring Application Context will plugin this component for Health Evaluation.

The below configuration instantiates MetricBasedSpringBootAdapter with MetricReaders only.
Both Parameters, healthCriteriaList and metricReaderList are injected automatically through Spring application
context. This happens due to auto configuration.

@Bean
public MetricBasedSpringBootAdapter metricBasedHealthIndicator(
HealthCriteriaList healthCriteriaList,
List<MetricReader> metricReaderList) {
return new MetricBasedSpringBootAdapter(healthCriteriaList.getCriterias(),
metricReaderList);
}

OR,

@Bean
public MetricBasedSpringBootAdapter metricBasedHealthIndicator(
List<HealthCriteria> healthCriteriaList,
List<MetricReader> metricReaderList) {
return new MetricBasedSpringBootAdapter(healthCriteriaList, metricReaderList);
}

OR,

@Bean
public MetricBasedSpringBootAdapter metricBasedHealthIndicator(
HealthCriteriaList healthCriteriaList,
List<MetricReader> metricReaderList,
List<PublicMetrics> publicMetricsList) {
return new MetricBasedSpringBootAdapter(healthCriteriaList.getCriterias(),
metricReaderList, publicMetricsList);
}

The above configuration can be useful wherein MetricReader is not available to read the Metric but Metric is
available publicly through PublicMetrics interface.
With the above configuration, all parameters are injected automatically by Spring.

Things to Note
* Name of Bean minus Suffix HealthIndicator (metricBased) is what is reported as HealthIndicator Name.
* AutoConfiguration of MetricReaders, PublicMetrics or Configuration could be disabled. If this is the case, either
enable AutoConfiguration or manually instantiate MetricReaders, PublicMetrics etc
* PublicMetrics interface can be expensive depending upon the number of metrics being maintained. Use it only if
Custom MetricReader cannot be written or Metrics are small in number.

Posted in Java, Technical | Tagged , , , , , , , , , , , , | Leave a comment

Data Contracts, XSDs and Redundant List Wrappers – XEW Plugin to rescue


This article talks about problem associated with defining List Complex Types, how can we overcome this problem using XEW Plugin and the benefits.

In Service Oriented Architecture (SOA) or MicroServices Architecture, data is exchanged between different components over the network.

Keeping in mind the INTEROPERABILITY, Data Contracts are created and shared.

Contracts either in the form of WSDL or XSDs etc are mutually agreed between the components to exchange the Structured data among them.

As part of these contracts, you may have a need to send a collection of similar data and for this purpose you may have defined different complexTypes in your xsd.

Consider you want to exchange a list of AirSegments under an Itinerary like:

<OriginDestinationBooked>
    
<AirSegmentBookedList>
        
<AirSegmentBooked>
            
<CarrierCode>AC</CarrierCode>
            
<FlightNumber>12</FlightNumber>
        
</AirSegmentBooked>
        
<AirSegmentBooked>
            
<CarrierCode>AC</CarrierCode>
            
<FlightNumber>13</FlightNumber>
        
</AirSegmentBooked>
        
<AirSegmentBooked>
            
<CarrierCode>AC</CarrierCode>
            
<FlightNumber>189</FlightNumber>
        
</AirSegmentBooked>
    
</AirSegmentBookedList>
</OriginDestinationBooked>

 

To accomplish this, you will define Something like below:

<xs:complexType name="OriginDestinationBookedType">
    <xs:sequence>
 <xs:element name="AirSegmentBookedList" type="SegmentBookedListType"/>
    </xs:sequence>
  </xs:complexType>


<xs:complexType name="SegmentBookedListType">
    <xs:sequence>
 <xs:element maxOccurs="unbounded" name="AirSegmentBooked" type="SegmentBookedType"/>
    </xs:sequence>
  </xs:complexType>


  <xs:complexType name="SegmentBookedType">
    <xs:sequence>
      <xs:element name="CarrierCode" type="CarrierCodeType"/>
      <xs:element name="FlightNumber" type="FlightNumberType"/>
    </xs:sequence>
  </xs:complexType>

This looks good. Looks Good until we generate the Proxy classes out of these Contracts. Give it a try to generate the classes out of these XSDs using Plugins: JAXB-MAVEN, CXF etc.

You will notice that 3 proxy classes get generated.

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "OriginDestinationBookedType", propOrder = {
"segmentBookedList"
})
public class OriginDestinationBookedType {
@XmlElement(name = "SegmentBookedList", required = true)
protected SegmentBookedListType segmentBookedList;
public SegmentBookedListType getSegmentBookedList() {
return segmentBookedList;
}
public void setSegmentBookedList(SegmentBookedListType value) {
this.segmentBookedList = value;
}
}

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "SegmentBookedListType", propOrder = {
"segmentBookeds"
})
public class SegmentBookedListType {
@XmlElement(name = "SegmentBooked", required = true)
protected List segmentBookeds;
public List getSegmentBookeds() {
if (segmentBookeds == null) {
segmentBookeds = new ArrayList();
}
return this.segmentBookeds;
}
}

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "SegmentBookedType", propOrder = {
"carrierCode",
"flightNumber"
})
public class SegmentBookedType {
@XmlElement(name = "CarrierCode", required = true)
protected String carrierCode;
@XmlElement(name = "FlightNumber", required = true)
protected String flightNumber;
}

With the above classes, if you want to get an access to a Segment within an OD, you will have to write:

OriginDestinationBookedType od; // Initialized properly and you have a non-null reference
od.getSegmentBookedList().getSegmentBookeds().get(segIndex);

Bold part above is redundant and not needed for sure. Instead, we want to have:

od.getSegmentBookeds().get(segIndex);

How can we directly get a list of segments under an OD?

Solution
Integrate XEW Plugin into your repository and get it executed during Code generation phase.
Simply,

 

<plugin>
        <groupId>org.jvnet.jaxb2.maven2</groupId>
        <artifactId>maven-jaxb2-plugin</artifactId>
        <version>0.13.1</version>
        <dependencies>
            <dependency>
                <groupId>org.jvnet.jaxb2_commons</groupId>
                <artifactId>jaxb2-basics</artifactId>
                <version>0.6.3</version>
            </dependency>
        </dependencies>
        <executions>
            <execution>
                <id>air-ticket-schema</id>
                <goals>
                      <goal>generate</goal>
                </goals>
                <configuration>
                    <extension>true</extension>
                    <args>
                        <arg>-Xannotate</arg>
                        <arg>-Xxew</arg>
                        <arg>-Xxew:control ${basedir}/src/main/resources/xsds/xewInclusionExclusion.txt</arg>
                    </args>
                    <plugins>
                        <plugin>
                            <groupId>org.jvnet.jaxb2_commons</groupId>
                            <artifactId>jaxb2-basics-annotate</artifactId>
                            <version>1.0.2</version>
                        </plugin>
                        <plugin>
                            <groupId>com.github.jaxb-xew-plugin</groupId>
                            <artifactId>jaxb-xew-plugin</artifactId>
                            <version>1.9</version>
                        </plugin>
                        <plugin>
                            <groupId>com.sun.xml.bind</groupId>
                            <artifactId>jaxb-xjc</artifactId>
                            <version>2.2.11</version>
                        </plugin>
                    </plugins>
                    <vmArgs>
                        <vmArg>-Djavax.xml.accessExternalSchema=all</vmArg>
                    </vmArgs>
                    <schemaDirectory>${basedir}/src/main/resources/xsds</schemaDirectory>
                    <schemaIncludes>
                        <include>yourXSDsHere.xsd</include>
                    </schemaIncludes>
                    <generateDirectory>${basedir}/target/generated-sources</generateDirectory>
                    <bindingDirectory>${basedir}/src/main/resources/xsds</bindingDirectory>
                    <bindingIncludes>
                        <bindingInclude>bindings.xjb</bindingInclude>
                    </bindingIncludes>
                      <removeOldOutput>false</removeOldOutput>
                      <clearOutputDir>false</clearOutputDir>
                      <extension>true</extension>
                </configuration>
              </execution>
        </executions>
    </plugin>

 

With the above configuration, only 2 classes will be generated.

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "OriginDestinationBookedType", propOrder = {
"segmentBookedList"
})
public class OriginDestinationBookedType {
@XmlElement(name = "SegmentBookedList", required = true)
protected List segmentBookedList;
public List getSegmentBookedList() {
return segmentBookedList;
}
public void setSegmentBookedList(List value) {
this.segmentBookedList = value;
}
}

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "SegmentBookedType", propOrder = {
"carrierCode",
"flightNumber"
})
public class SegmentBookedType {
@XmlElement(name = "CarrierCode", required = true)
protected String carrierCode;
@XmlElement(name = "FlightNumber", required = true)
protected String flightNumber;
}

And you are all set. No cursing on XSDs 🙂

ADVANTAGES

    No more List wrapper classes.No more additional clumsy code
    No redundant Null checks
    More readability
    Less Machine Instructions to execute
    Less Memory Footprint of Virtual Functions Table
    More maintainability


Posted in Java | Leave a comment

Decompiling Powershell CmdLet Code


Many of us are involved in writing scripts, be it for development or testing or deployment.
We make use of different scripting languages. One of them is Powershell.
As the name suggests, it’s really powerful.

You can accomplish so many things in Powershell. But what if you already have something developed in .NET and have an Assembly (remember *.dll file) available with you.

Would you like to mimmick everything in Powershell? Or, would you wish if same .NET assembly can be reused?

I fall in latter category wherever possible. 🙂
Yes, You can reuse .NET library.
Aahhh!!! Great!!! Sounds interesting!!!!

Many of us are aware of this and may be, few of us are not.

Why am i writing this?
I was working on Automating or writing a workflow to deploy Virtual Machines (aka, Persistent VM Role) on Microsoft Azure Cloud.
I did it using Powershell script(You can see a lot of support and sample Powershell scripts already available on MS community sites).

That became simple. However, that’s not all for me.

I am hungry :), Hunger to understand things, go till the roots.

I wanted to understand the code working behind the scenes.

Read this post further…

What are Powershell CmdLets?
In actual, Powershell cmdlets are actually exposed through .NET assemblies only. Bunch of assemblies targetting .NET framework execute to get as results which we want.

If you have worked in .NET, you would have came across Attributes. Yeah, that is how Powershell CMDLETS are exposed.

Classes and fields/parameters are attributed with CmdLet and Parameter.
That’s it. Powershell execution engine can now load these types and execute them.

Bottomline is: Cmdlets are classes annotated with Cmdlet attribute.

How to decompile?
Now, we know that it’s actually a .NET Type in .NET assembly which is getting things done. We all know how to decompile .NET assembly.
We may use 3rd Party Tools, some are free while some are not.
This is not a big deal.

However, how do you identify and locate the Assembly containing this specific CMDLET?

You may say that you are not CLR which is responsible for locating, loading and executing the types besides other things.

Then HOW, you may ask.

For this, we’ll again make use of Powershell Command Prompt.

Open up Command Prompt and execute the following command:


$commandDetailsObj=Get-Command nameOfCommand

where,
$commandDetailsObj is how you declare a variable in Powershell,
Get-Command is another Powershell cmdlet, gcm is an alias of this cmdlet,
and,
nameOfCommand is the name of cmdlet which you want to decompile. Say, Add-AzureAccount

The above command will get the details about the cmdlet and store it in $commandDetailsObj variable.
Since cmdlet name can actually be an alias to an actual cmdlet, we keep on doing the below till we get the actual command.


while($commandDetailsObj.CommandType -eq "Alias")
{
$commandDetailsObj = Get-Command ($commandDetailsObj.definition)
}

Next is, we want to get the Type exposed as CMDLET. Issue the following command:


$commandDetailsObj.ImplementingType

The above command after executing will print the Full Qualified Class Name in the console.

Next is, we want to get the Assembly(DLL) name containing the Type exposed as CMDLET. Issue the following command:


$commandDetailsObj.DLL

The above command after executing will print the Full path of the Assembly in the console.

With the above information, we can now open this dll in any of the .NET Decompilation Tool to view the code.

This article doesn’t tell what you want? Need help? Contact me.

Posted in Powershell, Technical | Tagged , , , , , , | Leave a comment

Activation error occured while trying to get instance of type Database, key “” | EntLib


The title of this post may sound a bit strange for those who have not faced this problem but it may sound a Sweet Tune Music 🙂 to those who want to resolve this nasty error in their application.

If you fall into the latter category, you can directly jump to the Solution section though everybody is definitely welcomed to read the entire post.

What is this about?
An error which occurs when using Enterprise Library Data Access Block in instantiating a Database using factory approach.
You may have followed the msdn article to setup DataAccessBlock with the correct code and the configuration in your application but always resulting into the error when you try to instantiate a database object.

Context
Typically, software solutions are multi-layered. One of them being a Data Access Layer, aka DAL, which interacts with the Data Store(s) and performs the CRUD operations on the data in the data store. In this layer, you can either opt for ADO.Net or Enterprise Library Data Access Block to connect to Data Store (database) besides other options.

Since, the post is talking about a specific error resulted in the EntLib, lets assume that we preferred to implement DAL using EntLib Data Access Block.

Problem / Error
Activation error occured while trying to get instance of type Database, key “”
This error occurs on the below code statement, the very first statement to perform the CRUD operation into the DataStore.

Database dataStore = DatabaseFactory.CreateDatabase();

or,

Database dataStore = DatabaseFactory.CreateDatabase("someKey");

Cause
Enterprise library consists of number of classes in different namespaces and assemblies.
Two of them are:

  1. Microsoft.Practices.EnterpriseLibrary.Data
  2. Microsoft.Practices.EnterpriseLibrary.Common

The above code statement is present in the former assembly. After a series of function calls in the same assembly and the latter assembly, a function in the latter assembly tries to load the former assembly using the partial name.

Note: Loading of an assembly using Partial name
This is what leads to the error if the Enterprise libraries assemblies are GACed and not copied locally into the application directory.
Assembly with a partial name won’t be found in the GAC and then the search/probing of an assembly will continue to Local Application Directory or sub-directories with the same name or as per configuration.
Since assembly is not present anywhere else except GAC, assembly loading will fail and leading to this error.

You can see this in action by launching Fusion Log Viewer utility, which comes by default. Command is : “fuslogvw” in case yof could not locate the utility. Type the command in the Visual Studio Command Prompt.
You may need to customize the Log Viewer to log all binding to disks to view every log.

[You can opt to open this assembly into a Reflector or ILSpy and go through each code statement and function call post the above code statement to understand more.]

So, is there a solution or a workaround for the above problem?

Resolution
This problem is solvable. 🙂
Problem can be solved in many ways, you choose what suits you the best.

  1. You can deploy the enterprise library, “Microsoft.Practices.EnterpriseLibrary.Data” locally to the applcation bin directory. [This may lead to maintaining multiple copies of the same assembly]
  2. Another option is to have the below configuration in the application configuration file. This appears a bit clean approach but again the same configuration change has to be done at multiple places if they are using this library

    <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
    <qualifyAssembly partialName="Microsoft.Practices.EnterpriseLibrary.Data" fullName="Microsoft.Practices.EnterpriseLibrary.Data, Version=5.0.414.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
    </assemblyBinding>
    </runtime>

    Thanks to Mr. Philippe for this 2nd solution posted @ CodePlex.
Posted in .NET | Tagged , , , , , | 8 Comments

DeterminantOfMatrix


4. Determinant of a 2D matrix

Problem:

Given a 2D matrix, Determine it’s Determinant.

Solution:

This implementation is done using C#.NET. Rectangular Matrix is declared using int[,] syntax.

public static long EvaluateDeterminant(int[,] matrix)
{
long determinant = 0;

if (matrix == null || matrix.GetUpperBound(0) != matrix.GetUpperBound(1))
{
Console.WriteLine("Non-square matrix can't have a determinant");

return determinant;
}

int row_UB = matrix.GetUpperBound(0);

determinant = Determinant(matrix, row_UB + 1);

return determinant;
}

private static long Determinant(int[,] matrix, int size)
{
long determinant = 0;

if (size == 1) // 1x1 MAtrix
{
determinant = matrix[0, 0];
}
else if (size == 2) // 2x2 MAtrix
{
determinant = matrix[0, 0] * matrix[1, 1] - matrix[0, 1] * matrix[1, 0]; // can hash this multiplication
}
else
{
int multiplier = 1;

for (int i = 0; i < size; i++)
{
multiplier = (i % 2 == 0) ? 1 : -1;

determinant += multiplier * matrix[0, i] * Determinant(GetMinor(matrix, size, 0, i), size - 1);
}
}

return determinant;
}

/// <summary>
/// Gets the Minor of a Square Matrix
/// </summary>
/// <param name="matrix"></param>
/// <param name="size"></param>
/// <param name="rowIndex"></param>
/// <param name="colIndex"></param>
/// <returns></returns>
/// <remarks>
/// If function has to be Public, Certain checks on rowIndex, ColIndex should be made and size need not to be passed
/// </remarks>

private static int[,] GetMinor(int[,] matrix, int size, int rowIndex, int colIndex)
{
int minorSize = size - 1;
int[,] minor = new int[minorSize, minorSize];

for (int i = 0; i < rowIndex; i++)
{
for (int j = 0; j < colIndex; j++)
{
minor[i, j] = matrix[i, j];
}
}

for (int i = rowIndex + 1; i < size; i++)
{
for (int j = 0; j < colIndex; j++)
{
minor[i - 1, j] = matrix[i, j];
}
}

for (int i = 0; i < rowIndex; i++)
{
for (int j = colIndex + 1; j < size; j++)
{
minor[i, j - 1] = matrix[i, j];
}
}

for (int i = rowIndex + 1; i < size; i++)
{
for (int j = colIndex + 1; j < size; j++)
{
minor[i - 1, j - 1] = matrix[i, j];
}
}

return minor;
}

Posted in Algorithms | Tagged , , , , | Leave a comment

Infix to Prefix Conversion


3. Convert Infix Expression to Prefix Expression

Problem:

Given an infix expression, output the expression in Prefix (Polish Notation) form.

For e.g.

Solution:

This implementation is done using C#.NET.

ALGORITHM:

This algorithm maintains two stacks. 1st stack for all operators and 2nd stack to store the operands.

1)  Validate the Infix Expression for correctness

a) ‘(‘ and ‘)’ are in pairs

b) Operator is in between 2 operands (Binary operators are considered only)

2)  If Infix Expression is Valid then

a)  Push ‘(‘ to Operator Stack and Append ‘)’ to Infix Expression

b)  Scan each character in the Infix expression

i)  If scanned character is ‘(‘

Then

Push it to Operator Stack

ii)  Else If scanned character is ‘)’

Then

(a)  Repeat the below steps until ‘(‘ is popped out from Operator Stack

(b)  Pop from Operator Stack into OPERATOR

(c)  If OPERATOR  != ‘(‘

(i)  Pop twice from Operand Stack into OPERAND2 and OPERAND1

(ii)  Push “OPERATOR OPERAND1 OPERAND2” in Operand Stack

iii)  Else If scanned character is an Operator

Then

(a) Repeat the below steps until Operator having low precedence than  Scanned character is popped out from Operator Stack

(b) Pop from Operator Stack into OPERATOR

(c)  If OPERATOR  has Higher or Equal Precedence than scanned character

(i)  Pop twice from Operand Stack into OPERAND2 and OPERAND1

(ii)  Push “OPERATOR OPERAND1 OPERAND2” in Operand Stack

(d) Otherwise,

(i) Push the last Popped operator back in Operator Stack

(ii) Push the scanned character in Operator Stack

iv) Otherwise, Scanned character is an operand. Thus, Push it in Operand Stack

c) Pop from Operand Stack which is the final expression and return;


public void ConvertInfixToPrefix(string infixExpression)
{

try
{

ValidateInfixExpression(ref infixExpression);

}
catch (Exception ex)
{

Console.WriteLine("Invalid infix expression. Error Details:{0}", ex.Message);
return null;

}

Stack<char> operatorStack = new Stack<char>();
Stack<string> operandStack = new Stack<string>();

operatorStack.Push('(');
infixExpression += ')';

foreach (char ch in infixExpression)
{

if (ch == '(')
{

operatorStack.Push(ch);

}
else if (ch == ')')
{

// Pop from operator Stack until '(' is encountered
char poppedOperator = operatorStack.Pop();

while (poppedOperator != '(')
{

operandStack.Push(PrefixExpressionBuilder(operandStack, poppedOperator));

poppedOperator = operatorStack.Pop();

}

}
else if (IsOperator(ch))
{

// Pop all operators from Operator Stack which have same or higher precedence
char poppedOperator = operatorStack.Pop();

bool sameOrHighPrecedence = CheckSameOrHighPrecedence(poppedOperator, ch);

while (sameOrHighPrecedence)
{

operandStack.Push(PrefixExpressionBuilder(operandStack, poppedOperator));

poppedOperator = operatorStack.Pop();

sameOrHighPrecedence = CheckSameOrHighPrecedence(poppedOperator, ch);

}

operatorStack.Push(poppedOperator);

operatorStack.Push(ch);

}
else
{

operandStack.Push(ch.ToString());

}

}

return operandStack.Pop();

}

/// <summary>
/// Validates the infix expression for correctness
/// </summary>
/// <param name="expression">Infix expression to be validated</param>
/// <returns>True if expression is valid</returns>
private static void ValidateInfixExpression(ref string expression)
{

expression = expression.Replace(" ", string.Empty);
// Rule 1: '(' and ')' pair
// Rule 2: Every two operands must have one operator in between

}

/// <summary>
/// Checks if character is a listed operator or not
/// </summary>
/// <param name="character">Charaxter to be tested</param>
/// <returns>False if not otherwise True</returns>
private static bool IsOperator(char character)
{

if ((character == '+') || (character == '-') || (character == '*') || (character == '/'))
{
  return true;
}

return false;

}

/// <summary>
/// Checks if popped operator has same or higher precedence than Current operator
/// </summary>
/// <param name="elementToTest">Popped operator</param>
/// <param name="checkAgainst">Current operator in the expression</param>
/// <returns>True if equal or higher precedence</returns>
private static bool CheckSameOrHighPrecedence(char elementToTest, char checkAgainst)
{

bool flag = false;

switch (elementToTest)
{

case '/':
case  '*':

flag = true;
break;

case '+':
case '-':

if ((checkAgainst == '+') || (checkAgainst == '-'))
{

flag = true;

}
break;

default: // for any other popped element

flag = false;
break;

}

return flag;

}

private static string PrefixExpressionBuilder(Stack<string> operandStack, char operatorChar)
{

string operand2 = operandStack.Pop();
string operand1 = operandStack.Pop();

string infixExpression = string.Format("{0}{1}{2}", operatorChar, operand1, operand2);

return infixExpression;

}

Posted in Algorithms | Tagged , , , , , , , , | 11 Comments

Is Binary Tree a Binary Search Tree?


2. Is tree BinarySearchTree?

Problem:

Given a binary tree, determine if it is a Binary Search Tree (BST) or not?

Definition:

What is BST?

BST is a binary tree in which value of root is always greater than the value of every node on it’s left and is less than or equal to the value of every node on it’s right.

Solution:

This implementation is done using C#.NET.

class BinaryTree
{
public BinaryTreeNode Root { get; set; }

public bool IsBinarySearchTree()
{
Console.WriteLine("Checking if Tree is BST or not:");

if (this.Root != null)
{
int value = 0;

return this.Check(this.Root, ref value);
}

return true;
}

private bool Check(BinaryTreeNode currentNode, ref int lastNodeValue)
{
bool isTreeBST = false, leftTreePresent, rightTreePrsent ;

leftTreePresent = currentNode.LeftTree == null ? false : true;
rightTreePrsent = currentNode.RightTree == null ? false : true;

if (leftTreePresent)
{
isTreeBST = this.Check(currentNode.LeftTree, ref lastNodeValue);
}
else
{
isTreeBST = true;
}

if (isTreeBST && currentNode.Info > lastNodeValue)
{
Console.WriteLine("Processing Node With Value:{0}", currentNode.Info);

lastNodeValue = currentNode.Info;

isTreeBST = true;
}
else
{
isTreeBST = false;
}

if (isTreeBST && rightTreePrsent)
{
isTreeBST = this.Check(currentNode.RightTree, ref lastNodeValue);
}

return isTreeBST;
}
}

class BinaryTreeNode
{
public BinaryTreeNode LeftTree { get; set; }
public BinaryTreeNode RightTree { get; set; }
public int Info { get; set; }
}

Problem with the above code is that if a tree has Duplicate Values, it will fail.

The approach could be then to pass the Range in terms of Minimum and the Maximum Value of a particular node. Since we are traversing down from Root and knowing the min and max value of a root, we can appropriately limit the range and pass on to Left and Right Trees.


private bool Check(BinaryTreeNode node, int min, int max)
{
if (node == null)
return true;

if (node.Info < min || node.Info > max)
return false;
else
{
return this.Check(node.LeftTree, min, node.Info) && this.Check(node.RightTree, node.Info + 1, max);
}
}

Posted in Algorithms | Tagged , , , , , , , , , , , , | Leave a comment