Back in January I announced my departure from Oracle and my arrival to Pivotal. One of my first goals was to help build a robust distributed runtime platform for Spring XD. Most (if not all) Java developers are familiar with the Spring framework, but Spring XD is not as well known.

Spring XD is a system used for defining streams and/or batch processing of data. Although developers may introduce their own modules into the system, writing code is not a prerequisite for use. This by itself makes Spring XD different from Spring framework. Spring XD is a standalone distributed application. Streams and jobs are created via an interactive shell.

What is a stream? The best way to demonstrate is via an example using the UNIX shell. Consider the following:

tail -f /tmp/log.txt | grep ERROR

The tail command will continuously display the file. The | will pipe the output of tail to grep, which will filter out all lines that don’t include the string ERROR.

This command can be translated to the Spring XD shell:

stream create --name error-filter --definition "tail --name=/tmp/log.txt | filter --expression=payload.contains('ERROR') | log" --deploy


Why is this useful? For this example Spring XD is not very practical. However, imagine building a system to grep for errors in a 1000 node farm of servers. Perhaps these messages should be persisted in HDFS for further analysis in a Hadoop cluster. The creation and management of these types of distributed streams is where Spring XD shines. In addition to streams, Spring XD can handle the scheduling and execution of jobs across a cluster.

As in any distributed system, it is desirable to scale the system up or down to meet demand. Single points of failure should be eliminated to the extent possible. In order to meet these challenges, Spring XD has adopted ZooKeeper as the basis for distributed runtime configuration and coordination.

Apache ZooKeeper is a “centralized service for maintaining configuration information, naming, providing distributed synchronization…[for] distributed applications.” It is used by many projects, including Kafka, Storm and HBase. One of the bits of core functionality that ZooKeeper provides is the ability to emit notifications when data is updated. Spring XD relies on this functionality for stream and job deployments. Much of this functionality, in addition to ZooKeeper data caching, is made much easier via the use of Apache Curator.

A container process in Spring XD has the ability to host modules for streams or jobs. An admin process can accept HTTP requests to manage streams and jobs. It is also responsible for deployment coordination. Multiple admin processes may exist in a Spring XD cluster. One of these processes is designated with the “supervisor” role. Curator’s leader election recipe is used to select which admin process is the supervisor. This process is the one responsible for deploying stream and job modules to containers.

Commands entered into the shell are sent to an admin server via HTTP/REST. Most commands will consist of the creation and deployment of streams and jobs. Stream and job definitions are stored in ZooKeeper by the admin server. When streams and jobs are deployed, a node is written in ZooKeeper by the admin process that received the REST request for deployment. The node triggers the admin supervisor (which may or may not be the same admin process that received the initial REST request) to start the deployment process. This process consists of selecting which containers will host the various modules that comprise the stream/job and writing to ZooKeeper a request for module deployment. The respective containers will deploy the modules, thus establishing the stream/job.

When a container joins or leaves the cluster, the supervisor will determine if any action is required in terms of stream/job module deployment. In most cases, if a departing container was hosting a module, the supervisor will select another container to host that module. Should the system be without a supervisor for a period of time, the new supervisor will query the state of the system upon startup and take any actions required.

See the reference documentation for more details on how Spring XD uses ZooKeeper.

New on the Coherence blog is an article on best practices for near caching written by yours truly. This article went through several reviews by PM and engineering (both internal and field engineers) and is a good summary of our experiences with near caching. The key takeaway is to understand the near cache invalidation strategies and to explicitly configure the one best suited to the application needs.

See the full Near Cache Best Practices article on the official Oracle Coherence blog.

• REST support in the proxy. In addition to Java, C# and C++, now you can access the grid with any language that has a REST API.
• POF enhancements, including annotations and the ability to eliminate Java key classes (for pure C# and C++ applications)
• Query Explain Plan. If you’ve ever had a problem with a production application missing indexes (because you won’t notice it in development and you may not notice it during testing) you’ll be interested in this feature. It analyzes queries (filter based or CohQL) and indicates where indexes are used (and not used.)

We’ve been adding screencasts to the Coherence YouTube channel going over these features – in most cases the screencasts are delivered by the architect/developer responsible for the feature.

We also just wrapped up JavaOne and Oracle Open World a few weeks ago. I had the privilege of co-presenting the Live Object Pattern with Brian Oliver. The description and PDF can be downloaded from the JavaOne site.

If you didn’t get to see the talk, you’re in luck if you’re in New York or London. Noah Arliss, who worked with Brian on the Live Objects concept and the abstract of the JavaOne talk, will be presenting on this topic at the New York SIG on October 28th and the London SIG on November 4th.

All of the SIGs this fall (including the Bay Area SIG on November 3rd) will cover the new features in Coherence 3.7.1. Come out and join us!

Note that all images below are thumbnails; click on the images to expand.

You have the option of downloading Eclipse with the OEPE plugins pre-installed, but I already had a copy of Eclipse Helios 3.6; therefore I went for the plugin install. The plugin install is straight forward, similar to any other Eclipse plugin. These instructions are lifted straight from the doc:

1. Select Help > Install New Software.
2. Click Add to add a new update site.
3. In the Add Repository dialog, enter the location as http://download.oracle.com/otn_software/oepe/helios, and then click OK.
4. Select Oracle Enterprise Pack for Eclipse, verify that all of the subcomponents are selected, and then click Next.
5. Confirm information presented in the Install Details, and then click Finish.

Once the install is complete and you’ve restarted Eclipse, the next step is to install Coherence as a User Library. I’ve got the latest 3.6 bits from our source code repository, so I’ll install that as a library. Note that the plugin also supports Coherence 3.5.2.

Now, let’s create a new Java project. As part of the project creation, add Coherence as a project library.

After the project is created, we’ll have to enable Facets to use the Coherence plugin. Bring up the project properties window and search for “Facets” in the search field.

Once Facets are enabled, select Oracle Coherence.

Upon selection, a link indicating that further configuration is required will appear. Click on the link. Select the “Oracle Coherence 3.6″ library. Note how it provides the option to generate configuration files. Let’s leave all of the checkboxes selected.

Now we are ready to start a cache server. Select File | Run Configurations to bring up the Run Configurations dialog. First select “Oracle Coherence” under the list of run configurations. Next, select the “New” button on the upper left portion of the dialog to create a new run configuration.

Under the “Main” tab, enter com.tangosol.net.DefaultCacheServer as the main class. Of course you are free to create configurations with your own classes; however this example will focus on starting up a cache server.

Note the presence of a “Coherence” tab. This tab allows for operational configuration (items typically found in tangosol-coherence-override.xml) such as the cache configuration file name, multicast address configuration, management/JMX, and so on. Here I decided to leave all of the defaults as is.

After clicking on “Run”, here’s what I get:

We can see that the node started up and formed a cluster, but there are no services listed. This is because the OEPE plugin generated a cache configuration file that defaults to all caches being local. Next, let’s examine the cache configuration file (located under src and add a distributed/partitioned cache to the configuration.

One of the nice features the plugin provides is pre-configured auto complete for Coherence configuration files via the DTD.

Here’s the cache configuration file I used:

<!--?xml version="1.0"?-->
 
      *
      partitioned
 
      partitioned
 
      true

With the modified cache configuration, we now see the partitioned cache service start up:

I can see the Coherence plugin for OEPE being quite useful for Coherence developers on Eclipse not only for quickly starting up a Coherence project (since config files are generated) but also for enabling configuration validation out of the box.

What is the relationship between a partition, cache, and a backing map? 

To answer this, first we should go through some definitions:

Cache: An object that maps keys to values. In Coherence, caches are referred to by name (thus the interface NamedCache.) Caches are also typically clustered, i.e. they can be accessed and modified from any member of a Coherence cluster.

Partition: A unit of transfer for a partitioned/distributed cache. (In Coherence terminology, partitioned and distributed are used interchangeably, with preference towards the former.)

Backing Map: Data structure used by a storage node to hold contents of a cache. For partitioned caches, this data is in binary (serialized) form.

I’ll add one more term for completeness:

Service: Set of threads dedicated to handling requests; a cache service handles requests such as put, get, etc. The DistributedCache service is present on all members of the cluster that read/write/store cache data, even if the node is storage disabled.

Now let’s put these concepts together to see how a clustered partitioned cache works.

First we’ll start with a backing map. It is a straight forward data structure, usually an instance of LocalCache:

The contents of this map are managed by a partitioned cache service. In a typical thread dump, this would be the “DistributedCache” thread:

"DistributedCache" daemon prio=5 tid=101a91800 nid=0x118445000 in Object.wait() [118444000]

A single cache service can manage multiple backing maps. This is the default behavior if multiple services are not specified in the cache configuration file. (Multiple partitioned cache services are beyond the scope of this post; this topic will be addressed in a future posting.)

So how do partitions come into play? Each backing map is logically split into partitions:

This splitting is used for transferring data between storage enabled members. Here are log file snippets of data transfer between two nodes:

Member 1:

(thread=DistributedCache, member=1): 3> Transferring 128 out of 257 primary partitions to member 2 requesting 128

Member 2:

(thread=DistributedCache, member=2): Asking member 1 for 128 primary partitions

As members are added to the cluster, the partitions are split up amongst the members. Note that this includes backup copies of each partition.

As of Coherence 3.5, it is possible to configure backing maps to physically split partitions into their own respective backing maps. This configuration may be desirable for backing maps that are stored off heap (i.e. NIO direct memory or disk) as it will make data transfer more efficient.

An Introduction to Data Grids for Database developers (GatorJUG June 23rd)

This talk will introduce the concept of data grids to developers that have experience with Java EE and relational databases such as Oracle. The programming model will be explored (including caching patterns and similarities to NoSQL) as well as the performance & scalability improvements a data grid offers.

From IDE to Data Center – What Every Developer Should Know About Deploying Distributed Systems to Production (Orlando JUG June 24th)

Taking a distributed system from development into a working production environment is a challenge that many developers take for granted. This talk will explore these challenges, especially scenarios that are not typically seen in a development setting.

I’m especially excited about the OJUG talk as I think it will cover many topics of interest to Developers and OPS guys. It is a set of general guidelines that came about from seeing dozens of Coherence applications in production. We will cover such things as:

• What to look for when using vmstat
• Must-have production level JVM settings/flags
• Developer Do’s and Dont’s
• Crash course on thread dumps and heap dumps

We will also be giving away a copy of Oracle Coherence 3.5 at each event! If you are coming please follow the links to the events above and RSVP (you need to be a member of CodeTown to sign up, but registration is free and painless.)

On occasion I am asked about best practices for creating classes to be used as keys in Coherence. This usually comes about due to unexpected behavior that can be explained by incorrect key implementations.

First and foremost, equals and hashCode need to be implemented correctly for any type used as a key. I won’t describe how to do this – instead I’ll defer to Josh Bloch who has written the definitive guide on this topic.

There is an additional requirement that needs to be addressed. All serializable (non transient) fields in the key class must be used in the equals implementation. To understand this requirement, let’s explore how Coherence works behind the scenes.

First, let’s try the following experiment:

public class Key
        implements Serializable
    {
    public Key(int id, String zip)
        {
        m_id = id;
        m_zip = zip;
        }
 
    //...
 
    @Override
    public boolean equals(Object o)
        {
        // print stack trace
        new Throwable("equals debug").printStackTrace();
        if (this == o)
            {
            return true;
            }
        if (o == null || getClass() != o.getClass())
            {
            return false;
            }
 
        Key key = (Key) o;
 
        if (m_id != key.m_id)
            {
            return false;
            }
        if (m_zip != null ? !m_zip.equals(key.m_zip) : key.m_zip != null)
            {
            return false;
            }
        return true;
        }
 
    @Override
    public int hashCode()
        {
        // print stack trace
        new Throwable("hashCode debug").printStackTrace();        
        int result = m_id;
        result = 31 * result + (m_zip != null ? m_zip.hashCode() : 0);
        return result;
        }
 
    private int m_id;
    private String m_zip;
    }

This key prints out stack traces in equals and hashCode. Now use this key with a HashMap:

public static void testKey(Map m)
    {
    Key key = new Key(1, "12345");
 
    m.put(key, "value");
    m.get(key);
    }
 
//...
 
testKey(new HashMap());

Output is as follows:

java.lang.Throwable: hashCode debug
	at oracle.coherence.idedc.Key.hashCode(Key.java:60)
	at java.util.HashMap.put(HashMap.java:372)
	at oracle.coherence.idedc.KeyTest.testKey(KeyTest.java:46)
	at oracle.coherence.idedc.KeyTest.testKey(KeyTest.java:52)
	at oracle.coherence.idedc.KeyTest.main(KeyTest.java:18)
java.lang.Throwable: hashCode debug
	at oracle.coherence.idedc.Key.hashCode(Key.java:60)
	at java.util.HashMap.get(HashMap.java:300)
	at oracle.coherence.idedc.KeyTest.testKey(KeyTest.java:47)
	at oracle.coherence.idedc.KeyTest.testKey(KeyTest.java:52)
	at oracle.coherence.idedc.KeyTest.main(KeyTest.java:18)

Try it again with a partitioned cache this time:

testKey(CacheFactory.getCache("dist-test"));

Note the absence of stack traces this time. Does this mean Coherence is not using the key’s equals and hashCode? The short answer (for now) is yes. Here is the flow of events that occur when executing a put with a partitioned cache:

1. Invoke NamedCache.put
2. Key and value are serialized
3. Hash is executed on serialized key to determine which partition the key belongs to
4. Key and value are transferred to the storage node (likely over the network)
5. Cache entry is placed into backing map in binary form

Note that objects are not deserialized before placement into the backing map – objects are stored in their serialized binary format. As a result, this means that two keys that are equal to each other in object form must be equal to each other in binary form so that the keys can be later be used to retrieve entries from the backing map. The most common way to violate this principle is to exclude non transient fields from equals. For example:

public class BrokenKey
        implements Serializable
    {
    public BrokenKey(int id, String zip)
        {
        m_id = id;
        m_zip = zip;
        }
 
    //...
 
    @Override
    public boolean equals(Object o)
        {
        if (this == o)
            {
            return true;
            }
        if (o == null || getClass() != o.getClass())
            {
            return false;
            }
 
        BrokenKey brokenKey = (BrokenKey) o;
 
        if (m_id != brokenKey.m_id)
            {
            return false;
            }
 
        return true;
        }
 
    @Override
    public int hashCode()
        {
        int result = m_id;
        result = 31 * result;
        return result;
        }
    }

Note this key has two fields (id and zip) but it only uses id in the equals/hashCode implementation. I have the following method to test this key:

public static void testBrokenKey(Map m)
    {
    BrokenKey keyPut = new BrokenKey(1, "11111");
    BrokenKey keyGet = new BrokenKey(1, "22222");
 
    m.get(keyPut);
    m.put(keyPut, "value");
 
    System.out.println(m.get(keyPut));
    System.out.println(m.get(keyGet));
    }

Output using HashMap:

value
value

Output using partitioned cache:

value
null

This makes sense, since keyPut and keyGet will serialize to different binaries. However, things get really interesting when combining partitioned cache with a near cache. Running the example using a near cache gives the following results:

value
value

What happened? In this case, the first get resulted in a near cache miss, resulting in a read through to the backing partitioned cache. The second get resulted in a near cache hit because the object’s equals/hashCode was used (since near caches store data in object form.)

In addition to equals/hashCode, keep the following in mind:

• Keys should be immutable. Modifying a key while it is in a map generally isn’t a good idea, and it certainly won’t work in a distributed/partitioned cache.
• Key should be as small as possible. Many operations performed by Coherence assume that keys are very light weight (such as the key based listeners that are used for near cache invalidation.)
• Built in types (String, Integer, Long, etc) fit all of this criteria. If possible, consider using one of these existing classes.)

In Boston however I found the environment much more conducive to running, given that there are sidewalks and bike paths, not to mention it feels like half of the people here run. Also, the weather in the fall is very comfortable to run in (which probably explains why the NYC marathon is in November.)

Winter running, on the other hand, came with its own set of challenges. During those months I learned some lessons that I’m passing along.

At first I tried running with a thick cotton sweater. This turned out to be a bad idea, because the sweater (being a warm sweater as advertised) prevented heat from escaping my body. This, paradoxically, causes you to freeze your ass off since you now have a pool of sweat on your skin. The best practice is to wear material that wicks sweat away, including the top layer. Even on the coldest of days (my personal low was 15F) I would get comfortable after about a mile.

The biggest challenge for me was not the air temperature (at least after warming up), it was ice. On icy days my runs were much slower. On really icy days I would just skip the run. I know some hard core runners that would throw on spikes or yak tracks, but I didn’t take it to that level. A related challenge is when it isn’t quite cold enough to freeze deeper pools of water. I had the distinctly unpleasant experience of stepping into a 1 inch puddle on the bike path after wrongly assuming it was frozen over. That day my run was definitely shorter than it would have been otherwise.

For the given temperatures, I really was wearing very thin layers of clothing. This means that I had to keep moving in order to stay warm. I never slowed down to a walk because that would have been far more unpleasant than dealing with the fatigue. Running on the road in general has that advantage over the treadmill – with the treadmill you can quit anytime, whereas on the road you have to make it back home sometime, so you may as well run the distance.

Due to things being really busy lately my mileage has been decreasing. I’m hoping to turn this around soon, especially since the warm spring weather is bringing out so many runners.

So how has the running affected my fitness? It used to be that my heart and lungs were ready to break out of my chest after chasing down a bus. After taking up running, I was not only able to perform this exercise without loosing breath, but I was able to do it while carrying my 40 lbs son!

As they say, when it rains it pours. I just came back from the New York SIG where I talked about real world challenges in customer deployments and passed along lessons that we learned in solving these problems. After leaving Tokyo, I will be joining Cameron and Noah in Toronto for the inaugural SIG where I will deliver the same presentation I gave in NY. Cameron will talk about the past and future of Coherence, and Noah will give an update on the latest innovations in the Coherence Incubator project.

Has everyone received their copy of the Coherence book yet? One of my favorite parts of the book is the very beginning where Aleks describes what it takes to build scalable applications. In fact a good portion of the first chapter doesn’t even mention Coherence; it just talks about aspects of scalability that developers and architects of high scale systems should be familiar with. A copy of this book was raffled at the NY SIG, and more copies will be given away at the Bay SIG and Toronto SIG.

New Coherence Bloggers

We have two new bloggers sharing their experiences with Coherence!

The first is by Oracle JDBC expert Pas Apicella who recently took on Coherence. Upon his introduction to Coherence he immediately proceeded to create a CacheStore example using PL/SQL, followed by an example of using the Oracle JDBC Data Change Notification mechanism to push updates from the database to a cache.

Additionally we have Coherence architect Andy Nguyen debuting with a detailed description of a sophisticated distributed bulk loading technique he’s employed on several customer projects.

Coherence Book in March

After many months of blood, sweat, and blisters from too much typing, Aleksandar Seovic has completed the highly anticipated Coherence book published by Packt. Having worked closely with Aleks on reviews and contributions, I believe this book will be a terrific resource for developers and architects that need to write scalable applications. Both experienced users of Coherence and new users will find relevant and useful content. Aleks was recently interviewed by Cameron Purdy about the book which can be downloaded as an MP3.

User Group Meetings

The UK SIG in London is the last Coherence user group meeting for the winter, it is coming up on February 26th. The spring events are currently being planned; stay tuned for details!

Also coming up on February 24th is the first Boston SUG meeting of the year. Although the topic won’t be Coherence this time, it will be of interest for developers and architects interested in scalable systems. We’ll be meeting up for drinks and snacks at Bertucci’s afterwards. And I’ll be there if anyone wants to chat about Coherence or any other topic!