Close to two years ago I had my first look at Terracotta when researching available technology for sharing state and scaling Aloha. First impressions were good. Terracotta delivered what promised: easy integration with an existing single node application and virtual shared heap available across whole cluster. However it has much more to offer. I have been looking again recently in the distributed cache and data grids area trying to assess options for a project I am working on currently. Below is a digest from my web-crawling of publicly available data on the topic. In each of the paragraphs I will try to put some light on a specific aspect that someone would like to know when considering Terracotta.
Resilience
Terracotta clusters offer simple and scalable HA implementations based on the Terracotta server array. Terracotta server arrays can vary from a basic two-node array consisting of an active server instance backed by a hot standby (for instant failover), to a multi-node array with many active server instances providing scale and deep failover coverage. Terracotta give high availability of the data by not only replicating it across servers for redundancy, but also by providing a simultaneous permanent storage of all current shared (in-memory) data to disk – available to all server instances (no loss of application state). Temporarily disconnected server instances and clients are automatically reconnected, hot standbys restored without operator intervention. There is no single point of failure and bottleneck in the Terracotta architecture.
Failover
Terracotta server array provides instant failover and continuous uptime and services using a hot standby or multiple active servers. Multiple Terracotta servers can be clustered together in active-active or active-passive configurations, helping to further scale Java applications and to provide disaster recovery.
DB Persistence
Terracotta helps to protect database in a scale out architecture. There is a problem with usual approach with a cache in front of the database. The performance of the cache is degradated when the number of nodes increases. Hibernate offers a second-level cache option, which can improve the performance of the Hibernate application by eliminating the need to fetch often used data objects from the database, instead storing, or caching, that data in process with your application. Without any mechanism to provide replication of the underlying cache, data written in one node can be stale on another node. This problem is not solved by Hibernate, rather it is up to the underlying cache to provide a replication scheme to keep multiple copies synchronized. Terracotta provides a consistent mechanism to keep any cache written in Java synchronized, and therefore provides this support for caches such as EHCache. Advantages: Performance — Reading and writing to replicated data with Terracotta can be 10x - 100x faster than other solutions, Cluster-wide load — Replication of data eliminates the need for more than one node to load data from the database. In a scaled-out application, this can reduce database load by a significant factor due to the fact that only one node needs to request any piece of data for a cache miss and all subsequent requests on other nodes will result in cache hits.
Distributed caching implementation
Terracotta uses concept of the Network-Attached Memory. It clusters JVMs directly underneath applications providing distributed, shared heap. The implementation doesn’t rely on serialization of whole objects, rather on instrumentation of the byte code and injecting hooks in there which allow synchronizing of deltas in the object state. State changes in a Client JVM are published to Terracotta Server Array and all other Client JVMs are seamlessly updated. Other Terracotta’s features apart from shared heap are: distributed locking, distributed data structures – (HashMap, ArrayList, LinkedBlockingQueue, any POJO), guaranteed coherent clustered objects. Objects do not have to implement Serializable to work with Terracotta.
OS/Platform restrictions
Terracotta is a 100% Java solution.
Certified platforms: Solaris 10, Solaris 9, RedHat ES4, SUSE ES10.1 (32/64bit), Windows Server 2003 R2, WindowsXP (dev only)
Server JDKs: Sun Hotspot 1.5.0_16, Sun Hotspot 1.6.0_07
Client JDKs: Sun Hotspot 1.4.2_18, Sun Hotspot 1.5.0_16, Sun Hotspot 1.6.0_07
Containers: Tomcat 6.0.18 / 5.5.26 / 5.0.28, Apache Geronimo 1.1.1, BEA WebLogic 8.1 sp6, BEA WebLogic 9.2.MP2, JBoss 3.2.8.sp1 / 4.0.5, Jetty 6.1.11, WebSphere CE 1.0.1.2 / AS 6.1.0.7, Glassfish V1 ur1-p01-b02, Glassfish-v2 ur2-b04
Ease of implementation
Big advantage of Terracotta. A clustered application looks no different than a standard Java applications. In theory a correct multi-threaded application can be clustered just by providing a simple xml config defining roots of application state hierarchy which one wants to share across multiple JVMs. Distributed locking and thread coordination uses usual language artefacts like ’synchronized’, ‘wait’/'notify’ and java.util.concurrent support. Pre-existing Terracotta Integration Modules (TIMs): purpose-built, tested, and tuned to implement a particular clustered architecture pattern or integrate with a particular third-party technology, for example use TIMs for Spring, Hibernate, Quartz, Compass, Lucene, Guice, Camel, Struts, Joda time and more to seamlessly add clustering to an application. Last but not least: Terracotta is an open source product with an active community.
Hardware requirements
Client:
Memory Sizing - Java Heap: each client JVM should do work on data that fits in its local heap
Memory Sizing - RAM: enough RAM to accommodate the heap of the Terracotta client JVM
Number of client nodes: determined by the total size of the active clustered data set and total required throughput. Entire data set will fit in the sum of the total heap available for clustered object storage across all Terracotta client JVMs.
Server:
Memory Sizing - Java Heap: Ideally, all of the clustered object data should fit into its heap, note: allow for garbaged objects, not recommended heap > 6/8GB
Memory Sizing - RAM: enough RAM to accommodate the heap of the Terracotta server JVM. In addition, there should be enough RAM headroom to keep the blocks of the Sleepycat data store files in the host operating system’s filesystem cache for fast access.
CPU: dedicated server, capacity planning needed. Note: The more garbage your application creates, the more CPU resources will be consumed by the Terracotta Distributed Garbage Collector. Likewise, the higher the object read or write rate between Terracotta client nodes and a Terracotta server instance, the more CPU resources that server instance consumes.
Number of server nodes: min 2 for HA and failover
Network
Gigabit full duplex between server and clients, and server array, low latency network, redundant NICs and redundant switches with automatic failover
Performance
According to Terracotta: it is faster than other distributed cache, data grid solutions. It is due to the nature of synchronizing only deltas of object state. Terracotta scales much better than i.e. Coherence even with big transaction payloads. At small scale it has throughput 2x Oracle’s at all transaction sizes. At high scale (8 nodes) throughput was as much as 10x higher than Oracle. However independent tests of a specific use case scenario should be performed to back up the results.
Support package
There are Enterprise Subscription packages: production support GOLD 7X24 | SILVER 5X12 with issue escalation privileges and procedures. Also consulting and training services. Pricing individually discussed.
Code stability
Current version 2.7.1. Code is mature as the company started to work on Terraccota in 2003. Two years ago project has been opensourced. Licensed as Terracotta Public License, based on the Mozilla Public License. Free version requires displaying the words “Powered by Terracotta” in user interfaces. Commercial Version removes this requirement. Typical price $7,000 per node. A single Terracotta server can handle up to 150 nodes, but on heavy Java applications, such as those used in online reservation systems, that gets reduced to somewhere between 15 to 20 nodes per Terracotta server.
Partitioning
Workload partitioning:
Terracotta does not ship with a particular workload partitioning strategy in the box. Layer 7 sticky load balancer that is able to route requests for a particular user session to the same Terracotta client application server and is likewise able to balance user sessions evenly across all available application servers can be used to achieve workload partitioning. Use of round-robin or a non-sticky load balancing is strongly discouraged. Other forms of workload partitioning may be required for different use cases. Implementing workload partitioning algorithms in Java that can then be made cluster-aware via Terracotta is often a simple way to meet custom workload partitioning requirements.
Data partitioning
Each application server should only have the data that its workload requires access to in heap. (Magic warning!) Terracotta detects how your application uses its data and optimizes the data location automatically to minimize network updates. Terracotta manages replicating in the most efficient way. Only deltas to your object graph are pushed, objects automatically reside in JVMs that need them, and not in JVMs that don’t (vs. alternatives that always force you to configure and partition up front).
Transactional
Terracotta transactions are sets of clustered object changes that must be applied atomically. Transactions are bounded by lock acquisition and release. When a clustered lock is acquired by a thread, a Terracotta transaction is started and all changes to clustered objects made within the scope of that lock are added to that transaction. When the lock is released, the Terracotta transaction is committed. If a thread attempts to modify a clustered object outside the context of a terracotta transaction, a runtime exception will be thrown.
Replication and consistency
Fully synchronous replication between nodes. The data is always coherent and doesn’t suffer any cache drifts.
Popularity: 69% [?]
Share This
It is an awesome tour of Paris. The route leads from the Champs-Elysées to the Place de la Concorde, Rue de Rivoli along the Louvre, then Place de la Bastille and roundabout of the Place de la Nation. Not far next is the second biggest park in the city: Bois des Vincennes where runners make a 7km loop. Half marathon mark is on the way back from the park to Bastille where slight downhill helps save some energy for the second and final part. From Bastille the route goes towards the river Saine and the next 10km give everyone chance to enjoy river side views of Paris. The famous-infamous 31km mark is next to the Eiffel Tower where couple of hundred meter up hill stints remind about the distance behind and the already drained to minimum resources in the body. The biggest park in Paris, Parc de Boulogne, is a place for final effort where each passed kilometer brings us closer to the finish line on the Avenue Foch overlooking Arc de Triomphe.
On personal note I am happy with my own performance, far from above mentioned time, but within my sub-4h target. The 
Update on my running journey. Week and a bit ago, on Sunday 22nd March 2009, I’ve participated in the 
I spent some time at the beginning of the year looking for a marathon training plan suitable for me. If I had to find a common pattern in the structure of the plans I found it would be fair to say that most of them consist of a one long weekend run (20km - 25 km) and, depending how long and intensive the plan is, 2 - 3 other shorter runs (5km - 10km, intervals, hills) a week. The main message usually is: the more kilometers a week you run, the better you are prepared. D’oh!
If you don’t want to use 