CometCloud is an autonomic computing engine for Cloud and Grid environments. It is based on the Comet decentralized coordination substrate, and supports highly heterogeneous and dynamic cloud/Grid infrastructures, integration of public/private clouds and autonomic cloudbursts. CometCloud provides a shared coordination space over the Chord overlay network and various types of programming paradigms such as Master/Worker, Workflow, and MapReduce/Hadoop.
CometCloud supports autonomic cloudbursts and autonomic cloud-bridging on a virtual cloud which integrates local computational environments and public cloud services on-the-fly. Also it supports real-world scientific and engineering applications.
What is CometCloud capable of?
- Support core programming paradigms for real-world data and compute intensive applications
- Enables autonomic cloudbursts and cloud-bridging and on-demand scale-out and scale-in, driven by dynamic policies, economic model, QoS constraints, etc.
- Programming system support deployment of native (Java as well as non-Java) applications without any change.
- Current deployments include a virtual cloud integrating Amazon EC2, Eucalyptus, local clusters, and TeraGrid.
- Incorporates mechanisms for fault-tolerance to handle node and communication failures and recovery.
What are CometCloud Features?
- Pull-based task consumption: workers pull tasks from the Comet space whenever they are available
- Policy-based autonomic cloudbursts: CometCloud supports on-demand scale-out and scale-in by policy and economic model.
- Cloud-bridging: CometCloud provides a cloud-bridging over a virtual cloud which integrates datacenter, clouds such as Amazon EC2 and Eucalyptus, and Grids such as Teragrid.
- Support for multi-core processors: users can run multiple workers on a node and utilized multiple cores.
- Support MapReduce with naïve disk writing: instead of using Hadoop file system, CometCloud uses the Comet space for storing data to support MapReduce and writes data into the disk if available memory amount goes below a threshold.
- Master throttling: CometCloud can control the rate of task generation of masters. If the rate is high, then more tasks keep in the space.
- Multiple masters: multiple masters can generate tasks and insert them into the same Comet space to collaborate.
- Task update: if a task with the same task id is inserted, the existing task is updated.
- Garbage collection: if tasks remain in the space for some time without being consumed, then they are cleared.
- Distributed task generation by workers: even workers can generate tasks and insert them into the space. Because every task should have a unique task id, CometCloud provides a way to provide a globally unique id to distributed workers.
- Fault-tolerance by replicating task space: each node replicates the preceding node’s local space, and it merges the replicated space when its preceding node fails.