Hadoop is a framework for running applications on large clusters of commodity hardware. The Hadoop framework transparently provides applications both reliability and data motion. Hadoop implements a computational paradigm named map/reduce, where the application is divided into many small fragments of work, each of which may be executed or reexecuted on any node in the cluster. In addition, it provides a distributed file system that stores data on the compute nodes, providing very high aggregate bandwidth across the cluster. Both map/reduce and the distributed file system are designed so that node failures are automatically handled by the framework.

Amazon recently released their EC2 Elastic Computing cloud which allows developers to acquisition computing power a the rate of $0.10 per hour consumed. Recently work as been done to allow Hadoop to run on EC2. This combination will allow developers to write scalable algorithms and then bring up large numbers of servers for computing power which can then be then shut them down when they are not needed.