The Holumbus Developer

Finally, I have managed to release the new first version of the distributed MapReduce system which should be used some day to build inverted indexes for the Holumbus-Searchengine. The existing Holumbus-Searchengine library already offers a MapReduce implementation, but it does not provide mechanisms for distributed computing. That why three new libraries have been developed:

• Holumbus-MapReduce 0.0.1
• Holumbus-Distribution 0.0.1
• Holumbus-Storage 0.0.1

The newly developed Holumbus-MapReduce library contains of tools and modules for building individualized distributed MapReduce systems.

Holumbus-MapReduce is based on two libraries: Holumbus-Distributed and Holumbus-Storage. Maybe they could be useful for building other distributed systems, not only a MapReduce system.

Holumbus-Distribution consists of modules and tools for the implementation of distributed systems in general. Besides common data types and small helper functions, this library provides a mailbox-based data transfer mechanism. It offers an easy-to-use interface for inter- and intra-process communication like in Erlang or Mozart/Oz.

Holumbus-Storage is designed to build a distributed storage system which can be linked into other programs. It is used by the MapReduce system for the data management.

All three libraries are still under heavy development and should not be considered for creating applications in productive environments. But I think, it’s time to give them to the community for suggestions on further development and features.

The first step is done, lets go further!

This entry was posted on Friday, March 6th, 2009 at 22:36 and is filed under Holumbus. Tags: distribution, mapreduce, storage.