Google Open Sources C/C++ MapReduce Framework
Google Open Sources C/C++ MapReduce Framework
Written by Kay Ewbank   
Friday, 06 March 2015

MapReduce Framework for C (MR4C) will let you run native code in Hadoop, allowing you to use image processing libraries developed in C and C++ on data held in Hadoop.

hadmapreducebanner

 

The framework was originally developed at Skybox Imaging, a satellite imagery company that Google acquired in June 2014, for the purpose of large scale satellite image processing and geospatial data science. There are a number of proprietary systems that execute native code in MapReduce frameworks, but MR4C is designed to be more flexible and as it is open source, can be freely used and further developed.

According to the blog post by Ty Kennedy-Bowdoin  about the release, MR4C has a few simple concepts that make it easier to move your native code to Hadoop. Algorithms are stored in native shared objects that access data from the local filesystem or any uniform resource identifier (URI), while input/output datasets, runtime parameters, and any external libraries are configured using JavaScript Object Notation (JSON) files. Splitting mappers and allocating resources can be configured with Hadoop YARN based tools or at the cluster level for MRv1.

mr4c

 

You can also string workflows of multiple algorithms together using an automatically generated configuration. There are callbacks in place for logging and progress reporting, and the reports can be viewed using the Hadoop JobTracker interface. Your workflow can be built and tested on a local machine using exactly the same interface employed on the target cluster.

The blog post says that the goal of this project is to abstract the important details of the MapReduce framework and allow users to focus on developing valuable algorithms.

There’s more information on the MR4C github page: https://github.com/google/mr4c

mapredsq

More Information

MR4C on GitHub

MapReduce for C: Run Native Code in Hadoop

Skybox Imaging

Related Articles

Google Moves On From MapReduce, Launches Cloud Dataflow 

Agile Data Science (book review) 

Big Data Analytics (book review) 

 

To be informed about new articles on I Programmer, install the I Programmer Toolbar, subscribe to the RSS feed, follow us on, Twitter, FacebookGoogle+ or Linkedin,  or sign up for our weekly newsletter.

 

Banner


Google's Fuchsia OS And The Forking Of Swift
21/11/2017

This more a "what is going on" sort of story than any prediction of the future. About a week ago the Google team developing the mysterious Fuchsia operating system forked Apple's Swift language. Now i [ ... ]



Algorithmic Designs Of Wind-up Toys
19/11/2017

Oh, the joys of being a programmer - you can put your nose into any topic and do something fun. In this case a team of researchers has automated and improved the design of wind-up toys.


More News

 

 
 

 

blog comments powered by Disqus

Last Updated ( Friday, 06 March 2015 )
 
 

   
Banner
RSS feed of news items only
I Programmer News
Copyright © 2017 i-programmer.info. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.