Hive on Hadoop for MongoDB
Hive on Hadoop for MongoDB
Written by Kay Ewbank   
Thursday, 22 August 2013

There’s a new version of 10gen’s MongoDB Connector for Hadoop with added support for Apache Hive and incremental MapReduce jobs.

 

The MongoDB Connector for Hadoop presents MongoDB as a Hadoop-compatible file system so that real-time data from MongoDB can be read and processed by Hadoop MapReduce jobs. It examines the MongoDB collection and calculates a set of splits from the data. Each split is assigned to a node in the Hadoop cluster, and in parallel, Hadoop nodes pull data for their splits from MongoDB (or BSON) and process them locally. Hadoop then merges the results and streams the output back to MongoDB or BSON.

The major changes to the new version start with the Apache Hive with SQL-like queries across live MongoDB data sets. Hive is a query engine for Hadoop that provides an alternative to writing MapReduce jobs for analyzing Hadoop Distributed File System (HDFS) datasets. Using Hive with MongoDB won’t be completely straightforward; some MongoDB data types such as ObjectID don’t have direct matches in Hive, and it may be tricky to work out how to express field mappings between Hive fields and MongoDB fields so that all cases are handled correctly because of the different underlying data models.

mongohadoophive

 

The new version adds support for MongoDB’s native BSON (Binary JSON) backup files which can be stored locally in HDFS so reducing data movement between MongoDB and Hadoop. The ability to work on MongoDB backup files also opens the possibility of reducing the load on an operational cluster; analysis could be carried out on the backup without significant loss of accuracy.

The new version also adds support for incremental MapReduce jobs making it easier to carry out efficient ad-hoc analytics. This is achieved using a new feature, MongoUpdateWriteable, that allows Hadoop to modify an existing collection in MongoDB, rather than only writing to new collections. Using this, you can run incremental MapReduce jobs to aggregate trends or pattern matching on a daily basis, which can then be efficiently queried in a single collection by MongoDB.

There’s a good webinar explaining the new features and its also summarized in this slideset: 

 

 

Banner


Microsoft Open Sources Drone Simulator
17/02/2017

An open source simulator that you can use to crash-test drones and robots has been released by Microsoft. The simulator can be used to test the devices virtually rather than wrecking them in the real  [ ... ]



New Magazine for Computing Education
29/01/2017

The Raspberry Pi Foundation has launched Hello World a magazine about computing and digital making aimed at, and written by, educators. Issue 1 is available now as a print magazine in the UK and  [ ... ]


More News

Last Updated ( Thursday, 22 August 2013 )
 
 

   
Banner
RSS feed of news items only
I Programmer News
Copyright © 2017 i-programmer.info. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.