Hive on Hadoop for MongoDB
Hive on Hadoop for MongoDB
Written by Kay Ewbank   
Thursday, 22 August 2013

There’s a new version of 10gen’s MongoDB Connector for Hadoop with added support for Apache Hive and incremental MapReduce jobs.

 

The MongoDB Connector for Hadoop presents MongoDB as a Hadoop-compatible file system so that real-time data from MongoDB can be read and processed by Hadoop MapReduce jobs. It examines the MongoDB collection and calculates a set of splits from the data. Each split is assigned to a node in the Hadoop cluster, and in parallel, Hadoop nodes pull data for their splits from MongoDB (or BSON) and process them locally. Hadoop then merges the results and streams the output back to MongoDB or BSON.

The major changes to the new version start with the Apache Hive with SQL-like queries across live MongoDB data sets. Hive is a query engine for Hadoop that provides an alternative to writing MapReduce jobs for analyzing Hadoop Distributed File System (HDFS) datasets. Using Hive with MongoDB won’t be completely straightforward; some MongoDB data types such as ObjectID don’t have direct matches in Hive, and it may be tricky to work out how to express field mappings between Hive fields and MongoDB fields so that all cases are handled correctly because of the different underlying data models.

mongohadoophive

 

The new version adds support for MongoDB’s native BSON (Binary JSON) backup files which can be stored locally in HDFS so reducing data movement between MongoDB and Hadoop. The ability to work on MongoDB backup files also opens the possibility of reducing the load on an operational cluster; analysis could be carried out on the backup without significant loss of accuracy.

The new version also adds support for incremental MapReduce jobs making it easier to carry out efficient ad-hoc analytics. This is achieved using a new feature, MongoUpdateWriteable, that allows Hadoop to modify an existing collection in MongoDB, rather than only writing to new collections. Using this, you can run incremental MapReduce jobs to aggregate trends or pattern matching on a daily basis, which can then be efficiently queried in a single collection by MongoDB.

There’s a good webinar explaining the new features and its also summarized in this slideset: 

 

 

Banner


A New Rift for Node.js
25/08/2017

The Node.js community faces a new crisis which has led to a new fork of the project. Ayo.js. The fact that "ayo" is pronounced the same as "io" is a reference to the previous major fork of the project [ ... ]



Crowd Sourced Solar Eclipse Megamovie
22/08/2017

Using photographs contributed by observers on the path of totality, Google computers have stitched together images of the 2017 solar eclipse to create the “Eclipse Megamovie.” The resulting movie  [ ... ]


More News

Last Updated ( Thursday, 22 August 2013 )
 
 

   
Banner
RSS feed of news items only
I Programmer News
Copyright © 2017 i-programmer.info. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.