Choosing Between ElasticSearch, MongoDB & Hadoop

An interesting trend has been developing in the IT landscape over the past few years.  Many new technologies develop and immediately latch onto the “Big Data” buzzword.  And as older technologies add “Big Data” features in an attempt to keep up with the Joneses, we are seeing a blurring of the boundaries between various technologies.  Say you have search engines such as ElasticSearch or Solr storing JSON documents, MongoDB storing JSON documents, or a pile of JSON documents stored in HDFS on a Hadoop cluster. Interestingly enough, you can fulfill many of the same use cases with any of these three configurations.

ElasticSearch as a NoSQL database?  Superficially, that doesn’t sound right, but nonetheless it is a valid scenario.  Likewise, MongoDB with support for MapReduce over sharded collections can accomplish many of the same things as Hadoop.  And, of course, with the many tools that you can layer on top of a Hadoop base (Hive, HBase, Pig, and the like) you can query data from your Hadoop cluster in a multitude of ways.

Given that, can we now say that Hadoop, MongoDB and ElasticSearch are all exactly equivalent?  Of course not.  Each tool still has a niche for which it is most ideally suited, but each has enough flexibility to fulfill multiple roles.  The question now becomes “What is the ideal use for each of these technologies”, and that my friends is what we will explore now.

ElasticSearch has begun to spread beyond its roots as a “pure” search engine and now adds some features for analytics and visualization - but at its core, remains primarily a full-text search engine for locating documents by keyword.  ElasticSearch builds on top of Lucene and supports extremely fast lookup and a rich query syntax.  If you have millions (or more) of text documents and you need to locate documents by keywords located in that text, ElasticSearch fits the bill perfectly.  Yes, if your documents are JSON you can treat ElasticSearch as a sort of lightweight “NoSQL database”.  But ElasticSearch is not quite a “database engine” and provides less support for complex calculations and aggregation as part of a query - although the “statistics” facet does provide some ability to retrieve calculated statistical information scoped to the given query.   Facets in ElasticSearch are intended mainly to support a “faceted navigation” facility.

If you are looking to return a (usually small) collection of documents in response to a keyword query, and want the ability to support faceted navigation around those documents, then ElasticSearch is probably your best, first choice.  If you need to perform more complex calculations, run server-side scripts against your data, and easily run MapReduce jobs on your data, then MongoDB or Hadoop enter the picture.

如果你的目的是通过指定关键字查询取得一个文档集合(通常是较小的)，并且具有支持对文档基于切面的导航，elasticsearch会非常合适，但是如果你希望支持更多复杂的计算，在服务器端你的数据上运行脚本，很容易的在你的数据上运行mapreduce成寻，那么MongoDB和Hadoop就在被考虑的范围里边了。

MongoDB is a “NoSQL” database which is designed from the ground up to be highly scalable, with automatic sharding support and a number of additional performance optimizations.  MongoDB is a document oriented database which stores document in a “JSON like” format (technically BSON) with some extensions beyond plain JSON - for example, a native date type.  MongoDB provides a text index type for supporting full-text search against fields which contain text, so we can see that there is overlap between what you can do with ElasticSearch and MongoDB, in terms of basic keyword search against a collection of documents.  

Where MongoDB goes beyond ElasticSearch is its support for features like server-side scripts in Javascript, aggregation pipelines, MapReduce support and capped collections.   With MongoDB, you can use aggregation pipelines to process documents in a collection, streaming them through a sequence of pipeline operators where each operator transforms the document.  Pipeline operators can generate entirely new documents or remove documents from the final output.   This is a very powerful facility for filtering, processing and transforming data as it is retrieved.   MongoDB also supports running map/reduce jobs over the data in a collection, using custom Javascript functions for the map and reduce phases of the operation.  This allows for ultimate flexibility in performing any type of calculation or transformation to the selected data. 

与ES相比，MongoDB超过es的地方是支持服务器端的javascript和聚合管道 以及MapReduce的支持和capped collections

Another extremely powerful feature in MongoDB is known as “capped collections”.  With the capped collections facility, a user can define a maximum size for a collection - after which the collection can simply be written to blindly, and it will roll-over data as necessary to maintain the specified size limit.   This feature is extremely useful for capture logs and other streaming data for analysis.

另外一个非常NB的特征是capped collections;通过capped collections，用户可以定义为一个collection定义最大的size，用来插入数据(只能插入更新 不能删除)，按照LRU挤出数据存放新插入的数据,这个特点非常适合获取log数据和流数据的存储和分析

科普一下capped collections

特点：

1.只能插入，更新，不能删除doc，可以使用drop()删除整个collection

2.LRU列表，相信大家对这个应该很了解了，oracle里面很多地方就是用的这个规则，如果指定的集合大小满了，那么会按照LRU挤出数据存放新插入的数据，这里记得更新是不能超出collection的大小的，不能挤出空间存放更新的数据，这个也合情合理。

3.插入的记录都是按照插入的顺序排列，普通的collection在_id上是肯定有索引的，但是这里是没有的

4.可以快速的查询和插入，如果写比读的比例大，建议不要建立索引，否则写会耗费很多额外的资源。

As you can see, while ElasticSearch and MongoDB have some overlap in possible use cases, they are not the same tool.  But what about Hadoop? Isn’t Hadoop “just MapReduce” which is supported by MongoDB anyway? Is there really a use case for Hadoop where MongoDB is just as suitable.

In a word, yes.  Hadoop is the grand-father of MapReduce based cluster computing frameworks.  Hadoop provides probably the overall most flexible and powerful environment for processing large amounts of data, and definitely fits niches for which you would not use ElasticSearch or MongoDB.   

To understand why this is true, look at how Hadoop abstracts storage - via HDFS - from the associated computational facility.   With data stored in HDFS, any arbitrary job can be run against that data, using either Java code written to the core MapReduce API, or arbitrary code written in native languages using Hadoop Streaming.   And starting with Hadoop2 and YARN, even the core programming model is abstracted so that you aren’t limited to MapReduce.  With YARN you can, for example, implement MPI on top of Hadoop and write jobs in that style.

Additionally, the Hadoop ecosystem provides a staggering array of tools that build on top of HDFS and core MapReduce to query, analyze and process data.  Hive provides a “SQL like” language that allows Business Analysts to query data using a syntax they are already familiar with.  HBase provides a column oriented database on top of Hadoop.  Pig and Sizzle provide two more alternative programming models for querying Hadoop Data.  With data stored in HDFS using Hadoop, you inherit the ability to simply plug in Apache Mahout to use advanced machine learning algorithms on your data.  While using RHadoop is straightforward to use the R statistical language to perform advanced statistical analyses on Hadoop data.

So while Hadoop and MongoDB also have some overlapping use cases, and share some useful functionality (seamless horizontal scalability, for example) it remains the case that each tool serves a specific purpose in enterprise computing.  If you simply want to locate documents by keyword and perform simple analytics, then ElasticSearch may fit the bill.  If you need to query documents that can be modeled as JSON and perform moderately more sophisticated analysis, then MongoDB becomes a compelling choice.  And if you have a huge quantity of data that needs a wide variety of different types of complex processing and analysis, then Hadoop provides the broadest range of tools and the most flexibility.  

As always, it is important to choose the right tool(s) for the job at hand.  And in the “Big Data” space the sheer number of technologies and the blurry lines can make this difficult.  As we can see, there are specific scenarios which best suit each of these technologies and, more importantly, the differences do matter.  Though, the best news of all is you are not limited to using only one of these tools.   Depending on the details of your use case, it may actually make sense to build a combination platform.  For example, ElasticSearch and Hadoop are known to work well together, with ElasticSearch providing rapid keyword search, and Hadoop jobs powering the more complicated analytics.  

In the end, it takes ample research and careful analysis to make the best choices for your computing environment.   Before selecting any technology or platform, take the time to evaluate it carefully, understand what scenarios it was designed to optimize for, and what tradeoffs and sacrifices it makes.  Start with a small pilot project to “kick the tires” before converting your entire enterprise to a new platform, and slowly grow into the new stack.

Follow these steps and you can successfully navigate the maze of “Big Data” technologies and reap the associated benefits.\

本文转自：http://www.osintegrators.com/opensoftwareintegrators%7CChoosing-Between-ElasticSearch-MongoDB-%2526-Hadoop

ps：只翻译了部分，其他部分将会在晚点时候完成～，翻译不对的地方还请各位指正

原文：http://blog.csdn.net/sunflower_cao/article/details/40039499