mongo与elasticsearch数据同步插件
内容导读
互联网集市收集整理的这篇技术教程文章主要介绍了mongo与elasticsearch数据同步插件，小编现在分享给大家，供广大互联网技能从业者学习和参考。文章包含3454字，纯文字阅读大概需要5分钟。
内容图文
前提条件：
mongo集群必须是副本集，因为同步插件读取的是日志文件
1.需要安装pip插件（如果已经安装不需再安装）
yum install python-setuptools && easy_install pip
2.在mongo端 安装 mongo-connector。
pip install mongo-connector
3.安装 elastic2-doc-manager（注意这个是针对es2.x版本的，如果es安装的是1.x版本，用elastic-doc-manager）
pip install elastic-doc-manager
插件网址：https://github.com/mongodb-labs
4.启动同步插件
- 不使用配置文件
- mongo-connector -m mongo的ip:27017 -t elasticsearch的IP:9200 -d elastic2_doc_manager
- 可以在命令行里面加入
- --auto-commit-interval=0此命令保证实时同步数据,0代表无延迟
- --continue-on-error 当数据中有错误会跳过
2.使用配置文件
| { "__comment__": "Configuration options starting with ‘__‘ are disabled", "__comment__": "To enable them, remove the preceding ‘__‘", "mainAddress": "192.168.106.220:27017", "oplogFile": "/home/mongo/oplog.timestamp", 自定义 "noDump": false, "batchSize": -1, "verbosity": 0, "__continueOnError": true,当数据中有错误会跳过 "logging": { "type": "file", "filename": "/home/mongo/mongo-connector.log", 自定义 "__format": "%(asctime)s [%(levelname)s] %(name)s:%(lineno)d - %(message)s", "__rotationWhen": "D", "__rotationInterval": 1, "__rotationBackups": 10, "__type": "syslog", "__host": "localhost:514" }, "authentication": { "__adminUsername": "username", "__password": "password", "__passwordFile": "mongo-connector.pwd" }, "__comment__": "For more information about SSL with MongoDB, please see http://docs.mongodb.org/manual/tutorial/configure-ssl-clients/", "__ssl": { "__sslCertfile": "Path to certificate to identify the local connection against MongoDB", "__sslKeyfile": "Path to the private key for sslCertfile. Not necessary if already included in sslCertfile.", "__sslCACerts": "Path to concatenated set of certificate authority certificates to validate the other side of the connection", "__sslCertificatePolicy": "Policy for validating SSL certificates provided from the other end of the connection. Possible values are ‘required‘ (require and validate certificates), ‘optional‘ (validate but don‘t require a certificate), and ‘ignored‘ (ignore certificates)." }, "__fields": ["field1", "field2", "field3"], "__namespaces": { "excluded.collection": false, "excluded_wildcard.*": false, "*.exclude_collection_from_every_database": false, "included.collection1": true, "included.collection2": {}, "included.collection4": { "includeFields": ["included_field", "included.nested.field"] }, "included.collection5": { "rename": "included.new_collection5_name", "includeFields": ["included_field", "included.nested.field"] }, "included.collection6": { "excludeFields": ["excluded_field", "excluded.nested.field"] }, "included.collection7": { "rename": "included.new_collection7_name", "excludeFields": ["excluded_field", "excluded.nested.field"] }, "included_wildcard1.*": true, "included_wildcard2.*": true, "renamed.collection1": "something.else1", "renamed.collection2": { "rename": "something.else2" }, "renamed_wildcard.*": { "rename": "new_name.*" }, "gridfs.collection": { "gridfs": true }, "gridfs_wildcard.*": { "gridfs": true } }, "docManagers": [ { "docManager": "elastic_doc_manager", "targetURL": ["192.168.106.218:9200","192.168.106.217:9200","192.168.106.219:9200"], "bulkSize": 1000, "__uniqueKey": "_id", "autoCommitInterval": 1 } ] } |
启动命令：mongo-connector -c /etc/mongo-connector-conf.json & （json的目录自定义）
注意：写入mongo的速度比同步速度快，经过测试elasticsearch配置良好的情况下同步速度大概在1000条/s左右
插件安装的目录在：/usr/lib/python2.7/site-packages下面
elasticsearch、mongo-connector（如果想修改源码可以进入此目录）
原文：http://www.cnblogs.com/yhx-0408/p/7267704.html
内容总结
以上是互联网集市为您收集整理的mongo与elasticsearch数据同步插件全部内容，希望文章能够帮你解决mongo与elasticsearch数据同步插件所遇到的程序开发问题。 如果觉得互联网集市技术教程内容还不错，欢迎将互联网集市网站推荐给程序员好友。
内容备注
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 gblab@vip.qq.com 举报，一经查实，本站将立刻删除。
内容手机端
扫描二维码推送至手机访问。