首页 > 数据库 >MongoDB数据的导出导入及日志分析

MongoDB数据的导出导入及日志分析

时间:2023-01-04 18:24:16浏览次数:40  
标签:11 59 MongoDB flowtest 18T09 导入 2022 0800 日志

一、远程连接导出报错超时

mongodump -h 10.110.63.150:27017 -u'admin' -p'passwd!' --authenticationDatabase flowtest --db flowtest -o /home/mongod/bak > mongodump.log 2>&1 &
导出报错:
2022-11-17T18:58:54.663+0800    [###############.........]  flowtest.processDataObj  68313/106962  (63.9%)
2022-11-17T18:58:56.506+0800    Failed: error writing data for collection `flowtest.processDataObj` to disk: error reading collection: connection pool for 10.110.63.150:27017 was cleared because another operation failed with: connection(10.110.63.150:27017[-2]) incomplete read of message header: read tcp 7.7.7.11:53305->10.110.63.150:27017: i/o timeout

二、本地导出不会超时

bin/mongodump -u'admin' -p'passwd!' --authenticationDatabase flowtest --db flowtest -o /home/nrms/thirdparty/mongodb/mongodb-linux-x86_64-3.6.13/bak > mongodump.log 2>&1 &
同时可见,导出不压缩,数据量比源目录大了很多,大约2倍,侧面说明了mongodb库本身的内部数据压缩效果不错:
1.4G    data
2.8G    bak
233M    bak.tar

三、导入1个集合 history_task 到 testdb 库

mongorestore -h node1:20000 -u'liking' -p'passwd!2#' --authenticationDatabase testdb --db testdb --collection history_task /home/mongod/bak/flowtest/history_task.bson > mongorestore.log 2>&1 &

四、导入全部

mongorestore -h node1:20000 -u'admin' -p'passwd!2#' --authenticationDatabase admin --db flowtest /home/mongod/bak/flowtest > mongorestore.log 2>&1 &

五、导入日志分析

确定还原的集合列表,并读取各个集合的元数据:

2022-11-18T09:59:51.909+0800    The --db and --collection flags are deprecated for this use-case; please use --nsInclude instead, i.e. with --nsInclude=${DATABASE}.${COLLECTION}
2022-11-18T09:59:51.910+0800    building a list of collections to restore from /home/mongod/bak/flowtest dir
2022-11-18T09:59:51.911+0800    reading metadata for flowtest.history_task from metadata
2022-11-18T09:59:51.912+0800    reading metadata for flowtest.processConfiguration from metadata
2022-11-18T09:59:51.925+0800    reading metadata for flowtest.processDataObj from metadata
2022-11-18T09:59:51.930+0800    reading metadata for flowtest.processDataObjInit from metadata
2022-11-18T09:59:51.931+0800    reading metadata for flowtest.snapshotResource from metadata
2022-11-18T09:59:51.932+0800    reading metadata for flowtest.activityConfiguration from metadata
2022-11-18T09:59:51.945+0800    reading metadata for flowtest.processDataObjDraft from metadata
2022-11-18T09:59:51.945+0800    reading metadata for flowtest.processsDataObjInit from metadata
2022-11-18T09:59:51.946+0800    reading metadata for flowtest.role from metadata

同时开启4个并发 restoring:

2022-11-18T09:59:52.290+0800    restoring flowtest.snapshotResource
2022-11-18T09:59:52.335+0800    restoring flowtest.history_task
2022-11-18T09:59:52.369+0800    restoring flowtest.processDataObj
2022-11-18T09:59:52.396+0800    restoring flowtest.processDataObjInit

每 finished 1个,即再开始 restoring 另1个:

2022-11-18T09:59:52.824+0800    finished restoring flowtest.snapshotResource (790 documents, 0 failures)
2022-11-18T09:59:52.878+0800    restoring flowtest.processDataObjDraft
2022-11-18T09:59:53.096+0800    finished restoring flowtest.processDataObjDraft (123 documents, 0 failures)
2022-11-18T09:59:53.159+0800    restoring flowtest.activityConfiguration

每 finished 2个,即再开始 restoring 另2个:

2022-11-18T09:59:53.335+0800    finished restoring flowtest.activityConfiguration (1308 documents, 0 failures)
2022-11-18T09:59:53.382+0800    finished restoring flowtest.history_task (3629 documents, 0 failures)
2022-11-18T09:59:53.464+0800    restoring flowtest.processConfiguration
2022-11-18T09:59:53.544+0800    restoring flowtest.processsDataObjInit
2022-11-18T09:59:53.577+0800    finished restoring flowtest.processConfiguration (168 documents, 0 failures)
2022-11-18T09:59:53.622+0800    restoring flowtest.role from /home/mongod/bak/flowtest/role.bson
2022-11-18T09:59:55.108+0800    finished restoring flowtest.role (10 documents, 0 failures)
2022-11-18T09:59:55.108+0800    finished restoring flowtest.processsDataObjInit (4 documents, 0 failures)
2022-11-18T10:00:28.471+0800    finished restoring flowtest.processDataObjInit (96008 documents, 0 failures)
2022-11-18T10:00:45.768+0800    finished restoring flowtest.processDataObj (106968 documents, 0 failures)

开始恢复 indexes:

2022-11-18T10:00:45.781+0800    restoring indexes for collection flowtest.processDataObjInit from metadata
2022-11-18T10:00:45.799+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"flowId_1_activityConfiguration.activityNameEn_1", "ns":"flowtest.processDataObjInit", "v":2}, Key:primitive.D{primitive.E{Key:"flowId", Value:1}, primitive.E{Key:"activityConfiguration.activityNameEn", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2022-11-18T10:00:45.799+0800    no indexes to restore for collection flowtest.snapshotResource
2022-11-18T10:00:45.799+0800    no indexes to restore for collection flowtest.activityConfiguration
2022-11-18T10:00:45.799+0800    no indexes to restore for collection flowtest.history_task
2022-11-18T10:00:45.799+0800    restoring indexes for collection flowtest.processDataObj from metadata
2022-11-18T10:00:45.799+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"flowId_1_activityConfiguration.activityNameEn_1", "ns":"flowtest.processDataObj", "v":2}, Key:primitive.D{primitive.E{Key:"flowId", Value:1}, primitive.E{Key:"activityConfiguration.activityNameEn", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2022-11-18T10:00:45.799+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"flowNo_1", "ns":"flowtest.processDataObj", "v":2}, Key:primitive.D{primitive.E{Key:"flowNo", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2022-11-18T10:00:45.799+0800    no indexes to restore for collection flowtest.processsDataObjInit
2022-11-18T10:00:45.799+0800    no indexes to restore for collection flowtest.processConfiguration
2022-11-18T10:00:45.799+0800    no indexes to restore for collection flowtest.processDataObjDraft
2022-11-18T10:00:45.800+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"flowNo_1", "ns":"flowtest.processDataObjInit", "v":2}, Key:primitive.D{primitive.E{Key:"flowNo", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2022-11-18T10:00:45.800+0800    no indexes to restore for collection flowtest.role

最后的总结:恢复了 209008 document(s)

2022-11-18T10:00:48.031+0800    209008 document(s) restored successfully. 0 document(s) failed to restore.

标签:11,59,MongoDB,flowtest,18T09,导入,2022,0800,日志
From: https://www.cnblogs.com/likingzi/p/17025668.html

相关文章

  • 查询服务器日志时的操作
      查看log.log日志文件(实时滚动刷新)tail-flog.log通用的查日志方式,使用less进入日志文件比如查看当前目录下的console.log文件lessconsole.log#查找某个......
  • SiteFactory编辑器支持Word图文自动导入
    ​ ueditor粘贴不能粘贴word中的图片是一个很头疼的问题,在我们的业务场景中客户要求必须使用ueditor并且支持word的图片粘贴,因为这个需求头疼了半个月,因为前端方面因为安......
  • MongoDB6.0的安装「2023年」
    你好,我是悦创。优质原文格式:https://bornforthis.cn/column/crawler/supplement/mongodb-install.html点进去有惊喜。吐槽,这篇博客的产生是因为本人被MongoDB的安装坑......
  • 数据库日志——binlog、redo log、undo log扫盲
    日志是数据库中比较重要的组成部分,很多核心的功能必须依靠日志才能完成。该篇文章简要介绍了binlog、redolog与undolog,能够在一定程度上拓宽对mysql日志的整体认识。......
  • 如何导入本地数据?
    具体步骤打开数据中心上传CSV文件(目前仅支持CSV格式文件)确认导入行数调整表结构导入完成步骤分解:打开数据中心上传 CSV 文件确认导入行数调整表结构导入完成......
  • .net core(.net 6) 日志记录 log4net ----将日志写入txt文本
    .net6框架内置了log,但是该log只能在控制台打印日志,在实际项目中我们需要将日志实现持久化,将日志写进文档、写入数据库等,所以选择了log4net。1、引入NuGet包Microsoft.......
  • MongoDB从入门到实战之.NET Core使用MongoDB开发ToDoList系统(2)-Swagger框架集成
    Swagger是什么?Swagger是一个规范且完整API文档管理框架,可以用于生成、描述和调用可视化的RESTful风格的Web服务。Swagger的目标是对RESTAPI定义一个标准且和语......
  • 从零开始学习 MySQL 系列--索引、视图、导入和导出
    前言上篇文章我们学习了数据库和数据表操作语句,今天我们学习下数据库索引,视图,导入和导出的知识。作为基础篇,不会涉及到关于索引和视图的高级应用和核心概念,但是基本操作大家......
  • ELK日志系统:Elasticsearch + Logstash + Kibana 搭建教程 good
     elk中kibna搜索时,如果搜索 包含 单个 双引号 的字符串时:"'/goods"&&"result\":true"  使用双引号包起来作为一个短语搜索"likeGecko"#字段也可以按页面左侧显示......
  • Linux三剑客日志处理系列
    三剑客日志处理系列一、特殊符号1.引号系列必会引号含义单引号所见即所得,单引号里的内容会原封不动输出双引号和单引号类似,对双引号里面的特殊符号进......