首页 > 其他分享 >记录一次hudi 编译过程遇到过的问题

记录一次hudi 编译过程遇到过的问题

时间：2023-08-15 20:23:18浏览次数：58  
标签：java hudi 记录 hadoop 编译 ERROR apache org

准备工作

pom中初始依赖组件版本配置如下

<java.version>1.8</java.version>
<hadoop.version>3.1.1.3.1.0.0-78</hadoop.version>
<hive.version>3.1.0.3.1.0.0-78</hive.version>
<kafka.version>2.0.0</kafka.version>
起始命令
mvn clean package -U -DskipTests -Dcheckstyle.skip -Dmaven.javadoc.skip=true -Pspark3.1 -Pflink1.14 -am

1.hudi-hadoop-mr 模块和hive版本不兼容问题

报错内容如下

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project hudi-hadoop-mr: Compilation failure: Compilation failure: 
[ERROR] /root/bigdata/hudi/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveAvroSerializer.java:[302,93] incompatible types: org.apache.hadoop.hive.common.type.Date cannot be converted to java.sql.Date
[ERROR] /root/bigdata/hudi/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveAvroSerializer.java:[305,72] incompatible types: org.apache.hadoop.hive.common.type.Timestamp cannot be converted to java.sql.Timestamp
[ERROR] /root/bigdata/hudi/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveAvroSerializer.java:[309,98] incompatible types: org.apache.hadoop.hive.common.type.Date cannot be converted to java.sql.Date

1.1 修改代码

      case DATE:
        return DateWritable.dateToDays(((DateObjectInspector)fieldOI).getPrimitiveJavaObject(structFieldData));
      case TIMESTAMP:
        Timestamp timestamp =
            ((TimestampObjectInspector) fieldOI).getPrimitiveJavaObject(structFieldData);
        return timestamp.getTime();
      case INT:
        if (schema.getLogicalType() != null && schema.getLogicalType().getName().equals("date")) {
          return DateWritable.dateToDays(new WritableDateObjectInspector().getPrimitiveJavaObject(structFieldData));
        }

修改为

      case DATE:
        return ((DateObjectInspector)fieldOI).getPrimitiveJavaObject(structFieldData).toEpochDay();
      case TIMESTAMP:
        Timestamp timestamp =
                ((TimestampObjectInspector) fieldOI).getPrimitiveJavaObject(structFieldData).toSqlTimestamp();
        return timestamp.getTime();
      case INT:
        if (schema.getLogicalType() != null && schema.getLogicalType().getName().equals("date")) {
          return new WritableDateObjectInspector().getPrimitiveJavaObject(structFieldData).toEpochDay();
        }

2.hudi-hadoop-mr 模块代码风格检查问题

2.1 报错如下：

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.0.0:check (default) on project hudi-hadoop-mr: You have 1 Checkstyle violation. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

2.2 编译时指定跳过代码风格检查

mvn clean package -U -DskipTests -Dcheckstyle.skip -Dmaven.javadoc.skip=true -Pspark3.1 -Pflink1.14 -am -rf :hudi-hadoop-mr

3.hudi-hive-sync模块和zookeeper 版本不兼容问题

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:testCompile (default-testCompile) on project hudi-hive-sync: Compilation failure
[ERROR] /data/HDP_Codebase/hudi-0.13.1/hudi-sync/hudi-hive-sync/src/test/java/org/apache/hudi/hive/testutils/HiveTestUtil.java:[188,15] method shutdown in class org.apache.zookeeper.server.ZooKeeperServer cannot be applied to given types;
[ERROR]   required: no arguments
[ERROR]   found: boolean
[ERROR]   reason: actual and formal argument lists differ in length
[ERROR] 
[ERROR] -> [Help 1]

3.1 调整pom 里zookeeper 的版本依赖为

3.5.7
调整为
3.4.6.3.1.0.0-78

3.2 修改HiveTestUtil源码

查看 wesure-hive-release 源码中都是 zkServer.shutdown();
修改代码 /HDP_Codebase/hudi-0.13.1/hudi-sync/hudi-hive-sync/src/test/java/org/apache/hudi/hive/testutils/HiveTestUtil.java

    if (zkServer != null) {
      zkServer.shutdown(true);
    }

修改为

    if (zkServer != null) {
      zkServer.shutdown();
    }

4.hudi-timeline-service模块参数不兼容问题

4.1 报错如下

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project hudi-timeline-service: Compilation failure
[ERROR] /data/HDP_Codebase/hudi-0.13.1/hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/TimelineService.java:[348,115] incompatible types: int cannot be converted to java.lang.ClassLoader
[ERROR] 
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hudi-timeline-service

4.2 代码传参异常，ScheduledExecutorScheduler 第三个参数为ClassLoader

    final Server server = new Server(pool);
    ScheduledExecutorScheduler scheduler = new ScheduledExecutorScheduler("TimelineService-JettyScheduler", true, 8);
    server.addBean(scheduler);

支持2个参数，去掉第三个参数调整为
final Server server = new Server(pool);
ScheduledExecutorScheduler scheduler = new ScheduledExecutorScheduler("TimelineService-JettyScheduler", true);
server.addBean(scheduler);

5.hudi-utilities 模块io.confluent 依赖无法解析

5.1 报错内容如下

[ERROR] Failed to execute goal on project hudi-utilities_2.11: Could not resolve dependencies for project org.apache.hudi:hudi-utilities_2.11:jar:0.13.1: The following artifacts could not be resolved: io.confluent:kafka-avro-serializer:jar:5.3.4, io.confluent:common-config:jar:5.3.4, io.confluent:common-utils:jar:5.3.4, io.confluent:kafka-schema-registry-client:jar:5.3.4: Could not find artifact io.confluent:kafka-avro-serializer:jar:5.3.4 in HDPReleases (http://repo.hortonworks.com/content/repositories/releases) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hudi-utilities_2.11

5.2 调整 mvn 中的confluent仓库检索顺序

<mirrorOf>*,!HDPReleases</mirrorOf>

调整为
<mirrorOf>*,!HDPReleases,!confluent</mirrorOf>

6.编译成功

标签：java,hudi,记录,hadoop,编译,ERROR,apache,org
From： https://www.cnblogs.com/tommyjiang/p/17632338.html

相关文章

  • 【随手记录】harbor安装及登录
    harbor仓库安装：1、harbor包下载 https://github.com/goharbor/harbor/releases2、解压 有harbor需要的镜像包(redis、nginx、harbor-core等)和启动脚本，需要复制harbor.yml.tmpl，改为harbor.yml然后编辑此配置文件，主要调整以下三项：3、赋予脚本install.sh和prepa......
  • 记录--vue3问题：如何实现微信扫码授权登录？
    这里给大家分享我在网上总结出来的一些知识，希望对大家有所帮助一、需求微信扫码授权，如果允许授权，则登录成功，跳转到首页。二、问题1、微信扫码授权有几种实现方式？2、说一下这几种实现方式的原理是什么？3、vue中的微信扫码授权登录，与uniapp和原生小程序的微信授权登录，它们......
  • 记录学习day1
    今天在boss上统计了一下.net初级开发技能要求接下来就按照这个学习路线来进行了，随机找了南宁的5家公司下面是要求  前端技术:JavaScript(6)vueAjax(4)bootstrap(2)jquery(2)UniappknokoutJS(不如vue)前端库:jquery-easyuielem后端:webapi(4)ASP.NETMVC（3）多......
  • vscode使用记录
    1、ctrl+p打开全文搜索，快速查找文件（有个查找小技巧，比如需要查找一个叫DemoOpenGameInfo的文件，可以输入demoInfo,这样子可以直接排除剩下类似同名文件）2、Shift+Alt+方向键↓拷贝当前一行代码到下一行（远离ctrl+c和ctrl+v）3、Alt+方向键↓移动当前一行代码到......
  • 记录 FFmpeg开发常用功能封装
    说明记录下个人在开发中使用到的FFmpeg常用功能，避免相同功能代码的重复编写，使用时直接复制提升效率。由于音视频处理的场景众多，无法编写完全通用的方法接口，可能需根据实际场景进行一定的修改，本文章中的代码也将持续更新优化。代码这里提供ffmpegheader.h，ffmpegheader.cpp。配......
  • dav 编译报错 v8内存溢出
    dav编译报错v8内存溢出FATALERROR:ReachedheaplimitAllocationfailed-JavaScriptheapoutofmemory  到node_modules中/.bin/roadhog.cmd把最后一句改成endLocal&goto#_undefined_#2>NUL||title%COMSPEC%&"%_prog%"--max_old_space_size=8192......
  • 编译greenDao的时候出现KaptExecution:java.lang.reflect.InvocationTargetException
    一、概述在编译greendao的时候出现了以下异常。Executionfailedfortask':common_base:kaptDebugKotlin'.>Afailureoccurredwhileexecutingorg.jetbrains.kotlin.gradle.internal.KaptExecution>java.lang.reflect.InvocationTargetException(noerrormes......
  • Git使用问题记录
    问题一fatal:unabletoaccess‘https://…git/’:SSLcertificateproblem:selfsignedcertificateincertificatechain打开GitBash运行如下命令exportGIT_SSL_NO_VERIFY=truegitconfig--globalhttp.sslVerify"false"或者在windows的命令行中，进入到git命令所......
  • iTOP-RK3588开发板单独编译Android固件-打包update.img
    在ubootkernelandroid都编译完成的情况下，才可以打包update.img，所以一般在完整编译的时候用。输入以下命令：./build.sh-u打包完成会在rockdev/Image-rk3588_s目录下生成update.img镜像。更多内容：B站搜索迅为RK3588开发板......
  • dpdk编译开发
    下载源码http://core.dpdk.org/download/编译http://core.dpdk.org/doc/quick-start/安装python3安装ninjayuminstallninja-build安装mesonpip3installmeson开始编译tarxfdpdk.tar.gzcddpdkmesonbuildninja-Cbuild确定配置好大页内存mkdir......

赞助商

阅读排行

网站主页关于我们联系我们网站地图

本网站内容转载自其他媒体，侵权联系[admin##ips99.com]。

Copyright © 2020-2023 IPS99 版权所有 IPS99