【Flink系列十八】Hudi hive_sync JDO报错 Could not find API definition for name "JDO"

问题现象

Error : Could not find API definition for name "JDO". Perhaps you dont have the requisite datanucleus-api-XXX jar in the CLASSPATH?

2024-10-10 11:12:31,251 ERROR DataNucleus.Persistence                                      [] - Error : Could not find API definition for name "JDO". Perhaps you dont have the requisite datanucleus-api-XXX jar in the CLASSPATH?
Caused by: org.datanucleus.exceptions.NucleusUserException: Error : Could not find API definition for name "JDO". Perhaps you dont have the requisite datanucleus-api-XXX jar in the CLASSPATH?
	at org.datanucleus.api.ApiAdapterFactory.getApiAdapter(ApiAdapterFactory.java:93) ~[hudi-flink-bundle-1.17.1-0.14.3-slankka.jar:0.14.0]
	at org.datanucleus.AbstractNucleusContext.<init>(AbstractNucleusContext.java:118) ~[hudi-flink-bundle-1.17.1-0.14.3-slankka.jar:0.14.0]
	at org.datanucleus.PersistenceNucleusContextImpl.<init>(PersistenceNucleusContextImpl.java:170) ~[hudi-flink-bundle-1.17.1-0.14.3-slankka.jar:0.14.0]
	at org.datanucleus.PersistenceNucleusContextImpl.<init>(PersistenceNucleusContextImpl.java:159) ~[hudi-flink-bundle-1.17.1-0.14.3-slankka.jar:0.14.0]
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:433) ~[hudi-flink-bundle-1.17.1-0.14.3-slankka.jar:0.14.0]
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:311) ~[hudi-flink-bundle-1.17.1-0.14.3-slankka.jar:0.14.0]
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:220) ~[hudi-flink-bundle-1.17.1-0.14.3-slankka.jar:0.14.0]
	at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_351]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_351]
	at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965) ~[hudi-flink-bundle-1.17.1-0.14.3-slankka.jar:0.14.0]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_351]
	at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960) ~[hudi-flink-bundle-1.17.1-0.14.3-slankka.jar:0.14.0]
	at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166) ~[hudi-flink-bundle-1.17.1-0.14.3-slankka.jar:0.14.0]
	... 53 more

问题分析

Could not find API definition for name "JDO"，直接查找hudi源码

位置：org.datanucleus.api.ApiAdapterFactory#getApiAdapter

api = (ApiAdapter) pluginMgr.createExecutableExtension("org.datanucleus.api_adapter", "name", 
                    name, "class-name", null, null);
                if (api == null)
                {
                    String msg = Localiser.msg("022001", name);
                    NucleusLogger.PERSISTENCE.error(msg);
                    throw new NucleusUserException(msg);
                }

显然是插件加载机制，找不到 org.datanucleus.api_adapter 名为 JDO的一个插件

所在位置 datanucleus-api-jdo-3.2.6.jar!\plugin.xml

    <extension point="org.datanucleus.api_adapter">
        <api-adapter name="JDO" class-name="org.datanucleus.api.jdo.JDOAdapter"/>
    </extension>

一看是配置文件，那么马上想到是打包的问题，检查发现，果然如此，是上述 hudi-flink-bundle-1.17.1-0.14.3-slankka.jar 所致。

shade打包，一般我们会特殊处理 META-INF下的services等。这类软件包的配置文件比较特殊。因此很可能没有特殊处理，因此 shade 会将同名文件随机互相覆盖。

• datanucleus-api-jdo-4.2.4.jar
• datanucleus-core-4.1.17.jar
• datanucleus-rdbms-4.1.19.jar

经过网络搜索发现 shade 插件打包不容易做到 merge xml文件。因此国际互联网上的解决方法是合并成一个 plugin.xml

解决方案

1. 要么放弃这种 shade 为一个独立的 jar，单独引入 上述datanucleus-api-jdo 等软件包
2. 要么 shade的时候，再维护一份合并好的 plugin.xml