注意事项
-
表要有主键
-
库名和表名不能有点号
是个 BUG，估计后续会修复。
- 表名不能有大写
也是个 BUG，估计后续会修复。
如果表名含有大写的字母，查询时日志可看到如下信息：
java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File does not exist: hdfs://hadoop/user/test/warehouse/test.db/ods_test
at org.sparkproject.guava.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:552)
at org.sparkproject.guava.util.concurrent.AbstractFuture.get(AbstractFuture.java:513)
at org.sparkproject.guava.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:90)
at org.sparkproject.guava.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:199)
at org.sparkproject.guava.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2312)
at org.sparkproject.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
at org.sparkproject.guava.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154)
at org.sparkproject.guava.cache.LocalCache$Segment.get(LocalCache.java:2044)
at org.sparkproject.guava.cache.LocalCache.get(LocalCache.java:3952)
at org.sparkproject.guava.cache.LocalCache$LocalManualCache.get(LocalCache.java:4871)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getCachedPlan(SessionCatalog.scala:158)
而底层 HDFS 路径保持了大写：
2023-05-11 18:18:27,943 INFO [47] [org.apache.hudi.util.StreamerUtil.initTableIfNotExists(StreamerUtil.java:335)] - Table [hdfs://hadoop/user/test/warehouse/test.db/ODS_test/ODS_test] already exists, no need to initialize the table
2023-05-11 18:18:27,943 INFO [47] [org.apache.hudi.util.StreamerUtil.initTableIfNotExists(StreamerUtil.java:337)] - Table update under base path hdfs://hadoop/user/test/warehouse/test.db/ODS_test
2023-05-11 18:18:27,944 INFO [47] [org.apache.hudi.common.table.HoodieTableMetaClient.updateTableAndGetMetaClient(HoodieTableMetaClient.java:507)] - Update hoodie table with basePath hdfs://hadoop/user/test/warehouse/test.db/ODS_test
2023-05-11 18:18:27,949 INFO [47] [org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:125)] - Loading HoodieTableMetaClient from hdfs://hadoop/user/test/warehouse/test.db/ODS_test
2023-05-11 18:18:27,950 INFO [47] [org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:366)] - Hadoop UGI authentication : TAUTH
2023-05-11 18:18:27,993 INFO [47] [org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:295)] - Loading table properties from hdfs://hadoop/user/test/warehouse/test.db/ODS_test/.hoodie/hoodie.properties
2023-05-11 18:18:28,212 INFO [47] [org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:144)] - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from hdfs://hadoop/user/test/warehouse/test.db/ODS_test
2023-05-11 18:18:29,730 INFO [47] [org.apache.hudi.common.table.HoodieTableMetaClient.updateTableAndGetMetaClient(HoodieTableMetaClient.java:512)] - Finished update Table of type MERGE_ON_READ from hdfs://hadoop/user/test/warehouse/test.db/ODS_test
| Flink写 | Spark读 |
|---|---|
| hdfs://hadoop/user/test/warehouse/test.db/ODS_test | hdfs://hadoop/user/test/warehouse/test.db/ods_test |