主要是dremio parquet zstd 压缩支持尝试，说明下思路，大家可以参考
修改
- ExecConstants 配置 sabot/kernel/src/main/java/com/dremio/exec/ExecConstants.java
EnumeratedStringValidator PARQUET_WRITER_COMPRESSION_TYPE_VALIDATOR = new EnumeratedStringValidator(
PARQUET_WRITER_COMPRESSION_TYPE, "snappy", "snappy", "gzip", "none");
- kernel 的 sabot/kernel/src/main/java/com/dremio/exec/store/parquet/ParquetRecordWriter.java
switch (codecName) {
case "snappy":
codec = CompressionCodecName.SNAPPY;
break;
case "lzo":
codec = CompressionCodecName.LZO;
break;
case "gzip":
codec = CompressionCodecName.GZIP;
break;
case "zstd":
codec = CompressionCodecName.ZSTD;
break;
case "none":
case "uncompressed":
codec = CompressionCodecName.UNCOMPRESSED;
break;
default:
throw new UnsupportedOperationException(String.format("Unknown compression type: %s", codecName));
}
构建
修改之后需要自己编译构建，具体参考 dremio github 官方
运行
- docker-compose 环境
注意需要 zstd-jni-1.5.0-1.jar,默认dremio 木有包含需要自己添加下，可以到3rdpart 中
Dockerfile
FROM dremio/dremio-oss:23.0.1
USER root
COPY dremio-env-profile /opt/dremio/conf/dremio-env
# 支持dfs 配置
COPY dremio.conf /opt/dremio/conf/dremio.conf
COPY --from=hengyunabc/arthas:latest /opt/arthas /opt/arthas
COPY zstd-jni-1.5.0-1.jar /opt/dremio/jars/3rdparty/
COPY libhadoop.so.1.0.0 /opt/dremio/lib/libhadoop.so
# 自己构建的dremio-sabot-kernel
COPY dremio-sabot-kernel-23.0.1-202210141019030815-c1de8bcc.jar /opt/dremio/jars/dremio-sabot-kernel-23.0.1-202210141019030815-c1de8bcc.jar
USER dremio
docker-compose 文件
version: "3"
services:
mongo:
image: mongo:4.2
environment:
- "MONGO_INITDB_ROOT_USERNAME=root"
- "MONGO_INITDB_ROOT_PASSWORD=dalongdemo"
ports:
- "27017:27017"
minio:
image: minio/minio
ports:
- "9002:9000"
- "19001:19001"
environment:
MINIO_ACCESS_KEY: minio
MINIO_SECRET_KEY: minio123
command: server --console-address :19001 --quiet /data
mysql:
image: mysql:5.6
environment:
- "MYSQL_ROOT_PASSWORD=dalong"
ports:
- "3306:3306"
dremio:
build: ./
image: dalongrong/dremio:23.1-jprofiler
volumes:
- ./dat:/data
ports:
- "9047:9047"
- "8849:8849"
- "31010:31010"
- 开启配置
dremio support key 添加 store.parquet.compression=zstd 
问题
- hadoop zstd 支持
解决方法，下载一个包含了zstd 的libhadoop.so 文件，我已经找到一个提供在github 中
- 创建效果
- 查询数据
- 查看生成的parquet 文件
使用parquet-tools 自己下载安装（最好使用python venv），然后下载s3 的数据 
说明
具体运行环境我已经放到github 中了，大家可以参考
参考资料
https://github.com/luben/zstd-jni
https://github.com/dremio/dremio-oss
https://facebook.github.io/zstd/
https://github.com/apache/parquet-format/blob/master/Compression.md
https://github.com/dremio/dremio-oss/blob/d41cb52143b6b0289fc8ed4d970bfcf410a669e8/services/jobs/src/main/java/com/dremio/service/jobs/JobResultsStore.java
https://github.com/rongfengliang/dremio-parquet-zstd