AppBundleGenerator 的目的是方便yarn 应用的运行，dremio 自己开发了一个方便软件打包的服务，可以简化
参考处理
默认生成的jar 包名称dremio-bundle.jar
public Path generateBundle() throws IOException {
// Create an application bundle jar based on current application classpath
Path yarnBundledJarPath = Files.createTempFile(DREMIO_BUNDLE_PREFIX, ".jar");
// dremio包装的JarGenerator ，生成jar 包
try (JarGenerator jarGenerator = JarGenerator.of(new JarOutputStream(Files.newOutputStream(yarnBundledJarPath)))) {
// First add prefix classpath entries
// Second, add content of classpath to bundle jar
// Then add extra classpath entries
List<URI> jarEntries;
try (Stream<Path> prefixStream = toPathStream(classPathPrefix);
Stream<Path> classLoaderStream = toPathStream(classLoader);
Stream<Path> classPathStream = toPathStream(classPath)) {
jarEntries = addPathsToBundle(jarGenerator,
Stream.concat(prefixStream, Stream.concat(classLoaderStream, classPathStream)));
}
// After that add native libraries
List<URI> nativeLibrariesEntries = addPathsToBundle(jarGenerator, nativeLibraryPath.stream().map(Paths::get));
// After that add plugins
URI pluginsPathEntry = addPathToJar(jarGenerator, pluginsPath);
// jar Manifest 添加
// Finally, add classpath and native library path entries in jar manifest
// Following spec for class-path, string is a list of URI separated by space...
Manifest manifest = new Manifest();
final Attributes mainAttributes = manifest.getMainAttributes();
mainAttributes.put(Attributes.Name.MANIFEST_VERSION, "1.0");
mainAttributes.put(Attributes.Name.CLASS_PATH,
jarEntries.stream()
.map(URI::toString)
.collect(Collectors.joining(DELIMITER)));
mainAttributes.putValue(X_DREMIO_LIBRARY_PATH_MANIFEST_ATTRIBUTE,
nativeLibrariesEntries.stream().map(URI::toString).collect(Collectors.joining(DELIMITER)));
mainAttributes.putValue(X_DREMIO_PLUGINS_PATH_MANIFEST_ATTRIBUTE, pluginsPathEntry.toString());
jarGenerator.addManifest(manifest);
}
return yarnBundledJarPath;
}
使用
DacDaemonYarnApplication 类的构造函数中
public DacDaemonYarnApplication(DremioConfig dremioConfig, YarnConfiguration yarnConfig, @NotNull Environment env) {
this.yarnConfig = yarnConfig;
// 注意需要一个DREMIO_HOME 环境变量
final String dremioHome = Preconditions.checkNotNull(env.getEnv(DREMIO_HOME),
"Environment variable DREMIO_HOME is not set");
this.keytabFileLocation = dremioConfig.getString(DremioConfig.KERBEROS_KEYTAB_PATH);
// Gather the list of jars to be added to the container classpath
Stream<String> classpathJars = Stream.concat(
yarnConfig.getTrimmedStringCollection(YarnDefaultsConfigurator.CLASSPATH_JARS).stream().map(d -> dremioHome.concat(d)),
dremioConfig.getStringList(DremioConfig.YARN_CLASSPATH).stream());
classpathJars.forEach(classpathJar -> {
Path jarFullPath = Paths.get(classpathJar);
Path parentDir = jarFullPath.getParent();
final String fileName = jarFullPath.getFileName().toString();
try(Stream<Path> paths = Files.list(parentDir)) {
paths
.filter(p -> p.getFileName().toString().matches(fileName))
.forEach(p -> {
jarNames.add(p.getFileName().toString());
this.classpathJarNames.add(p.toFile());
});
} catch(IOException e) {
logger.warn("Cannot list files in directory {}", parentDir, e);
}
});
// Create an application bundle jar based on current application classpath
AppBundleGenerator appBundleGenerator = AppBundleGenerator.of(dremioConfig);
try {
if (!isTestingModeOn) {
yarnBundledJarPath = appBundleGenerator.generateBundle();
} else {
yarnBundledJarPath = Paths.get("/temp");
}
} catch (IOException e) {
throw new RuntimeException(e);
}
}
说明
dremio 的AppBundleGenerator 主要是生成jar 文件（都使用了java jar 操作api），方便yarn 运行（实际上就是fat jar），后边YarnController 会使用到，具体会介绍
参考资料
provision/yarn/yarntwill/src/main/java/com/dremio/provision/yarn/AppBundleGenerator.java
provision/yarn/yarntwill/src/main/java/com/dremio/provision/yarn/DacDaemonYarnApplication.java
https://docs.dremio.com/software/advanced-administration/start-stop/#starting-up-with-yarn-deployments
https://docs.dremio.com/software/deployment/yarn-hadoop/