dremio 的配置基于了typesafe 的config
参考配置
- 模块级别的配置
sabot-module.conf
dremio.classpath.scanning {
base.classes += "com.dremio.dac.cmd.upgrade.UpgradeTask"
packages += "com.dremio.dac"
packages += "com.dremio.dac.support.SupportService"
package += "com.dremio.dac.support.QueryLogBundleService"
annotations += "com.dremio.dac.cmd.AdminCommand"
packages += "com.dremio.dac.cmd"
packages += "com.dremio.dac.cmd.upgrade"
}
- 参考配置（基础默认的）
dremio.conf 运行配置
paths: {
# the local path for dremio to store data.
local: ${DREMIO_HOME}"/data"
# the distributed path Dremio data including job results, downloads, uploads, etc
#dist: "pdfs://"${paths.local}"/pdfs"
}
services: {
coordinator.enabled: true,
coordinator.master.enabled: true,
executor.enabled: true,
flight.use_session_service: true
}
参考类图
typesafe Config 类图 
公共配置加载处理
DremioConfig 配置
public static DremioConfig create(final URL userConfigPath, SabotConfig sabotConfig) {
Config reference = null;
final ClassLoader[] classLoaders = ClasspathHelper.classLoaders();
for (ClassLoader classLoader : classLoaders) {
final URL configUrl = classLoader.getResource(REFERENCE_CONFIG);
if(configUrl == null){
continue;
}
Preconditions.checkArgument(reference == null, "Attempted to load more than one reference configuration.");
// 参考配置 dremio-reference.conf
reference = ConfigFactory.parseResources(classLoader, REFERENCE_CONFIG);
}
Preconditions.checkNotNull(reference, "Unable to find the reference configuration.");
Config userConfig = null;
if(userConfigPath == null){
for (ClassLoader classLoader : classLoaders) {
// 默认配置 dremio.conf
final URL configUrl = classLoader.getResource(DEFAULT_USER_CONFIG);
if(configUrl == null){
continue;
}
Preconditions.checkArgument(userConfig == null, "Attempted to load more than one user configuration.");
userConfig = ConfigFactory.parseResources(classLoader, DEFAULT_USER_CONFIG);
}
} else {
userConfig = ConfigFactory.parseURL(userConfigPath, ConfigParseOptions.defaults().setAllowMissing(false));
}
final Config effective;
if(userConfig != null){
effective = userConfig;
} else {
effective = reference;
}
final Config skinned =
applySystemProperties(
applyLegacySystemProperties(effective),
reference);
return new DremioConfig(sabotConfig, skinned, reference, determineNode());
}
SabotConfig 配置加载
sabot-override.conf 加载处理
private static SabotConfig doCreate(String overrideFileResourcePathname,
Properties overriderProps) {
final StringBuilder logString = new StringBuilder();
final Stopwatch watch = Stopwatch.createStarted();
overrideFileResourcePathname =
overrideFileResourcePathname == null
? CommonConstants.CONFIG_OVERRIDE_RESOURCE_PATHNAME
: overrideFileResourcePathname;
// 1. Load defaults configuration file.
Config fallback = null;
final ClassLoader[] classLoaders = ClasspathHelper.classLoaders();
for (ClassLoader classLoader : classLoaders) {
// 加载默认的
final URL url =
classLoader.getResource(CommonConstants.CONFIG_DEFAULT_RESOURCE_PATHNAME);
if (null != url) {
logString.append("Base Configuration:\n\t- ").append(url).append("\n");
fallback =
ConfigFactory.load(classLoader,
CommonConstants.CONFIG_DEFAULT_RESOURCE_PATHNAME);
break;
}
}
// 加载模块的配置
// 2. Load per-module configuration files.
final Collection<URL> urls = ClassPathScanner.getConfigURLs();
logString.append("\nIntermediate Configuration and Plugin files, in order of precedence:\n");
for (URL url : urls) {
logString.append("\t- ").append(url).append("\n");
fallback = ConfigFactory.parseURL(url).withFallback(fallback);
}
logString.append("\n");
// 基于环境变量的配置
// 3. Load any specified overrides configuration file along with any
// overrides from JVM system properties (e.g., {-Dname=value").
// (Per ConfigFactory.load(...)'s mention of using Thread.getContextClassLoader():)
final URL overrideFileUrl =
Thread.currentThread().getContextClassLoader().getResource(overrideFileResourcePathname);
if (null != overrideFileUrl ) {
logString.append("Override File: ").append(overrideFileUrl).append("\n");
}
Config effectiveConfig =
ConfigFactory.load(overrideFileResourcePathname).withFallback(fallback);
// 4. Apply any overriding properties.
if (overriderProps != null) {
logString.append("Overridden Properties:\n");
for(Entry<Object, Object> entry : overriderProps.entrySet()){
logString.append("\t-").append(entry.getKey()).append(" = ").append(entry.getValue()).append("\n");
}
logString.append("\n");
effectiveConfig =
ConfigFactory.parseProperties(overriderProps).withFallback(effectiveConfig);
}
// 5. Create SabotConfig object from Config object.
logger.info("Configuration and plugin file(s) identified in {}ms.\n{}",
watch.elapsed(TimeUnit.MILLISECONDS),
logString);
return new SabotConfig(effectiveConfig.resolve());
}
说明
dremio 基于了typesafe 的config提供了强大的配置管理是一个值得借鉴的设计
参考资料
common/src/main/java/com/dremio/common/config/SabotConfig.java
common/src/main/java/com/dremio/config/DremioConfig.java
common/src/main/java/com/dremio/common/scanner/ClassPathScanner.java
common/src/main/java/com/dremio/common/config/CommonConstants.java
https://github.com/lightbend/config