@lakehouse-rs/flight-sql-client 是基于rust 开发的node arrow flight sql client ，dremio 目前也是推荐基于arrow flight sql 的访问模式
参考代码
- package.json
{
"name": "node-arrow-flight-sql",
"version": "1.0.0",
"main": "index.js",
"type": "module",
"license": "MIT",
"dependencies": {
"@lakehouse-rs/flight-sql-client": "^0.0.10",
"apache-arrow": "^15.0.2"
}
}
- app.js
自己部署一个dremio 服务，可以基于docker 进行快速启动
import { createFlightSqlClient } from '@lakehouse-rs/flight-sql-client';
import { tableFromIPC } from 'apache-arrow';
const options = {
username: 'xxxxx',
password: 'xxxxx',
tls: false,
host: '127.0.0.1',
port: 32010,
headers: [],
};
const client = await createFlightSqlClient(options);
const buffer = await client.query('select * from pg.public.sensor');
const table = tableFromIPC(buffer);
table.toArray().forEach((row) => {
console.log(JSON.stringify(row));
})
const bufferv2 = await client.getTables({ includeSchema: false });
const tablev2 = tableFromIPC(bufferv2);
tablev2.toArray().forEach((row) => {
console.log(JSON.stringify(row));
})
- 效果
说明
基于flight-sql的好处很明显，速度比较快，我以前写过基于odbc+arrow flight 模式的，直接基于arrow flight sql 是一种更加方便的模式
对于基于数据开发的应用flight-sql 是一个高性能的协议值得使用，。nodejs 的实现是基于了napi 这个使用rust 开发node addon 扩展的
框架，后边说明下内部处理原理,返回类似rest api response 格式的处理
const client = await createFlightSqlClient(options);
const buffer = await client.query(sql);
const table = tableFromIPC(buffer);
let res = {
rowCount: table.numRows,
schema: table.schema.fields.map((f) => {
return { name: f.name, type: { name: f.metadata.get("ARROW:FLIGHT:SQL:TYPE_NAME") } };
}),
rows: table.toArray(),
};
参考资料
https://www.npmjs.com/package/apache-arrow
https://github.com/roeap/flight-sql-client-node
https://napi.rs/