折腾 Quickwit，Rust 编写的分布式搜索引擎 - 可观测性之日志管理

Quickwit 从底层构建，旨在 高效地索引非结构化数据，并在云存储上轻松搜索这些数据。
此外，Quickwit 开箱即支持 OpenTelemetry gRPC 和 HTTP（仅 protobuf）协议，并提供了一个 REST API，可以接收任何 JSON 格式的日志。
这让 Quickwit 成为了日志的理想选择！.

• https://quickwit.io/docs/guides/schemaless

使用 OTEL Collector 发送日志

如果您已经有了自己的 OpenTelemetry Collector 并希望将日志导出到 Quickwit，您需要在 config.yaml 中配置一个新的 OLTP gRPC exporter：

macOS/Windows

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  otlp/quickwit:
    endpoint: host.docker.internal:7281
    tls:
      insecure: true  
    # By default, logs are sent to the otel-logs-v0_7.
    # You can customize the index ID By setting this header.
    # headers:
    #   qw-otel-logs-index: otel-logs-v0_7
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/quickwit]

Linux

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  otlp/quickwit:
    endpoint: 127.0.0.1:7281
    tls:
      insecure: true
    # By default, logs are sent to the otel-logs-v0_7.
    # You can customize the index ID By setting this header.
    # headers:
    #   qw-otel-logs-index: otel-logs-v0_7

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/quickwit]

测试您的 OTEL 配置

1. 安装 并启动一个 Quickwit server：

• https://quickwit.io/docs/get-started/installation

./quickwit run

2. 使用之前的配置启动一个 collector：

macOS/Windows

docker run -v ${PWD}/otel-collector-config.yaml:/etc/otelcol/config.yaml -p 4317:4317 -p 4318:4318 -p 7281:7281 otel/opentelemetry-collector

Linux

docker run -v ${PWD}/otel-collector-config.yaml:/etc/otelcol/config.yaml --network=host -p 4317:4317 -p 4318:4318 -p 7281:7281 otel/opentelemetry-collector

3. 使用 cURL 向您的 collector 发送一条日志：

curl -XPOST "http://localhost:4318/v1/logs" -H "Content-Type: application/json" \
--data-binary @- << EOF
{
 "resource_logs": [
   {
     "resource": {
       "attributes": [
         {
           "key": "service.name",
           "value": {
             "stringValue": "test-with-curl"
           }
         }
       ]
     },
     "scope_logs": [
       {
         "scope": {
           "name": "manual-test"
         },
         "log_records": [
           {
             "time_unix_nano": "1678974011000000000",
             "observed_time_unix_nano": "1678974011000000000",
             "name": "test",
             "severity_text": "INFO"
           }
         ]
       }
     ]
   }
 ]
}
EOF

您应该会在 Quickwit 服务器上看到类似以下的日志：

2023-03-16T13:44:09.369Z  INFO quickwit_indexing::actors::indexer: new-split split_id="01GVNAKT5TQW0T2QGA245XCMTJ" partition_id=6444214793425557444

这意味着 Quickwit 已经收到了日志并创建了一个新的分片。在搜索日志之前，请等待分片被发布。

通过 OTEL 发送 K8s 日志

本指南将帮助您解锁 Kubernetes 集群日志上的日志搜索功能。我们首先使用 Helm 部署 Quickwit 和 OTEL 收集器，然后了解如何索引和搜索这些日志。

• https://helm.sh/

前提条件

完成本教程，您需要以下工具：

• 一个 Kubernetes 集群。
• 命令行工具 kubectl。

• https://kubernetes.io/docs/reference/kubectl/

• 命令行工具 Helm。

• https://helm.sh/

• 对象存储（如 AWS S3、GCS、Azure Blob 存储或 Scaleway）的访问权限，用于存储索引数据。

使用 Helm 安装

首先，让我们创建一个命名空间来隔离我们的实验，并将其设置为默认命名空间。

kubectl create namespace qw-tutorial
kubectl config set-context --current --namespace=qw-tutorial

然后添加 Quickwit 和 Otel Helm 仓库：

• https://github.com/quickwit-oss/helm-charts
• https://github.com/open-telemetry/opentelemetry-helm-charts

helm repo add quickwit https://helm.quickwit.io
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

现在您应该能在 Helm 中看到这两个仓库：

helm repo list
NAME                	URL
quickwit            	https://helm.quickwit.io
open-telemetry      	https://open-telemetry.github.io/opentelemetry-helm-charts

部署 Quickwit

让我们创建一个基本的 chart 配置：

export AWS_REGION=us-east-1
export AWS_ACCESS_KEY_ID=XXXX
export AWS_SECRET_ACCESS_KEY=XXXX
export DEFAULT_INDEX_ROOT_URI=s3://your-bucket/indexes

# Create Quickwit config file.
echo "
searcher:
  replicaCount: 1
indexer:
  replicaCount: 1
metastore:
  replicaCount: 1
janitor:
  enabled: true
control_plane:
  enabled: true

environment:
  # Remove ANSI colors.
  NO_COLOR: 1

# Quickwit configuration
config:
  storage:
    s3:
      region: ${AWS_REGION}
      access_key_id: ${AWS_ACCESS_KEY_ID}
      secret_access_key: ${AWS_SECRET_ACCESS_KEY}
      # If you are not on AWS S3, you can define a flavor (gcs, minio, garage...)
      # and additional variables for your object storage.
      # flavor: gcs
      # endpoint: https://storage.googleapis.com

  # Metastore on S3.
  metastore_uri: ${DEFAULT_INDEX_ROOT_URI}

  default_index_root_uri: ${DEFAULT_INDEX_ROOT_URI}

  # Indexer settings
  indexer:
    # By activating the OTEL service, Quickwit will be able
    # to receive gRPC requests from OTEL collectors.
    enable_otlp_endpoint: true
" > qw-tutorial-values.yaml

在安装 Quickwit chart 之前，请确保您能访问 S3 并且 default_index_root_uri 中没有拼写错误。这可以通过 aws CLI 使用简单的 ls 命令轻松完成：

aws s3 ls ${DEFAULT_INDEX_ROOT_URI}

如果 CLI 没有返回错误，那么您就可以安装 chart 了：

helm install quickwit quickwit/quickwit -f qw-tutorial-values.yaml

过一会儿，您会看到正在运行 Quickwit 服务的 pod：

kubectl get pods
NAME                                      READY   STATUS    RESTARTS      AGE
quickwit-control-plane-7fc495f4c4-slqv4   1/1     Running   2 (84s ago)   87s
quickwit-indexer-0                        1/1     Running   2 (84s ago)   87s
quickwit-janitor-7f75f4bc8-jrfv6          1/1     Running   2 (84s ago)   87s
quickwit-metastore-6989978fc-9s82j        1/1     Running   2 (85s ago)   87s
quickwit-searcher-0                       1/1     Running   2 (84s ago)   87s

让我们检查 Quickwit 是否正常工作：

kubectl port-forward svc/quickwit-searcher 7280

然后在浏览器中打开 http://localhost:7280/ui/indexes。您应该能看到索引列表。如果一切正常，请继续运行 kubectl 命令，并打开一个新的终端。

部署 OTEL 收集器

我们需要对收集器进行一些配置，以便：

• 从 Kubernetes 收集日志
• 用 Kubernetes 属性丰富日志
• 将日志导出到 Quickwit 索引器。

echo "
mode: daemonset
presets:
  logsCollection:
    enabled: true
  kubernetesAttributes:
    enabled: true
config:
  exporters:
    otlp:
      endpoint: quickwit-indexer.qw-tutorial.svc.cluster.local:7281
      tls:
        insecure: true
      # By default, logs are sent to the otel-logs-v0_7.
      # You can customize the index ID By setting this header.
      # headers:
      #   qw-otel-logs-index: otel-logs-v0_7
  service:
    pipelines:
      logs:
        exporters:
          - otlp
" > otel-values.yaml

helm install otel-collector open-telemetry/opentelemetry-collector -f otel-values.yaml

几秒钟后，您应该会在索引器上看到显示索引已开始的日志。看起来像这样：

2022-11-30T18:27:37.628Z  INFO spawn_merge_pipeline{index=otel-log-v0 gen=0}: quickwit_indexing::actors::merge_pipeline: Spawning merge pipeline. index_id=otel-log-v0 source_id=_ingest-api-source pipeline_ord=0 root_dir=/quickwit/qwdata/indexing/otel-log-v0/_ingest-api-source merge_policy=StableLogMergePolicy { config: StableLogMergePolicyConfig { min_level_num_docs: 100000, merge_factor: 10, max_merge_factor: 12, maturation_period: 172800s }, split_num_docs_target: 10000000 }
2022-11-30T18:27:37.628Z  INFO quickwit_serve::grpc: Starting gRPC server. enabled_grpc_services={"otlp-log", "otlp-trace"} grpc_listen_addr=0.0.0.0:7281
2022-11-30T18:27:37.628Z  INFO quickwit_serve::rest: Starting REST server. rest_listen_addr=0.0.0.0:7280
2022-11-30T18:27:37.628Z  INFO quickwit_serve::rest: Searcher ready to accept requests at http://0.0.0.0:7280/
2022-11-30T18:27:42.654Z  INFO quickwit_indexing::actors::indexer: new-split split_id="01GK4WPTXK8GH3AGTRNBN9A8YG" partition_id=0
2022-11-30T18:27:52.643Z  INFO quickwit_indexing::actors::indexer: send-to-index-serializer commit_trigger=Timeout split_ids=01GK4WPTXK8GH3AGTRNBN9A8YG num_docs=22
2022-11-30T18:27:52.652Z  INFO index_batch{index_id=otel-log-v0 source_id=_ingest-api-source pipeline_ord=0}:packager: quickwit_indexing::actors::packager: start-packaging-splits split_ids=["01GK4WPTXK8GH3AGTRNBN9A8YG"]
2022-11-30T18:27:52.652Z  INFO index_batch{index_id=otel-log-v0 source_id=_ingest-api-source pipeline_ord=0}:packager: quickwit_indexing::actors::packager: create-packaged-split split_id="01GK4WPTXK8GH3AGTRNBN9A8YG"
2022-11-30T18:27:52.653Z  INFO index_batch{index_id=otel-log-v0 source_id=_ingest-api-source pipeline_ord=0}:uploader: quickwit_indexing::actors::uploader: start-stage-and-store-splits split_ids=["01GK4WPTXK8GH3AGTRNBN9A8YG"]
2022-11-30T18:27:52.733Z  INFO index_batch{index_id=otel-log-v0 source_id=_ingest-api-source pipeline_ord=0}:uploader:stage_and_upload{split=01GK4WPTXK8GH3AGTRNBN9A8YG}:store_split: quickwit_indexing::split_store::indexing_split_store: store-split-remote-success split_size_in_megabytes=0.018351 num_docs=22 elapsed_secs=0.07654519 throughput_mb_s=0.23974074 is_mature=false

如果您在此处看到一些错误，可能是由于对象存储配置错误导致的。如果您需要帮助，请在 GitHub 上提交问题或加入我们的 Discord 服务器。

• https://github.com/quickwit-oss/quickwit
• https://discord.gg/MT27AG5EVE

准备好搜索日志

现在您可以开始搜索了，等待 30 秒，您将看到第一批索引的日志：只需 打开 UI 并开始使用。有趣的是，您会在这个 UI 中看到 Quickwit 的日志