[ERROR： mysql_real_connect failed: errno: , error]

kubectl logs transform-pod-name -n kubeflow 

-->

INFO:absl:MetadataStore with gRPC connection initialized
WARNING:absl:mlmd client InternalError: mysql_real_connect failed: errno: , error:
ERROR:absl:[Transform] Input resolution error: Error while resolving inputs for Transform
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/ml_metadata/metadata_store/metadata_store.py", line 219, in _call_method
response.CopyFrom(grpc_method(request, timeout=self._grpc_timeout_sec))
File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 1030, in call
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "mysql_real_connect failed: errno: , error: "
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2024-01-30T09:52:37.395417001+00:00", grpc_status:13, grpc_message:"mysql_real_connect failed: errno: , error: "}"

kubectl logs metadata-grpc-deployment-pod-name -n kubeflow

-->
W0130 09:49:08.262073 4255 metadata_store_service_impl.cc:747] PutContexts failed: Given node already exists: type_id: 12
name: "detect-anomolies-on-wafer-tfdv-schema-lnz6m"
INTERNAL: Cannot create node for type_id: 12 name: "detect-anomolies-on-wafer-tfdv-schema-lnz6m"mysql_query failed: errno: Duplicate entry '12-detect-anomolies-on-wafer-tfdv-schema-lnz6m' for key 'Context.type_id', error: Duplicate entry '12-detect-anomolies-on-wafer-tfdv-schema-lnz6m' for key 'Context.type_id'
W0130 09:49:26.348727 4258 metadata_store_service_impl.cc:869] PutParentContexts failed: Given parent_context already exists: child_id: 27
parent_id: 1
INTERNAL: mysql_query failed: errno: Duplicate entry '27-1' for key 'ParentContext.PRIMARY', error: Duplicate entry '27-1' for key 'ParentContext.PRIMARY' [mysql-error-info='\x08\xa6\x08']
E0130 09:50:47.328933 4263 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/to fix this issue.
W0130 09:50:47.329303 4263 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0130 09:52:27.551858 4294 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/to fix this issue.
W0130 09:52:27.552182 4294 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0130 09:52:37.185184 4281 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/to fix this issue.
W0130 09:52:37.185544 4281 metadata_store_service_impl.cc:944] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0130 09:54:15.667371 4323 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/to fix this issue.
W0130 09:54:15.667706 4323 metadata_store_service_impl.cc:944] Failed to connect to the database: mysql_real_connect failed: errno: , error:
(base) maye@maye-Inspiron-5547:~$

[ANALYSIS]

1. What's the meaning of "mysql_real_connect failed: errno: , error" ?

"mysql_real_connect failed: errno: , error":
mysql_real_connect failed, but errno and error are empty string, errno and error are from error message of database server, so this means that the database server has not received the request of connecting at all, not something wrong during connecting, if so, the database server will give error message.

2. so why the database server does not receive the request of connecting ?

From the log of pod metadata-grpc-deployment, It can be seen that "mysql_real_connect failed: errno: , error" occurs after "mysql_query failed: errno: Duplicate entry...", so is there is correlation between them ? but before transform, there are other components (such as StatisticsGen), and all components have the same context pipeline-run, and all components need to put context pipeline-run, and meet the error: Duplicated entry, but other components before transform just ignore the error, and have not re-connected to mysql and fail. so there should be no correlation beteew "Error: Dulicated entry" and re-connect to mysql and fail.

# ml-metadata/ml_metadata/metadata_store/metadata_store_service_impl.cc
::grpc::Status MetadataStoreServiceImpl::PutParentContexts(
    ::grpc::ServerContext* context, const PutParentContextsRequest* request,
    PutParentContextsResponse* response) {
  std::unique_ptr<MetadataStore> metadata_store;
  const ::grpc::Status connection_status =
      ConnectMetadataStore(connection_config_, &metadata_store);
  if (!connection_status.ok()) {
    LOG(WARNING) << "Failed to connect to the database: "
                 << connection_status.error_message();
    return connection_status;
  }
  const ::grpc::Status transaction_status =
      ToGRPCStatus(metadata_store->PutParentContexts(*request, response));
  if (!transaction_status.ok()) {
    LOG(WARNING) << "PutParentContexts failed: "
                 << transaction_status.error_message();
  }
  return transaction_status;
}


# ml-metadata/ml_metadata/metadata_store/mysql_metadata_source.cc 
Status MySqlMetadataSource::ConnectImpl() {

  // Initialize the MYSQL object.
  db_ = mysql_init(nullptr);
  if (!db_) {
    LOG(ERROR) << "MySQL error: " << mysql_errno(db_) << ": "
               << mysql_error(db_);
    return BuildErrorStatus(absl::StatusCode::kInternal, "mysql_init failed",
                            mysql_errno(db_), mysql_error(db_));
  }

  // Explicitly setup the thread-local initializer.
  MLMD_RETURN_WITH_CONTEXT_IF_ERROR(ThreadInitAccess(),
                                    "MySql thread init failed at ConnectImpl");

  // Set connection options
  if (config_.has_ssl_options()) {
    const MySQLDatabaseConfig::SSLOptions& ssl = config_.ssl_options();
    // The method set mysql_options, and always return 0. The connection options
    // are used in the `mysql_real_connect`.
    mysql_ssl_set(db_, ssl.key().empty() ? nullptr : ssl.key().c_str(),
                  ssl.cert().empty() ? nullptr : ssl.cert().c_str(),
                  ssl.ca().empty() ? nullptr : ssl.ca().c_str(),
                  ssl.capath().empty() ? nullptr : ssl.capath().c_str(),
                  ssl.cipher().empty() ? nullptr : ssl.cipher().c_str());
    my_bool verify_server_cert = ssl.verify_server_cert() ? 1 : 0;
    mysql_options(db_, MYSQL_OPT_SSL_VERIFY_SERVER_CERT, &verify_server_cert);
  }

  mysql_options(db_, MYSQL_DEFAULT_AUTH, "mysql_native_password");
  // Connect to the MYSQL server.
  db_ = mysql_real_connect(
          db_, config_.host().empty() ? nullptr : config_.host().c_str(),
          config_.user().empty() ? nullptr : config_.user().c_str(),
          config_.password().empty() ? nullptr : config_.password().c_str(),
          /*db=*/nullptr, config_.port(),
          config_.socket().empty() ? nullptr : config_.socket().c_str(),
          /*clientflag=*/0UL);

  if (!db_) {
    LOG(ERROR)
        << "MySQL database was not initialized. Please ensure your "
           "MySQL server is running. Also, this error might be caused by "
           "starting from MySQL 8.0, mysql_native_password used by MLMD is not "
           "supported as a default for authentication plugin. Please follow "
           "<https://dev.mysql.com/blog-archive/"
           "upgrading-to-mysql-8-0-default-authentication-plugin-"
           "considerations/>"
           "to fix this issue.";
    return BuildErrorStatus(absl::StatusCode::kInternal,
                            "mysql_real_connect failed", mysql_errno(db_),
                            mysql_error(db_));
  }

  // Return an error if the default storage engine doesn't support transactions.
  MLMD_RETURN_WITH_CONTEXT_IF_ERROR(
      CheckTransactionSupport(),
      "checking transaction support of default storage engine");


  // Create the database if not already present and skip_db_creation is false.
  if (!config_.skip_db_creation()) {
    const std::string create_database_cmd =
        absl::StrCat("CREATE DATABASE IF NOT EXISTS ", config_.database());
    MLMD_RETURN_WITH_CONTEXT_IF_ERROR(RunQuery(create_database_cmd),
                                      "Creating database ", config_.database(),
                                      " in ConnectImpl");
  }

  // Switch to the database.
  if (database_name_.empty()) {
    database_name_ = config_.database();
  }

  const std::string use_database_cmd = absl::StrCat("USE ", database_name_);
  MLMD_RETURN_WITH_CONTEXT_IF_ERROR(RunQuery(use_database_cmd),
                                    "Changing to database ", database_name_,
                                    " in ConnectImpl");

  return absl::OkStatus();
}