[ERROR: Directory '*py3-none-any.whl' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.]

kubectl lgos trainer-pod-name -n kubeflow

-->

, pipeline_info=id: "detect_anomolies_on_wafer_tfdv_schema"
, pipeline_run_id='detect-anomolies-on-wafer-tfdv-schema-lnz6m')
INFO:absl:Train on the 'train' split when train_args.splits is not set.
INFO:absl:Evaluate on the 'eval' split when eval_args.splits is not set.
INFO:absl:udf_utils.get_fn {'custom_config': '{"epochs": 50}', 'eval_args': '{}', 'module_path': 'detect_anomalies_in_wafer_trainer@/tfx/pipelines/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl', 'train_args': '{\n "num_steps": 21\n}'} 'run_fn'
INFO:absl:Installing '/tfx/pipelines/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl' to a temporary directory.
INFO:absl:Executing: ['/usr/bin/python', '-m', 'pip', 'install', '--target', '/tmp/tmp3dt6mzvy', '/tfx/pipelines/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl']
ERROR: Directory '/tfx/pipelines/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.

[notice] A new release of pip is available: 23.1.2 -> 23.3.2
[notice] To update, run: pip install --upgrade pip
INFO:absl:MetadataStore with gRPC connection initialized
ERROR:absl:Execution 27 failed.
INFO:absl:Cleaning up stateless execution info.
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.8/dist-packages/tfx/orchestration/kubeflow/container_entrypoint.py", line 510, in
main(sys.argv[1:])
File "/usr/local/lib/python3.8/dist-packages/tfx/orchestration/kubeflow/container_entrypoint.py", line 502, in main
execution_info = component_launcher.launch()
File "/usr/local/lib/python3.8/dist-packages/tfx/orchestration/portable/launcher.py", line 574, in launch
executor_output = self._run_executor(execution_info)
File "/usr/local/lib/python3.8/dist-packages/tfx/orchestration/portable/launcher.py", line 449, in _run_executor
executor_output = self._executor_operator.run_executor(execution_info)
File "/usr/local/lib/python3.8/dist-packages/tfx/orchestration/portable/python_executor_operator.py", line 135, in run_executor
return run_with_executor(execution_info, executor)
File "/usr/local/lib/python3.8/dist-packages/tfx/orchestration/portable/python_executor_operator.py", line 58, in run_with_executor
result = executor.Do(execution_info.input_dict, output_dict,
File "/usr/local/lib/python3.8/dist-packages/tfx/components/trainer/executor.py", line 174, in Do
run_fn = udf_utils.get_fn(exec_properties, 'run_fn')
File "/usr/local/lib/python3.8/dist-packages/tfx/components/util/udf_utils.py", line 56, in get_fn
return import_utils.import_func_from_module(module_path, fn_name)
File "/usr/local/lib/python3.8/dist-packages/tfx/utils/import_utils.py", line 59, in import_func_from_module
wheel_context_manager.enter()
File "/usr/local/lib/python3.8/dist-packages/tfx/components/util/udf_utils.py", line 307, in enter
install_to_temp_directory(dependency, temp_dir=self.temp_directory)
File "/usr/local/lib/python3.8/dist-packages/tfx/components/util/udf_utils.py", line 348, in install_to_temp_directory
subprocess.check_call(install_command)
File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python', '-m', 'pip', 'install', '--target', '/tmp/tmp3dt6mzvy', '/tfx/pipelines/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl']' returned non-zero exit status 1.

[ANALYSIS]

1. Run the failed command: /usr/bin/python, -m, pip, install, --target, /tmp/tmp3dt6mzvy, tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl-path-on-host , on host, to see if ok.

$ /home/maye/anaconda3/bin/python -m pip install --target /tmp/tmptse6gaxa pipelines/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl

Processing ./pipelines/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl
Installing collected packages: tfx-user-code-Trainer
Successfully installed tfx-user-code-Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db
$

And, *.whl is zip file, not need setup.py in it.

2. run the failed command in the container, to see if ok.

kubectl exec -it trainer-pod-name -n kubeflow -- bash

--> entered pod trainer:

root@detect-anomolies-on-wafer-tfdv-schema-lnz6m-331885160:/tfx/src# /usr/bin/python -m pip install --target /tmp/tmp3dt6mzvy /tfx/pipelines/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl

ERROR: Directory '/tfx/pipelines/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.

3. see if the needed packages wheel and setuptools for intalling *.whl, exist in the container.

kubectl exec -it trainer-pod-name -n kubeflow -- bash

--> entered pod trainer:

root@detect-anomolies-on-wafer-tfdv-schema-lnz6m-331885160:/usr/bin# /usr/bin/python

Python 3.8.10 (default, Mar 13 2023, 10:26:41) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import wheel
>>> import setuptools
>>> 

4. cd the Directory '*py3-none-any.whl' which is not installable. Neither 'setup.py' nor 'pyproject.toml' found. to see what in it.

kubectl exec -it trainer-pod-name -n kubeflow -- bash

--> entered pod trainer:

root@detect-anomolies-on-wafer-tfdv-schema-lnz6m-331885160:/tfx/src# cd path-of-*py3-none-any.whl 
root@detect-anomolies-on-wafer-tfdv-schema-lnz6m-331885160: path-of *py3-none-any.whl # ls

--> nothing shown

This means Directory *py3-none-any.whl is an empty directory.

5. check the hostPath which mounts to paht-of-*py3-none-any.whl, to see if ok.

There is directory: /home/maye/pipelnes/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl
on host, which is an newly created empty directory.

The volumes.hostPath.path is spelled wrong: "pipelnes", the right one is: "pipelines", in the file pipeline.yaml, and if the volumes.hostPath.path not exists on host, it will be created.
And the true *.whl is File, not Directory.

Attention:
volumeMounts.mountPath can not contain parent directory which does not exists, if so, no error will be raised, bu the mount Path just not work.

[SOLUTION]

The volumes.hostPath.path is spelled wrong: "pipelnes", the right one is: "pipelines", in the file pipeline.yaml.