ParalleRunner hangs on Linux Server

[Dekermanjian]

Description

I have a pipeline that I would like to run using the ParallelRunner. When I run this pipeline on my local windows machine it works just fine. However, when I try running the exact same pipeline on a Linux server (Rocky Linux) it will just hang at the loading datasets stage.

• Kedro version used (pip show kedro or kedro -V): 0.19.8
• Python version used (python -V): 3.11.8
• Operating system and version: Rocky Linux version 8.10

[noklam]

Can you provide some more context, if possible to share a simplified version of repository that we can try to reproduce locally.

[Dekermanjian]

@noklam Yeah, of course. Let me try to put together something simple that will hang on the server and then I'll share the repo with you.

[Dekermanjian]

@noklam Okay, I figured out why it is not working. I just don't understand why it doesn't work on Linux but it does on Windows. Here is a simple example: https://github.com/Dekermanjian/test-parallel-runner

The reason it is not working on the linux server is because I am loading a parquet file in my settings.py file. When I load that file in the simple example the ParallelRunner will hang at the loading dataset stage. If you comment that line out (line 6) then it will work. You can generate the data by running the notebook I created.

Sorry let me add the command to run: kedro run --runner=ParallelRunner -p data_processing