Revert previous attempt at Triton patch; use CustomCacheManger approach instead.

[tdoublep]

I tested the previous fix for the Triton cache collision issue (see: #34) and it didn't work.

I now see errors like:

FileNotFoundError: [Errno 2] No such file or directory: '/home/vllm/.triton/cache/1feb415f3280ca46eea8c4407a58c23e/fused_moe_kernel.json.tmp.pid_72_c0a0033e-6147-4520-ae3a-3847d02598f8'

which now shows the uuid instead of a random integer, but problem remains.

This PR implements a different workaround, proposed by @cyang49, that tells Triton to use a custom cache manager which assigns a different directory based on the process id.

This time I have tested it and it seems to work.

[njhill]

Thanks @tdoublep!

Would be good for @tjohnson31415 to review too

[tjohnson31415]

I did a little investigation into this issue; this is what I found:

The stacktrace pointed to the error as coming from CompiledKernel.init() where the code lists the files and then tries to read each of them (REF). Those files are created as artifacts of the compilation process in the code here.

What is interesting is that the error includes .tmp.pid in the name which is a temporary file that the compilation is written to before doing an os.replace. I think the issue is that, between listing the files and trying to read the files, the .tmp.pid file is oc.replaced by another process; hence leading to a FileNotFoundError. Even with unique filenames for each process, multiple processes are using the same cache directory.

So a fix in the Triton code could be to remove the temporary files from the listing or to have the temp files created in a separate directory. That way, the compiled artifacts can still be shared across processes (though with some redundant compilations).

That said, I think this fix with a custom cache manager giving each process its own directory will also work and is a good work-around until a fix can be merged into Triton.