Stream doesn't synchronize when operating from a thread?

[thomas-bouvier]

Hello,

First of all, many thanks for this great library. This makes it much easier to work with CUDA APIs :) I have a question that may be orthogonal to this particular library (unless it's a bug)?

I am using an async CUDA stream (cuda::stream::async, with flag cudaStreamNonBlocking), and issuying cudaMemcpy to that stream later. However I am doing that from within a std::thread, and in that scenario, the stream synchronization logic (stream.synchronize() or device.synchronize()) doesn't work properly, i.e., it doesn't behave like a synchronization barrier and copies are not necessarily finished beyond it.

Is that thread + cudaStreamSynchronize scenario supported? Am I using the API correctly?

Here is a snippet reproducing what I have. The ASSERT is failing, i.e., the line before doesn't wait for copies to finish.

void distributed_stream_loader_t::start() {
    // The thread which operate CUDA streams
    cpp_thread = std::thread(&distributed_stream_loader_t::async_process, this);
}

void distributed_stream_loader_t::async_process() {
    // Creation of the CUDA async stream, from within the thread
    auto device = cuda::device::current::get();
    std::generate_n(
        std::back_inserter(m_streams), 1,
        [&device]() {
            return device.create_stream(cuda::stream::async);
        }
    );

    // Infinite loop executing CUDA async copies on the stream
    while (true) {
        ...
        for (size_t j = 0; j < num_targets; j++) {
            cuda::memory::async::copy(
                dest_targets->data_ptr<long int>() + batch.aug_size,
                t_label.data_ptr(),
                t_label.element_size(),
                m_streams[0]
            );
        }
        ...

        // Synchronization of the stream
        m_streams[0].synchronize();
        //or cuda::device::current::get().synchronize();

        // This makes sure that the copies are finished..... this is failing
        ASSERT(async_copies_finished);
    }
}

Thanks!

[thomas-bouvier]

Not related to that library, closing.

[eyalroz]

Hello @thomas-bouvier ,

So, just to make sure - you've reached the conclusion that the problem you're seeing is not due to my wrappers library. Is that correct?

[thomas-bouvier]

Exactly, no problem with your wrappers. Thank you for the hard work!

[eyalroz]

Thanks for clarifying. :-)

By the way:

• It sometimes helps code readability of you define memory regions, or typed spans, for copying. In your case, it might have been something like:

  using cuda::memory::region_t;
  
  // ...
  
  for (size_t j = 0; j < num_targets; j++) {
      region_t source { t_label.data_ptr(), t_label.element_size() };
      region_t dest { dest_targets->data_ptr<long int>() + batch.aug_size, t_label.element_size() };
      m_streams[0].enqueue.copy(dest, source);
      // or using the freestanding function:
      // cuda::memory::async::copy(dest, source, m_streams[0]);
  }
  

• You might even go even further and use ranges, for code that would look something like this:

  auto dest_segments = segmentize_as_a_range_of_regions(dest_targets);
  auto source_segments = segmentize_as_a_range_of_regions(t_label);
  for([dest, source] : std::views::zip(dest_segments, source_segments))
      m_streams[0].enqueue.copy(dest, source);
  }
  

• I am always looking for:

  • People who want to try out betas/release-candidates of new versions (at least to make sure they don't break existing code) of the wrappers.
  • People who are interested in answering design questions/ponderings about the future of (some aspects of) the wrappers.
    ... so if you're interested in helping a bit, drop me a line.

[thomas-bouvier]

Thank you very much for the hints.

I will get back to you in september to try out new features :)