3D volume rendering

[luccascorrea]

Hi,

I am trying to implement rendering a volume in the way that is displayed in both videos below (the first video was taken from the 3D Slicer program). I couldn't find anything like this in the docs or the examples. Does anyone have a tip on how this could be implemented? I'd just need someone to point me in a direction so I can investigate this further.

I found this example on the VTK.js website that seems to do what I need but I'm not sure how to do these things through Cornerstone3D. It seems like I need to use a vtkImageMarchingCubes filter inside a viewport but I'm not sure how it's done.

Thank you!

Video 1: https://www.dropbox.com/s/zgn0a7ubugn2jsx/volume_rendering.mp4?dl=0
Video 2: https://www.dropbox.com/s/p0gnoso0u86h1t9/volume.mp4?dl=0

[sedghi]

There are two things here

1. 3D Volume Viewport support: currently our volume viewports are forcing the clipping planes on a tiny tiny slice which is not required for the 3D Volume viewport. So that is not yet implemented. Solution: you can create another volume viewport basically copy pasting the whole thing and just removing the places where we are enforcing the clipping planes
2. Application of filters which is another thing we don't support right now, I guess we need to add api to apply a filter on the image data before feeding it to the mapper (or revise the mapper to get it from the filtered image data)

Happy to review PRs for any of them

[luccascorrea]

Got it. Thank you very much for the information @sedghi. I will study the VolumeViewport source code to figure out the best way to do this.

[luccascorrea]

Hi @sedghi.
I am trying to implement option 1 that you suggested, but haven't had much success. I attempted commenting out some parts of the code in both the VolumeViewport and Viewport classes just to see if it would work before actually creating a new viewport class that implements this new feature.

I'm attaching below an image with the git diff of the changes I made just so you have an idea, but this didn't make much of a difference. The first image shows the changes to the packages/core/src/RenderingEngine/Viewport.ts file and the second one to the packages/core/src/RenderingEngine/VolumeViewport.ts file.

I am using the TrackballRotateTool to rotate the camera but it seems like only a frontal plane is shown (as well as another plane in the back when I rotate the volume 180 degrees). You can see what I'm talking about in the video below (I'm running the tutorial project with a single VolumeViewport to test this out).

3d_volume.mp4

Sorry to disturb you once again, but could you give me a few pointers?

Thank you very much!

[sedghi]

Clipping planes, not clipping range. Remove anywhere we are putting a clipping plane, and let clipping range be actually -infinity to infinity

[luccascorrea]

Hi @sedghi. Unfortunately, it's still not working.

I basically made a change to the resetCamera method in the VolumeViewport class so that the clippingRange is set to -Infinity, Infinity and commented out the code that adds the two clipping planes. I also made a change to the Viewport class so that the clipping range is also set to -Infinity, Infinity in the resetCamera method (which I think is actually irrelevant, as the VolumeViewport class overrides that method and changes the clippingRange right after the super call). You can see the exact changes below:

I still get pretty much the same behavior I described above. It seems like there is only one visible plane in the front and another one in the back. I can't really understand what's going on.

3d_volume_no_clipping_planes.mov

Thanks for the help!

[sedghi]

That is interesting, most likely our tool is wrong, but I don't see why the initial camera should be wrong too. Can you provide a branch?

[sedghi]

I had some time to look at this, see my branch here, which is just very ugly implementation, but you get the idea, if you can create a PR I would be happy to review it

cd packages/tools
yarn run example 3dViewport

[luccascorrea]

Thank you very much @sedghi!
Yes, that's what I was looking for. I will have a look at the code and submit a pull request. I am a bit caught up with some other things but I'll try to submit it next week.

Thank you again!

[luccascorrea]

Hi @sedghi. I reorganized the code and implemented a new type of viewport called VolumeViewport3D. You can see all my changes here.

I basically refactored the existing VolumeViewport class and extracted all common functionality to an abstract base class that I called BaseVolumeViewport. Then both the existing VolumeViewport class and the new VolumeViewport3D extend it.

I tried to make my commits easy to understand but let me know if there's anything you'd like me to improve before I submit the pull request.

[sedghi]

This makes sense to me, can you please go ahead and add the PR (mention this issue as well), and I can review and start the discussion there thanks

[mongsukim]

Hello. I'm a beginner developer.
Thank you for creating such a great library. I'd like to create a 3D volume rendering feature using the cornerstonejs library, similar to what's shown on the https://www.cornerstonejs.org/live-examples/volumeviewport3d page. However, I find the API documentation quite difficult to understand. Would it be possible to see a GitHub repository or example code that implements 3D volume rendering?

[jianyaoo]

Hello. I'm a beginner developer. Thank you for creating such a great library. I'd like to create a 3D volume rendering feature using the cornerstonejs library, similar to what's shown on the https://www.cornerstonejs.org/live-examples/volumeviewport3d page. However, I find the API documentation quite difficult to understand. Would it be possible to see a GitHub repository or example code that implements 3D volume rendering?

@mongsukim Try the implementation in this demo：https://github.com/jianyaoo/vue-cornerstone-demo/blob/main/src/views/basicUsage/Basic3DRender.vue

[mongsukim]

@jianyaoo Thank you very much!
The example you provided is in Vue. I use React, I will try to adapt the code for React

[mongsukim]

I always encounter the same issue. The error message is as follows:

Uncaught (in promise) Error: No volume loader for scheme cornerstoneStreamingImageVolume has been registered

Which part should I fix?"

const volume = await volumeLoader.createAndCacheVolume(volumeId, {
          imageIds,
        });

import React, { useState, useEffect, useRef } from 'react';
import {
  volumeLoader,
  RenderingEngine,
  Enums as csEnums,
  setVolumesForViewports,
  CONSTANTS,
  getRenderingEngine,
} from '@cornerstonejs/core';
import initCornerstone from '../initCornerstone';

import cornerstone from 'cornerstone-core';
import cornerstoneWADOImageLoader from 'cornerstone-wado-image-loader';

// Constants for volume and viewport setup
const volumeName = 'CT_VOLUME_ID';
const volumeLoaderScheme = 'cornerstoneStreamingImageVolume';
const volumeId = `${volumeLoaderScheme}:${volumeName}`;

const renderingEngineId = 'my_renderingEngine';
const viewportId = '3Dvp';

// Preset options for viewport configuration
const presetOptions = CONSTANTS.VIEWPORT_PRESETS.map((item) => item.name);

const CornerstoneViewer = () => {
  const [preset, setPreset] = useState('CT-Bone');

  const [imageIds, setImageIds] = useState([]);
  const [viewport, setViewport] = useState(null);

  const elementRef = useRef(null);

  // Initialize Cornerstone and rendering engine
  useEffect(() => {
    const init = async () => {
      await initCornerstone();

      const renderingEngine = new RenderingEngine(renderingEngineId);
      const viewportInputArray = [
        {
          viewportId: viewportId,
          type: csEnums.ViewportType.VOLUME_3D,
          element: elementRef.current,
          defaultOptions: {
            orientation: csEnums.OrientationAxis.CORONAL,
            background: CONSTANTS.BACKGROUND_COLORS.slicer3D,
          },
        },
      ];
      renderingEngine.setViewports(viewportInputArray);

      if (imageIds.length > 0) {
        const volume = await volumeLoader.createAndCacheVolume(volumeId, {
          imageIds,
        });
        await volume.load();
        await setVolumesForViewports(
          renderingEngine,
          [{ volumeId }],
          [viewportId]
        );

        const viewportInstance = cornerstone.getViewport(elementRef.current);
        setViewport(viewportInstance);

        viewportInstance.setProperties({ preset });
        viewportInstance.render();
      }
    };

    init();

    // Clean up when component unmounts
    return () => {
      if (elementRef.current) {
        cornerstone.disable(elementRef.current);
      }
    };
  }, [imageIds]);

  // Handle changes to viewport properties
  const handleChange = (type, value) => {
    const viewport =
      getRenderingEngine(renderingEngineId).getViewport(viewportId);
    viewport.setProperties({ [type]: value });
    viewport.render();
  };

  // Handle file input changes and process DICOM files
  const handleFileChange = async (event) => {
    const files = event.target.files;
    const imageIdPromises = [];

    if (files) {
      for (const file of files) {
        const imageId =
          cornerstoneWADOImageLoader.wadouri.fileManager.add(file);
        imageIdPromises.push(imageId);
      }

      const resolvedImageIds = await Promise.all(imageIdPromises);
      setImageIds(resolvedImageIds);
    }
  };

  return (
    <div>
      <div className="form">
        <div className="form-item">
          <label className="label">preset：</label>
          <select
            value={preset}
            onChange={(e) => {
              setPreset(e.target.value);
              handleChange('preset', e.target.value);
            }}
            style={{ width: '150px' }}
          >
            {presetOptions.map((item) => (
              <option key={item} value={item}>
                {item}
              </option>
            ))}
          </select>
        </div>

        <div className="form-item">
          <label className="label">file upload：</label>
          <input
            type="file"
            multiple
            onChange={handleFileChange}
            style={{ width: '300px' }}
          />
        </div>
      </div>
      <div id="demo-wrap">
        <div
          ref={elementRef}
          className="cornerstone-item"
          style={{
            position: 'relative',
            width: '300px',
            height: '300px',
            marginTop: '20px',
            marginRight: '20px',
            padding: '20px',
            border: '2px solid #96CDF2',
            borderRadius: '10px',
          }}
        ></div>
      </div>
    </div>
  );
};

export default CornerstoneViewer;

I apologize if the question was inappropriate.

[jianyaoo]

Did you execute the registerVolumeLoader register function inside the initCornerstone function?

import { volumeLoader } from "@cornerstonejs/core";
import {
  cornerstoneStreamingImageVolumeLoader,
  cornerstoneStreamingDynamicImageVolumeLoader,
} from "@cornerstonejs/streaming-image-volume-loader";

export default function initVolumeLoader() {
  volumeLoader.registerUnknownVolumeLoader(
    cornerstoneStreamingImageVolumeLoader
  );
  volumeLoader.registerVolumeLoader(
    "cornerstoneStreamingImageVolume",
    cornerstoneStreamingImageVolumeLoader
  );
  
  volumeLoader.registerVolumeLoader(
    "cornerstoneStreamingDynamicImageVolume",
    cornerstoneStreamingDynamicImageVolumeLoader
  );
}

other， I'm not sure if cornerstone-core and @cornerstonejs/core can be used together, but I recommend harmonizing cornerstone versions