Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TypeError: Super constructor null of nJ is not a constructor (for ImageSegmenter initialization) #5638

Open
vastava opened this issue Sep 20, 2024 · 0 comments
Assignees
Labels
platform:javascript MediaPipe Javascript issues task:image segmentation Issues related to image segmentation: Locate objects and create image masks with labels type:support General questions

Comments

@vastava
Copy link

vastava commented Sep 20, 2024

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

Yes

OS Platform and Distribution

Web Browser (Chrome)

MediaPipe Tasks SDK version

@mediapipe/tasks-vision@0.10.2

Task name (e.g. Image classification, Gesture recognition etc.)

Image Segmenter

Programming Language and version (e.g. C++, Python, Java)

Javascript

Describe the actual behavior

Error initializing ImageSegmenter in production, although it works locally.

Describe the expected behaviour

Successful initialization of ImageSegmenter and ability to perform image segmentation on webcam feed

Standalone code/steps you may have used to try to get what you need

import React, { useRef, useEffect, useState } from "react";
import Webcam from "react-webcam";
import { ImageSegmenter, FilesetResolver } from "@mediapipe/tasks-vision";

const legendColors = [
  [255, 197, 0, 255], // Vivid Yellow
  [128, 62, 117, 255], // Strong Purple
  // ... (rest of the colors)
];

const ImageSegmentationComponent = () => {
  const webcamRef = useRef(null);
  const canvasRef = useRef(null);
  const [imageSegmenter, setImageSegmenter] = useState(null);
  const [labels, setLabels] = useState([]);

  useEffect(() => {
    console.log("Initializing ImageSegmenter...");
    async function initializeImageSegmenter() {
      try {
        console.log("Creating FilesetResolver...");
        const filesetResolver = await FilesetResolver.forVisionTasks(
          "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.12/wasm"
        );
        console.log("FilesetResolver created successfully", filesetResolver);
        try {
            console.log("Creating ImageSegmenter...");
//code fails at this step
            const segmenter = await ImageSegmenter.createFromOptions(filesetResolver, {
              baseOptions: {
                // modelAssetPath: "https://storage.googleapis.com/mediapipe-models/image_segmenter/deeplab_v3/float32/1/deeplab_v3.tflite",
                modelAssetPath: "https://storage.googleapis.com/mediapipe-models/image_segmenter/hair_segmenter/float32/latest/hair_segmenter.tflite",
                delegate: "GPU"
              },
              runningMode: "VIDEO",
              outputCategoryMask: true,
              outputConfidenceMasks: false
            });
            console.log("ImageSegmenter created successfully");
            setImageSegmenter(segmenter);
          } catch (error) {
            console.error("Error initializing ImageSegmenter:", error);
            // Handle the error, perhaps by setting an error state
          }          
      } catch (error) {
        console.error("Error initializing ImageSegmenter:", error);
      }
    }

    initializeImageSegmenter();
  }, []);

  useEffect(() => {
    if (imageSegmenter && webcamRef.current && canvasRef.current) {
      console.log("Setting up video processing...");
      const video = webcamRef.current.video;
      const canvas = canvasRef.current;
      const ctx = canvas.getContext("2d");

      let lastVideoTime = -1;

      function processFrame() {
        if (video.currentTime !== lastVideoTime) {
          lastVideoTime = video.currentTime;
          canvas.width = video.videoWidth;
          canvas.height = video.videoHeight;
          ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
          
        //   if (webcamRunning) {
            console.log("Processing video frame...");
            imageSegmenter.segmentForVideo(video, performance.now(), (result) => {
              console.log("Segmentation result received");
              onResults(result, ctx, canvas.width, canvas.height);
            });
        //   }
        }
        requestAnimationFrame(processFrame);
      }

      processFrame();
    }
  }, [imageSegmenter]);

  const onResults = (result, ctx, width, height) => {
    console.log("Processing segmentation results...");
    let imageData = ctx.getImageData(0, 0, width, height).data;
    const mask = result.categoryMask.getAsFloat32Array();
    
    let j = 0;
    for (let i = 0; i < mask.length; ++i) {
      const maskVal = Math.round(mask[i] * 255.0);
      const legendColor = legendColors[maskVal % legendColors.length];
      imageData[j] = (legendColor[0] + imageData[j]) / 2;
      imageData[j + 1] = (legendColor[1] + imageData[j + 1]) / 2;
      imageData[j + 2] = (legendColor[2] + imageData[j + 2]) / 2;
      imageData[j + 3] = (legendColor[3] + imageData[j + 3]) / 2;
      j += 4;
    }

    const uint8Array = new Uint8ClampedArray(imageData.buffer);
    const dataNew = new ImageData(uint8Array, width, height);
    ctx.putImageData(dataNew, 0, 0);
    console.log("Segmentation overlay drawn");
  };

  return (
    <div>
      <Webcam
        audio={false}
        mirrored={true}
        ref={webcamRef}
        onLoadedMetadata={() => console.log("Webcam loaded")}
        style={{
          position: "absolute",
          marginLeft: "auto",
          marginRight: "auto",
          left: 0,
          right: 0,
          textAlign: "center",
          zIndex: 9,
          width: 640,
          height: 480,
        }}
      />
      <canvas
        ref={canvasRef}
        style={{
          position: "absolute",
          marginLeft: "auto",
          marginRight: "auto",
          left: 0,
          right: 0,
          textAlign: "center",
          zIndex: 9,
          width: 640,
          height: 480,
        }}
      />
    </div>
  );
};

export default ImageSegmentationComponent;

Also, since this is a nextjs project, I'm dynamically importing the component like so:

const ImageSegComponent = dynamic(() => import('../../../components/imageseg'), {
    ssr: false
  });

Other info / Complete Logs

The ImageSegmentationComponent component I created works perfectly in local environment, but breaks in production. Specifically is not able to initialize ImageSegmenter class.

Also, interestingly, when I swap out the model for the FaceLandmarker model and change the params accordingly, the component works perfectly. I am not sure why this would be the case.

Here is the full error trace I see in the console (it is not very informative):
TypeError: Super constructor null of nJ is not a constructor
at new nJ (211.25927c0cdec0fb60.js:1:49860)
at ie (211.25927c0cdec0fb60.js:1:17442)
at tX.Ie [as aa] (211.25927c0cdec0fb60.js:1:22081)
at tX.get (211.25927c0cdec0fb60.js:1:21885)
at 211.25927c0cdec0fb60.js:1:21489
at tX.forEach ()
at tX.forEach (211.25927c0cdec0fb60.js:1:21459)
at 211.25927c0cdec0fb60.js:1:135863
at rB.J (211.25927c0cdec0fb60.js:1:135904)
at 211.25927c0cdec0fb60.js:1:57803
console. @ watchConsoleLogs.8258a5d6.js:6
overrideMethod @ react_devtools_backend_compact.js:13851
(anonymous) @ 893.99515cd9419e7cfc.js:1
await in (anonymous)
(anonymous) @ 893.99515cd9419e7cfc.js:1
Rj @ framework-0c7baedefba6b077.js:9
Ik @ framework-0c7baedefba6b077.js:9
(anonymous) @ framework-0c7baedefba6b077.js:9
Qk @ framework-0c7baedefba6b077.js:9
Fk @ framework-0c7baedefba6b077.js:9
jg @ framework-0c7baedefba6b077.js:9
(anonymous) @ framework-0c7baedefba6b077.js:9

@kuaashish kuaashish added platform:javascript MediaPipe Javascript issues task:image segmentation Issues related to image segmentation: Locate objects and create image masks with labels type:support General questions labels Sep 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
platform:javascript MediaPipe Javascript issues task:image segmentation Issues related to image segmentation: Locate objects and create image masks with labels type:support General questions
Projects
None yet
Development

No branches or pull requests

2 participants