Support for Nvidia Tesla M40 cards? [update] Can work. Don't do it.

[hybridblue]

I had to modify main.py under the section "# Override compute_type if at least one non-Turing card" to change compute_type to float32 otherwise I was unable to get the model running.

Apparently the Tesla M40 does not support int8 and since its cuda_device_capability is '52' and below '70', setting to int8 keeps the system from loading.

I was able to get everything up and running but have yet to test it properly or to get willow pointed to it, I will update with results later.
I found this chart with the supported compute types. I wish I had known before I went out and bout the Tesla card or I would have bought one of the other cards on the list. Maybe this will help someone else as well.

https://docs.nvidia.com/deeplearning/tensorrt/support-matrix/index.html#hardware-precision-matrix

[hybridblue]

Unrelated but also of note. I got this error when running ./download_models.sh
+ ct2-transformers-converter --force --model openai/whisper-large-v2 --quantization float16 --output_dir models/openai-whisper-large-v2 /app/whisper.sh: line 20: 90 Killed ct2-transformers-converter --force --model "$MODEL" --quantization "$QUANT" --output_dir models/"$MODEL_OUT"

After inspecting the logs it turns out the container was running out of memory. I had the VM it is in set to 4GB. Changing that to 8GB allowed the process to complete properly.

[kristiankielhofner]

Did you buy the Tesla M40 specifically for WIS? We only recommend Pascal (device capability 6) series GPUs and up which for a Tesla would be a P4 (P = Pascal). Note that a Tesla is not a Tesla - it's the name of a product line and there are drastic differences between M40, P4, T4, etc.

The Tesla M40 is Maxwell (almost 10 years old) and we have no plans to officially support it. If anything we'll probably add a check for minimum compute 60 (we drop the decimal internally) and disable CUDA if less.

As you note it doesn't support int8/float16 and it's more-or-less end of support - Maxwell is dropped in CUDA 12 (which we will be switching to shortly). It's also just a terrible card all around - dual slot, 250 watts(!!!), horrible performance. The P4 came out roughly a year later, is the same price or less, and is night and day better in every way (VRAM on any of these cards isn't close to an issue).

In short, if you bought it for WIS you should return it (hopefully you can) and get a P4.

[hybridblue]

Unfortunately I did not do good enough research o.0 that happens. I will find a use for it one way or another but it was cheap so there is that.
Im tinkering with getting things running as-is since its what I have but im looking at ordering something else for WIS.

It looks like a P4 8gb card can be had for ~100-150. Any thoughts on future use? It would be a shame to get a P4 and in a year want to upgrade it.

And yeah, the more I read the more that the M40 is not such a great idea.. Apparently they had a problem with the power connectors burning up at some point.

Im not set on a "tesla" card but this would be going into my VM cluster with the primary purpose to be used for WIS or other CUDA workloads. Also not overly concerned on a higher price, more interested in future-proofing for something like WIS.

[kristiankielhofner]

I'd get rid of it if I were you :).

In terms of upgrading, I have no idea where you'll want to go with any other GPU tasks. What I can tell you about GPUs:

• Low power/no fan/server chassis/cheap - P4.
• Best "bang for the buck" in terms of power, speed, etc - GTX 1070 (power consumption is still very low, and my fans don't even activate). Of course if you want even more speed and 50% more VRAM (12GB) you can get a 1080 for a bit more.
• With every voice feature enabled (all Whisper models, TTS, and Speaker Verification) we use just under half of the 8GB available on these cards. We'll also eventually reduce TTS and Speaker Verification model size and let you configure which Whisper models to load. This will likely drop the required VRAM to 2GB or so - leaving plenty for future WIS functionality and any other applications you want way to run.
• Pascal is supported with CUDA 12 (which is relatively new). This likely means three years or so of maintained and up to date software releases from Nvidia (which we build on).
• In terms of ML/AI/etc generally the RTX 3090 is the best deal out there. Extremely fast, 24GB VRAM, can run LLMs, likely supported for another seven-eight years (or so), etc. All-in-all not bad for roughly $1000. Other than testing WIS I personally don't bother to buy anything less but of course these are my professional tools for WIS and other AI/ML work so my situation is a bit different from most WIS users.

[hybridblue]

Thanks for the response and extra info. It sounds like I may end up with a 1080 to start with and if that is no longer good enough for what I am doing I probably need to just go all in and get something like the 3090 or higher. It may be a bit before that is needed though.
Knowing that CUDA 12 should be in support for a few years is more reassuring as well.

[kristiankielhofner]

Generally the Pascal series cards are remarkably capable, readily available, and very cheap. If I wasn't so heavy in GPGPU I couldn't think of anything someone would want to do that can't be done on a 1070/1080/even P4 (with the exception of LLMs, which are a completely different animal).

A Pascal something can easily do WIS, Frigate, Plex, and a bunch of other random stuff concurrently - even StableDiffusion (GTX 1080 for the VRAM), etc.

[GRMrGecko]

FYI, I got the M40 to work with Willow pretty easily, I just changed compute_type = "int8" to compute_type = "default" in main.py and it worked from there. I'd be it eating 14GB of VRAM, but it works. I ended up getting an Nvida P41 as it supports Int8, and was about $50-30 more than the M40.

[kristiankielhofner]

I cannot stress enough how bad of an idea it is to run WIS on an M40.

It's basically the worst of all worlds - high power consumption, very high VRAM usage (float32), very slow, big (dual slot), and soon to be unsupported at all (with CUDA 12).

If suppose if you have one lying around it may offer higher performance than CPU (depending on CPU) but as I told @hybridblue I'll be adding a check to detect < compute 6.0 and disable CUDA in wisng. I don't want to even give the impression that it is officially supported in any way.

[kristiankielhofner]

As a compromise (for now) we'll detect pre-pascal cards, set float32, and heavily warn

[hybridblue]

Loud and clear ;) I have a 1080 on the way and will be using that instead.