allow configuring number of vnode in the cluster on startup

[BugenZhao]

In #3579, we discussed whether to adopt a vnode number other than 256 to achieve a better balance between scanning performance and maximal parallelism in a cluster. For quite some time afterwards, we find 256 works pretty well so the task was postponed.

Recently, we have observed some users running a RisingWave cluster with thousands of cores. While this is exciting, the performance of a single streaming job is unsurprisingly limited by the hardcoded vnode count of 256. I think instead of diving into the possibility of runtime dynamic or per-job vnode count, a low-hanging fruit could be allowing users configuring the vnode count on cluster startup as an immutable system parameter.

Thanks to #6513 and #8385, this seems to be made easy. However, it's still worth noting that the vnode count is limited by the algorithm of row-id generator since #9135.

[lmatz]

What's the current behavior of creating a materialized view with default parallelism in a cluster that has more than 256 CPUs? e.g. is it that more than 256 actors will be created but only 256 of them is actually working

[BugenZhao]

The parallelism was intended to be limited by the vnode count of 256, but there appear to be some edge cases. I will address that. 😄

risingwave/src/meta/src/rpc/ddl_controller.rs

Lines 1316 to 1324 in 1e84852

	if available_parallel_units.get() > VirtualNode::COUNT {
	tracing::warn!(
	"Too many parallel units, use {} instead",
	VirtualNode::COUNT
	);
	NonZeroUsize::new(VirtualNode::COUNT).unwrap()
	} else {
	available_parallel_units
	}

[BugenZhao]

Not resolved but only workarounded by #15941. Thus reopen.

[fuyufjh]

Time to work on this!

[fuyufjh]

It's worth mentioning that the vnode has already been written into the state table (as a prefix of the key), thus, I think the vnode number of existing jobs can't be changed anymore. We need to ensure that

1. The existing jobs can run properly with their correct vnode
2. The new jobs will be scheduled according to the new changed global vnode number

[BugenZhao]

2. The new jobs will be scheduled according to the new changed global vnode number

To achieve this, it appears that we are essentially providing support for different vnode count for different jobs. Then doing joins on these MVs can be tricky: we need to reimagine the Distribution in the optimizer to also reflect the number of vnodes. Still doable but I'm not sure whether it's worthwhile, considering 256 is sufficient for most of the customers and configurable vnode count is already a specialized request, not to mention local vnode counts.

A simpler approach is to always treat old clusters with unmodifiable 256 vnodes. Then only fresh clusters can benefit from this change. Note that the value still remains immutable after the cluster initializes.