[EPIC] Automatically generate all function content from code

[alamb]

Is your feature request related to a problem or challenge?

Currently, when we add a new function to DataFusion library we have to remember to document that function in the documentation, but currently we have to remember this during code review.

This has a few downsides:

1. we have forgotten to document some functions
2. the documentation has drifted over time,
3. help text for various functions can only be found on the DataFusion website, and not, for example within the function itself.

@Omega359 has created the basic framework for automatically generating content in #12668

This ticket tracks the work required to port the rest of the documentation to programatic form

Tasks

• Automatically generate function documentation from comments / code #12432
• Improve description of function migration #12743
• Remove redundant aggregate/window/scalar function documentation #12745
• Migrate all existing UDF's to the new format (see below)
• Remove old static function pages #12741
• Add support for DESCRIBE xyz; where xyz is a function. Ideally this would support not just the core functions in DF but also any added externally (for example those in https://github.com/datafusion-contrib/datafusion-functions-extra) and verify it works in the CLI
• Look into generating the function sections in expressions.md (it's a bit out of date)
• Make generate_function_docs.sh error if there is a function without documentation

Aggregate Function Documentation Migration

• Migrate documentation for aggregate functions from aggregate_functions.md to code #12827
• Port / Add Documentation for BitwiseOperation
• Port / Add Documentation for VarianceSample Port / Add Documentation for VarianceSample and VariancePopulation #12742
• Port / Add Documentation for VariancePopulation Port / Add Documentation for VarianceSample and VariancePopulation #12742

Scalar Function Documentation Migration

• [EPIC] Improve examples to make them easier to navigate #11172
• Migrate documentation for all string functions from scalar_functions.md to code  #12775
• Migrate documentation for all array functions from scalar_functions.md to code
• Migrate documentation for all crypto functions from scalar_functions.md to code #12828
• Migrate documentation for all datetime functions from scalar_functions.md to code #12859
• Migrate documentation for all encoding functions from scalar_functions.md to code #12858
• Migrate documentation for all math functions from scalar_functions.md to code
• Migrate documentation for all regex functions from scalar_functions.md to code
• Migrate documentation for all string functions from scalar_functions.md to code
• Migrate documentation for all unicode functions from scalar_functions.md to code
• Port / Add Documentation for ArrayToString
• Port / Add Documentation for ArrayHasAll
• Port / Add Documentation for MapExtract
• Port / Add Documentation for ArrayResize
• Port / Add Documentation for MapValuesFunc
• Port / Add Documentation for ArrayReplaceAll
• Port / Add Documentation for ArrayReplace
• Port / Add Documentation for ArrayRemoveN
• Port / Add Documentation for ArrayRemove
• Port / Add Documentation for GenSeries
• Port / Add Documentation for ArrayLength
• Port / Add Documentation for ArrayDistinct
• Port / Add Documentation for ArrayUnion
• Port / Add Documentation for ArrayRepeat
• Port / Add Documentation for ArrayPopFront
• Port / Add Documentation for ArrayElement
• Port / Add Documentation for ArrayConcat
• Port / Add Documentation for ArrayPrepend
• Port / Add Documentation for ArrayDims
• Port / Add Documentation for MakeArray
• Port / Add Documentation for ArrayReverse
• Port / Add Documentation for ArraySort
• Port / Add Documentation for ArrayAppend
• Port / Add Documentation for ArrayEmpty
• Port / Add Documentation for ArrayNdims
• Port / Add Documentation for ArrayDistance
• Port / Add Documentation for MapFunc
• Port / Add Documentation for ArraySlice
• Port / Add Documentation for ArrayPopBack
• Port / Add Documentation for ArrayAnyValue
• Port / Add Documentation for ArrayIntersect
• Port / Add Documentation for Flatten
• Port / Add Documentation for Range
• Port / Add Documentation for ArrayPosition
• Port / Add Documentation for ArrayPositions
• Port / Add Documentation for ArrayRemoveAll
• Port / Add Documentation for ArrayReplaceN
• Port / Add Documentation for MapKeysFunc
• Port / Add Documentation for ArrayExcept
• Port / Add Documentation for Cardinality
• Port / Add Documentation for ArrayHas
• Port / Add Documentation for ArrayHasAny
• Port / Add Documentation for StringToArray
• Port / Add Documentation for NVL2Func
• Port / Add Documentation for ArrowTypeOfFunc
• Port / Add Documentation for VersionFunc
• Port / Add Documentation for NullIfFunc
• Port / Add Documentation for ArrowCastFunc
• Port / Add Documentation for NVLFunc
• Port / Add Documentation for GetFieldFunc
• Port / Add Documentation for StructFunc
• Port / Add Documentation for NamedStructFunc
• Port / Add Documentation for CoalesceFunc
• Port / Add Documentation for MakeDateFunc
• Port / Add Documentation for ToTimestampMillisFunc
• Port / Add Documentation for ToTimestampFunc
• Port / Add Documentation for FromUnixtimeFunc
• Port / Add Documentation for CurrentDateFunc
• Port / Add Documentation for ToUnixtimeFunc
• Port / Add Documentation for ToCharFunc
• Port / Add Documentation for DatePartFunc
• Port / Add Documentation for ToLocalTimeFunc
• Port / Add Documentation for DateBinFunc
• Port / Add Documentation for CurrentTimeFunc
• Port / Add Documentation for NowFunc
• Port / Add Documentation for ToDateFunc
• Port / Add Documentation for ToTimestampSecondsFunc
• Port / Add Documentation for ToTimestampMicrosFunc
• Port / Add Documentation for ToTimestampNanosFunc
• Port / Add Documentation for DateTruncFunc
• Port / Add Documentation for EncodeFunc
• Port / Add Documentation for DecodeFunc
• Port / Add Documentation for NanvlFunc
• Port / Add Documentation for GcdFunc
• Port / Add Documentation for LogFunc
• Port / Add Documentation for IsZeroFunc
• Port / Add Documentation for LcmFunc
• Port / Add Documentation for AbsFunc
• Port / Add Documentation for PiFunc
• Port / Add Documentation for SignumFunc
• Port / Add Documentation for FactorialFunc
• Port / Add Documentation for RoundFunc
• Port / Add Documentation for IsNanFunc
• Port / Add Documentation for TruncFunc
• Port / Add Documentation for RandomFunc
• Port / Add Documentation for CotFunc
• Port / Add Documentation for PowerFunc
• Port / Add Documentation for RegexpMatchFunc
• Port / Add Documentation for RegexpLikeFunc
• Port / Add Documentation for RegexpReplaceFunc
• Port / Add Documentation for OverlayFunc
• Port / Add Documentation for SplitPartFunc
• Port / Add Documentation for RtrimFunc
• Port / Add Documentation for UpperFunc
• Port / Add Documentation for UuidFunc
• Port / Add Documentation for LtrimFunc
• Port / Add Documentation for EndsWithFunc
• Port / Add Documentation for InitcapFunc
• Port / Add Documentation for ToHexFunc
• Port / Add Documentation for ConcatWsFunc
• Port / Add Documentation for ContainsFunc
• Port / Add Documentation for RepeatFunc
• Port / Add Documentation for ConcatFunc
• Port / Add Documentation for LowerFunc
• Port / Add Documentation for BTrimFunc
• Port / Add Documentation for StartsWithFunc
• Port / Add Documentation for ReplaceFunc
• Port / Add Documentation for BitLengthFunc
• Port / Add Documentation for ChrFunc
• Port / Add Documentation for OctetLengthFunc
• Port / Add Documentation for LevenshteinFunc
• Port / Add Documentation for AsciiFunc
• Port / Add Documentation for CharacterLengthFunc
• Port / Add Documentation for RPadFunc
• Port / Add Documentation for LPadFunc
• Port / Add Documentation for StrposFunc
• Port / Add Documentation for RightFunc
• Port / Add Documentation for ReverseFunc
• Port / Add Documentation for SubstrFunc
• Port / Add Documentation for SubstrIndexFunc
• Port / Add Documentation for FindInSetFunc
• Port / Add Documentation for LeftFunc
• Port / Add Documentation for TranslateFunc

Window Function Documentation Migration

• Port / Add Documentation for Lead
• Port / Add Documentation for RowNumber

[alamb]

Update is I have a few follow on PRs

• Remove old static function pages #12741
• Automatically generate function documentation from comments / code #12432

Then I figure I will file one or two other tickets to see how easy it is to port other functions

And if all goes well I'll make a ticket storm to port the docs

[Omega359]

FYI I've finished up the string expression migration locally - just waiting on the updated code from #12742 to land before pushing a PR.

[alamb]

FYI I've finished up the string expression migration locally - just waiting on the updated code from #12742 to land before pushing a PR.

Thanks, I'll merge those as soon as they get a review (I can't merge them until they get an approval from another committer)

[Omega359]

No rush I just wanted to make sure no one was duplicating work needlessly