Consolidate UTF8 character encoding implementations

[am11]

We have two character encoding implementations for Utf8 <-> Utf16 <-> UCS4 in runtime; located at:

• src/coreclr/pal/src/locale/{utf8,unicode}.cpp
• src/mono/mono/eglib/giconv.c

The coreclr/pal implementation was ported from C# implementation in 2016: dotnet/coreclr#3809. The C# implementation has diverged / improved since with spanified and intrinsified APIs so much that the similarities between the two are unrecognizable.

We can move eglib/giconv.c from mono to src/native/minipal, and create a neutral C header, so both coreclr pal and mono glib can rely on unified APIs. There are also some unused methods in these files (e.g. g_utf8_to_ucs4_fast) which can be cleaned up at the same time.

[ghost]

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

Issue Details

---

We have two character encoding implementations for Utf8 <-> Utf16 <-> UCS4 in runtime; located at:

• src/coreclr/pal/src/locale/{utf8,unicode}.cpp
• src/mono/mono/eglib/giconv.c

The coreclr/pal implementation was ported from C# implementation in 2016: dotnet/coreclr#3809. The C# implementation has diverged / improved since with spanified and intrinsified APIs so much that the similarities between the two are unrecognizable.

We can move eglib/giconv.c from mono to src/native/minipal, and create a neutral C header, so both coreclr pal and mono glib can rely on a unified APIs. There are also some unused methods in these files (e.g. g_utf8_to_ucs4_fast) which can be cleaned up at the same time.

Author:	am11
Assignees:	-
Labels:	area-Meta
Milestone:	-

[jkotas]

Sounds good to me. Make sure that the shared implementation is at least as fast as the current one.

UCS4

We should not need that.

[marek-safar]

/cc @SamMonoRT @lateralusX

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-encodings-web
See info in area-owners.md if you want to be subscribed.

Issue Details

---

We have two character encoding implementations for Utf8 <-> Utf16 <-> UCS4 in runtime; located at:

• src/coreclr/pal/src/locale/{utf8,unicode}.cpp
• src/mono/mono/eglib/giconv.c

The coreclr/pal implementation was ported from C# implementation in 2016: dotnet/coreclr#3809. The C# implementation has diverged / improved since with spanified and intrinsified APIs so much that the similarities between the two are unrecognizable.

We can move eglib/giconv.c from mono to src/native/minipal, and create a neutral C header, so both coreclr pal and mono glib can rely on unified APIs. There are also some unused methods in these files (e.g. g_utf8_to_ucs4_fast) which can be cleaned up at the same time.

Author:	am11
Assignees:	-
Labels:	area-System.Text.Encodings.Web, untriaged
Milestone:	-

[am11]

@jeffhandley, it is not related to System.Text.Encodings.Web. This is about consolidating native implementations of encoding used internally by coreclr and mono. Managed libraries don't consume these implementations.

[ghost]

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

Issue Details

---

We have two character encoding implementations for Utf8 <-> Utf16 <-> UCS4 in runtime; located at:

• src/coreclr/pal/src/locale/{utf8,unicode}.cpp
• src/mono/mono/eglib/giconv.c

The coreclr/pal implementation was ported from C# implementation in 2016: dotnet/coreclr#3809. The C# implementation has diverged / improved since with spanified and intrinsified APIs so much that the similarities between the two are unrecognizable.

We can move eglib/giconv.c from mono to src/native/minipal, and create a neutral C header, so both coreclr pal and mono glib can rely on unified APIs. There are also some unused methods in these files (e.g. g_utf8_to_ucs4_fast) which can be cleaned up at the same time.

Author:	am11
Assignees:	-
Labels:	area-Meta, untriaged
Milestone:	-

[jeffhandley]

Ah; thanks, @am11.