[mono][tasks] Extract EmitWasmBundle into generic EmitBundle task and…

… enable bundling in mono self-contained library (#84191)

* [tasks] Move EmitWasmBundle out of wasm specific task

* [tasks] Make EmitBundle more generic

* [tasks] Generate EmitBundle task and replace invocations of old task

* Extend preloading assemblies with mono_assembly_open

* [LibraryBuilder] Integrate bundling

* Refactor LibraryBuilder targets to isolate Bundling

* [mono] Add mono_get_bundled_assembly in preparation for bundled runtimeconfig data

* [mono][librarymode] Bundle runtimeconfig.bin

* Address feedback

Redesign bundling apis to handle various types of resources

Move bundle registering into generated function and handle all forms of bundle registering there

Move bundling api into assembly.c and prototypes into assembly-internals.h

Rename assemblies to resources in bundling context

Include EmitBundle task in wasm nuget

Declare macro for case invariant string comparison

Add documentation for bundling apis

* Consolidate runtimeconfig functions

* [LibraryBuilder] Enable bundling by default

* Handle pdb bundling

* Default EmitBundle callback function to mono_add_bundled_resource

* Combine bundled resource types into generic bundled data

* Add api to free memory dynamically allocated for the bundled resources hash table

* Prepend enum and entries with mono

* Move bundling api into separate source and header

* Fix source file name

* Prefix struct with Mono

* Integrate new bundling api into mono debug

* Fix types in autoinit

* [mono] Base bundling apis around preallocation of MonoBundled*Resource structs

* [mono] Clean up bundled assembly and satellite assembly search

* [mono] Incorporate new bundling APIs into bundled resource retrievals

* [EmitBundle][task] Add templates for bundled source files

Templates added:
- header mirroring bundled-resources-internals.h and MonoBundledSatelliteAssembly
- source file template for each resource's byte data and size
- individual templates to build preallocated MonoBundled*Resource structs
- source file template to hold preallocated MonoBundled*Resource structs

* [EmitBundle] Enable consolidation of bundled resource symbols

* [EmitBundle] Generate bundled source file to preallocate MonoBundled*Resources

* [EmitBundle] Output symbol to resource type map for populating function to add bundled resources

* [LibraryBuilder] Generate source file to add preallocated MonoBundled*Resources

* [LibraryBuilder] Incorporate EmitBundle templates into targets flow

* [EmitBundle] Move bundle registration back into EmitBundle
[EmitBundle] Leverage Emit for file generation

* Update invocations of EmitBundle task

* Address feedback

Migrate logic to check hash table for assemblies or satellite assemblies back to bundled-resources.c
Prepend apis with consistent mono_bundled_resources prefix

* [mono] Prefer C99 compliant types for partially exposed APIs

* Address various fixups

Remove unused function
Remove unnecessary string duplication
Add length checks for comparisons
Fix spacing in MonoBundledResource structs
Avoid g_hash_table_lookup_extended precondition macro assertion fail

* Combine preallocation registration and rename to templates

* Prefer symbol data over symfile

* Differentiate resource name and symbol

* Update invocations of EmitBundle

* Add inlined resource getters

* Add unique id field for each MonoBundledResource

* Include culture information for satellite resources

* Generate preallocation and registration source file iff BundleFile provided

* Generate output itemgroup detailing resource data

* Leverage new bundling api from old bundling api and deprecate

* [mono][debug] Migrate symbol data registration to new bundling api

* Remove symbol data from MonoBundleSatelliteAssemblyResource

* Directly determine resource type

* Update bundled header and switch to non internal types

Update bundled header to declare resource symbols and registration function
Switch types from mono internal types to C99 compliant as bundling apis are partially exposed to public

* Update Autoinitialization logic to leverage bundled runtimeconfig symbols

* Integrate header into bundled source

* Add functions to free allocated MonoBundledResources

* Fixup Leverage new bundling api from old bundling api and deprecate

* Fixup Address various fixups

* Remove unnecessary includes

* Fix trailing commas in structs and extra lines in generated files

* Move specific bundled resource getters into source file

* Various cleanups

Instantiating boolean variables
More direct insertion into hash table
Removing unnecessary assertions
Proper type casting
Preferring symbol_data over symfile
Adding void to prototype functions arg
Fix indentation and spacing
Prefer id over name

* Cleanup logic surrounding EmitBundle task invocations

* Rename output metadata to DataLenSymbol and DataLenSymbolValue

* Simplify bundled resource hashtable insertion

* Cleanup variable instantiation

* Fixup mono-debug

* Cleanup header

* fixup wasi

* fixup template format

* Add flexibility to auto initialize runtime without bundled runtimeconfig

* Various fixups

* Use g_free for memory allocated through glib apis

* Move condition end tag into LibraryBuilder to avoid empty line and cluttered line

* Add bundled resource getters to access data values without MonoBundledResource structs

* Emit one object file for single resource data source file mode

* Rename resource size symbol

* Use public friendly API in file linking towards runtime

* Cleanup types populated through bundled resource getters

* Move extern block completely to LibraryBuilder side

* Update EmitBundle task invocations to use BundleRegistrationFunctionName

* Refactor EmitBundle to account for timezone duplicate resource contents

* wip

* wip

* WIP

* [mono] Alias assemblies with known extension in bundled resources hashtable

* [tasks] Calculate destination files within EmitBundle task itself

* Update invocations of EmitBundle to grab DestinationFiles from output

* Guard webcil specific logic

* Include OutputDirectory path into output destinationFiles

* Fix debugger tests and memory leak

* Various fixups and Documentation

* Try adding resources with known assembly extensions as dll

* Revert to custom hash and equal without WEBCIL guard
This reverts commit 0e33442.

* Consolidate file and bundledResource variables

* Add culture metadata to SatelliteAssembly resources

* Truncate Encoding for readability

* Enable custom hash for bundling resources on wasm

* Leverage new bundling api for wasm wasi drivers

* Add helper to dynamically add bundled assembly and satellite_assembly resources

* Limit when bundled resources can be added and freed

* Restrict MonoBundledResource getters

* Restrict exposure of underlying MonoBundledAssemblyResource modification

* WIP

* Free function chaining.

* Fix crash when output parameters are NULL.

* Fix WASM/WASI build errors.

* Align bundled resources registration struct

* [task] Check PE metadata to determine resource type

* Cleanup Satellite Assembly bundling

* Address Feedback

* Split MonoBundledAssemblyResource getter

* Addressing more feedback

* Validate OutputDirectory

---------

Co-authored-by: Larry Ewing <lewing@microsoft.com>
Co-authored-by: lateralusX <lateralusx.github@gmail.com>

src/mono/mono/component/mini-wasm-debugger.c

@@ -5,6 +5,7 @@
 #include <mono/metadata/mono-debug.h>
 #include <mono/metadata/assembly.h>
 #include <mono/metadata/assembly-internals.h>
+#include <mono/metadata/bundled-resources-internals.h>
 #include <mono/metadata/metadata.h>
 #include <mono/metadata/metadata-internals.h>
 #include <mono/metadata/mono-endian.h>
@@ -198,7 +199,14 @@ assembly_loaded (MonoProfiler *prof, MonoAssembly *assembly)
 		return;
 	}
 
-	if (mono_wasm_assembly_already_added(assembly->aname.name))
+	gboolean already_loaded = mono_bundled_resources_get_assembly_resource_values (assembly->aname.name, NULL, NULL);
+	if (!already_loaded && !g_str_has_suffix (assembly->aname.name, ".dll")) {
+		char *assembly_name_with_extension = g_strdup_printf ("%s.dll", assembly->aname.name);
+		already_loaded = mono_bundled_resources_get_assembly_resource_values (assembly_name_with_extension, NULL, NULL);
+		g_free (assembly_name_with_extension);
+	}
+
+	if (already_loaded)
 		return;
 
 	if (mono_has_pdb_checksum ((char *) assembly_image->raw_data, assembly_image->raw_data_len)) { //if it's a release assembly we don't need to send to DebuggerProxy
@@ -435,10 +443,12 @@ mono_wasm_send_dbg_command (int id, MdbgProtCommandSet command_set, int command,
 		}
 		else
 		{
+			const unsigned char* assembly_bytes = NULL;
 			unsigned int assembly_size = 0;
-			int symfile_size = 0;
-			const unsigned char* assembly_bytes = mono_wasm_get_assembly_bytes (assembly_name, &assembly_size);
-			const unsigned char* pdb_bytes = mono_get_symfile_bytes_from_bundle (assembly_name, &symfile_size);
+			mono_bundled_resources_get_assembly_resource_values (assembly_name, &assembly_bytes, &assembly_size);
+			const unsigned char* pdb_bytes = NULL;
+			unsigned int symfile_size = 0;
+			mono_bundled_resources_get_assembly_resource_symbol_values (assembly_name, &pdb_bytes, &symfile_size);
 			m_dbgprot_buffer_init (&buf, assembly_size + symfile_size);
 			m_dbgprot_buffer_add_byte_array (&buf, (uint8_t *) assembly_bytes, assembly_size);
 			m_dbgprot_buffer_add_byte_array (&buf, (uint8_t *) pdb_bytes, symfile_size);

src/mono/mono/metadata/CMakeLists.txt

@@ -48,6 +48,8 @@ set(metadata_common_sources
     appdomain-icalls.h
     assembly.c
     assembly-internals.h
+    bundled-resources.c
+    bundled-resources-internals.h
     cil-coff.h
     class.c
     class-getters.h

src/mono/mono/metadata/assembly.c

@@ -18,6 +18,7 @@
 #include <stdlib.h>
 #include <mono/metadata/assembly.h>
 #include "assembly-internals.h"
+#include <mono/metadata/bundled-resources-internals.h>
 #include <mono/metadata/image.h>
 #include "image-internals.h"
 #include "object-internals.h"
@@ -81,11 +82,6 @@ mono_assemblies_unlock (void)
 	mono_os_mutex_unlock (&assemblies_mutex);
 }
 
-/* If defined, points to the bundled assembly information */
-static const MonoBundledAssembly **bundles;
-
-static const MonoBundledSatelliteAssembly **satellite_bundles;
-
 /* Class lazy loading functions */
 static GENERATE_TRY_GET_CLASS_WITH_CACHE (debuggable_attribute, "System.Diagnostics", "DebuggableAttribute")
 
@@ -711,7 +707,8 @@ mono_assembly_get_assemblyref (MonoImage *image, int index, MonoAssemblyName *an
 static MonoAssembly *
 search_bundle_for_assembly (MonoAssemblyLoadContext *alc, MonoAssemblyName *aname)
 {
-	if (bundles == NULL && satellite_bundles == NULL)
+	if (!mono_bundled_resources_contains_assemblies () &&
+		!mono_bundled_resources_contains_satellite_assemblies ())
 		return NULL;
 
 	MonoImageOpenStatus status;
@@ -794,15 +791,15 @@ netcore_load_reference (MonoAssemblyName *aname, MonoAssemblyLoadContext *alc, M
 		}
 	}
 
-	if (bundles != NULL && !is_satellite) {
+	if (mono_bundled_resources_contains_assemblies () && !is_satellite) {
 		reference = search_bundle_for_assembly (mono_alc_get_default (), aname);
 		if (reference) {
 			mono_trace (G_LOG_LEVEL_DEBUG, MONO_TRACE_ASSEMBLY, "Assembly found in the bundle: '%s'.", aname->name);
 			goto leave;
 		}
 	}
 
-	if (satellite_bundles != NULL && is_satellite) {
+	if (mono_bundled_resources_contains_satellite_assemblies () && is_satellite) {
 		// Satellite assembly byname requests should be loaded in the same ALC as their parent assembly
 		size_t name_len = strlen (aname->name);
 		char *parent_name = NULL;
@@ -1451,47 +1448,19 @@ absolute_dir (const gchar *filename)
 	return res;
 }
 
-static gboolean
-bundled_assembly_match (const char *bundled_name, const char *name)
-{
-#ifndef ENABLE_WEBCIL
-	return strcmp (bundled_name, name) == 0;
-#else
-	if (strcmp (bundled_name, name) == 0)
-		return TRUE;
-	/* if they want a .dll and we have the matching .webcil, return it */
-	if (g_str_has_suffix (bundled_name, ".webcil") && g_str_has_suffix (name, ".dll")) {
-		size_t bprefix = strlen (bundled_name) - strlen (".webcil");
-		size_t nprefix = strlen (name) - strlen (".dll");
-		if (bprefix == nprefix && strncmp (bundled_name, name, bprefix) == 0)
-			return TRUE;
-	}
-	/* if they want a .dll and we have the matching .wasm webcil-in-wasm, return it */
-	if (g_str_has_suffix (bundled_name, MONO_WEBCIL_IN_WASM_EXTENSION) && g_str_has_suffix (name, ".dll")) {
-		size_t bprefix = strlen (bundled_name) - strlen (MONO_WEBCIL_IN_WASM_EXTENSION);
-		size_t nprefix = strlen (name) - strlen (".dll");
-		if (bprefix == nprefix && strncmp (bundled_name, name, bprefix) == 0)
-			return TRUE;
-	}
-	return FALSE;
-#endif
-}
-
 static MonoImage *
-open_from_bundle_internal (MonoAssemblyLoadContext *alc, const char *filename, MonoImageOpenStatus *status, gboolean is_satellite)
+open_from_bundle_internal (MonoAssemblyLoadContext *alc, const char *filename, MonoImageOpenStatus *status)
 {
-	if (!bundles)
+	if (!mono_bundled_resources_contains_assemblies ())
 		return NULL;
 
 	MonoImage *image = NULL;
-	char *name = is_satellite ? g_strdup (filename) : g_path_get_basename (filename);
-	for (int i = 0; !image && bundles [i]; ++i) {
-		if (bundled_assembly_match (bundles[i]->name, name)) {
-			// Since bundled images don't exist on disk, don't give them a legit filename
-			image = mono_image_open_from_data_internal (alc, (char*)bundles [i]->data, bundles [i]->size, FALSE, status, FALSE, name, NULL);
-			break;
-		}
-	}
+	char *name = g_path_get_basename (filename);
+
+	const uint8_t *data = NULL;
+	uint32_t size = 0;
+	if (mono_bundled_resources_get_assembly_resource_values (name, &data, &size))
+		image = mono_image_open_from_data_internal (alc, (char *)data, size, FALSE, status, FALSE, name, NULL);
 
 	g_free (name);
 	return image;
@@ -1500,22 +1469,18 @@ open_from_bundle_internal (MonoAssemblyLoadContext *alc, const char *filename, M
 static MonoImage *
 open_from_satellite_bundle (MonoAssemblyLoadContext *alc, const char *filename, MonoImageOpenStatus *status, const char *culture)
 {
-	if (!satellite_bundles)
+	if (!mono_bundled_resources_contains_satellite_assemblies ())
 		return NULL;
 
 	MonoImage *image = NULL;
-	char *name = g_strdup (filename);
+	char *bundle_name = g_strconcat (culture, "/", filename, (const char *)NULL);
 
-	for (int i = 0; !image && satellite_bundles [i]; ++i) {
-		if (bundled_assembly_match (satellite_bundles[i]->name, name) && strcmp (satellite_bundles [i]->culture, culture) == 0) {
-			char *bundle_name = g_strconcat (culture, "/", name, (const char *)NULL);
-			image = mono_image_open_from_data_internal (alc, (char *)satellite_bundles [i]->data, satellite_bundles [i]->size, FALSE, status, FALSE, bundle_name, NULL);
-			g_free (bundle_name);
-			break;
-		}
-	}
+	const uint8_t *data = NULL;
+	uint32_t size = 0;
+	if (mono_bundled_resources_get_satellite_assembly_resource_values (bundle_name, &data, &size))
+		image = mono_image_open_from_data_internal (alc, (char *)data, size, FALSE, status, FALSE, bundle_name, NULL);
 
-	g_free (name);
+	g_free (bundle_name);
 	return image;
 }
 
@@ -1525,7 +1490,7 @@ open_from_satellite_bundle (MonoAssemblyLoadContext *alc, const char *filename,
  * \param status return status code
  *
  * This routine tries to open the assembly specified by \p filename from the
- * defined bundles, if found, returns the MonoImage for it, if not found
+ * bundles hashtable in bundled-resources.c, if found, returns the MonoImage for it, if not found
  * returns NULL
  */
 MonoImage *
@@ -1537,12 +1502,10 @@ mono_assembly_open_from_bundle (MonoAssemblyLoadContext *alc, const char *filena
 	 */
 	MonoImage *image = NULL;
 	MONO_ENTER_GC_UNSAFE;
-	gboolean is_satellite = culture && culture [0] != 0;
-
-	if (is_satellite)
+	if (culture && culture [0] != 0)
 		image = open_from_satellite_bundle (alc, filename, status, culture);
 	else
-		image = open_from_bundle_internal (alc, filename, status, FALSE);
+		image = open_from_bundle_internal (alc, filename, status);
 
 	if (image) {
 		mono_image_addref (image);
@@ -1620,7 +1583,7 @@ mono_assembly_request_open (const char *filename, const MonoAssemblyOpenRequest
 
 	// If VM built with mkbundle
 	loaded_from_bundle = FALSE;
-	if (bundles != NULL || satellite_bundles != NULL) {
+	if (mono_bundled_resources_contains_assemblies ()) {
 		/* We don't know the culture of the filename we're loading here, so this call is not culture aware. */
 		image = mono_assembly_open_from_bundle (load_req.alc, fname, status, NULL);
 		loaded_from_bundle = image != NULL;
@@ -3172,11 +3135,16 @@ mono_assembly_get_name_internal (MonoAssembly *assembly)
 
 /**
  * mono_register_bundled_assemblies:
+ * Dynamically allocates MonoBundledAssemblyResources to leverage
+ * preferred bundling api mono_bundled_resources_add.
  */
 void
 mono_register_bundled_assemblies (const MonoBundledAssembly **assemblies)
 {
-	bundles = assemblies;
+	for (int i = 0; assemblies [i]; ++i) {
+		const MonoBundledAssembly *assembly = assemblies [i];
+		mono_bundled_resources_add_assembly_resource (assembly->name, assembly->name, (const uint8_t *)assembly->data, (uint32_t)assembly->size, NULL, NULL);
+	}
 }
 
 /**
@@ -3187,19 +3155,34 @@ mono_create_new_bundled_satellite_assembly (const char *name, const char *cultur
 {
 	MonoBundledSatelliteAssembly *satellite_assembly = g_new0 (MonoBundledSatelliteAssembly, 1);
 	satellite_assembly->name = strdup (name);
+	g_assert (satellite_assembly->name);
 	satellite_assembly->culture = strdup (culture);
+	g_assert (satellite_assembly->culture);
 	satellite_assembly->data = data;
 	satellite_assembly->size = size;
 	return satellite_assembly;
 }
 
+static void
+mono_free_bundled_satellite_assembly_func (void *resource, void *free_data)
+{
+	g_free (free_data);
+}
+
 /**
  * mono_register_bundled_satellite_assemblies:
+ * Dynamically allocates MonoBundledSatelliteAssemblyResources to leverage
+ * preferred bundling api mono_bundled_resources_add.
  */
 void
-mono_register_bundled_satellite_assemblies (const MonoBundledSatelliteAssembly **assemblies)
+mono_register_bundled_satellite_assemblies (const MonoBundledSatelliteAssembly **satellite_assemblies)
 {
-	satellite_bundles = assemblies;
+	for (int i = 0; satellite_assemblies [i]; ++i) {
+		const MonoBundledSatelliteAssembly *satellite_assembly = satellite_assemblies [i];
+		char *id = g_strconcat (satellite_assembly->culture, "/", satellite_assembly->name, (const char*)NULL);
+		g_assert (id);
+		mono_bundled_resources_add_satellite_assembly_resource (id, satellite_assembly->name, satellite_assembly->culture, (const uint8_t *)satellite_assembly->data, (uint32_t)satellite_assembly->size, mono_free_bundled_satellite_assembly_func, id);
+	}
 }
 
 /**

src/mono/mono/metadata/bundled-resources-internals.h

@@ -0,0 +1,103 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+//
+
+#ifndef __MONO_METADATA_BUNDLED_RESOURCES_INTERNALS_H__
+#define __MONO_METADATA_BUNDLED_RESOURCES_INTERNALS_H__
+
+#include <stdbool.h>
+#include <stdint.h>
+
+typedef enum {
+	MONO_BUNDLED_DATA,
+	MONO_BUNDLED_ASSEMBLY,
+	MONO_BUNDLED_SATELLITE_ASSEMBLY,
+	MONO_BUNDLED_RESOURCE_COUNT
+} MonoBundledResourceType;
+
+typedef void (*free_bundled_resource_func)(void *, void*);
+
+typedef struct _MonoBundledResource {
+	MonoBundledResourceType type;
+	const char *id;
+	free_bundled_resource_func free_func;
+	void *free_data;
+} MonoBundledResource;
+
+typedef struct _MonoBundledData {
+	const char *name;
+	const uint8_t *data;
+	uint32_t size;
+} MonoBundledData;
+
+typedef struct _MonoBundledDataResource {
+	MonoBundledResource resource;
+	MonoBundledData data;
+} MonoBundledDataResource;
+
+typedef struct _MonoBundledSymbolData {
+	const uint8_t *data;
+	uint32_t size;
+} MonoBundledSymbolData;
+
+typedef struct _MonoBundledAssemblyData {
+	const char *name;
+	const uint8_t *data;
+	uint32_t size;
+} MonoBundledAssemblyData;
+
+typedef struct _MonoBundledAssemblyResource {
+	MonoBundledResource resource;
+	MonoBundledAssemblyData assembly;
+	MonoBundledSymbolData symbol_data;
+} MonoBundledAssemblyResource;
+
+typedef struct _MonoBundledSatelliteAssemblyData {
+	const char *name;
+	const char *culture;
+	const uint8_t *data;
+	uint32_t size;
+} MonoBundledSatelliteAssemblyData;
+
+typedef struct _MonoBundledSatelliteAssemblyResource {
+	MonoBundledResource resource;
+	MonoBundledSatelliteAssemblyData satellite_assembly;
+} MonoBundledSatelliteAssemblyResource;
+
+void
+mono_bundled_resources_free (void);
+
+void
+mono_bundled_resources_add (MonoBundledResource **resources_to_bundle, uint32_t len);
+
+bool
+mono_bundled_resources_get_assembly_resource_values (const char *id, const uint8_t **data_out, uint32_t *size_out);
+
+bool
+mono_bundled_resources_get_assembly_resource_symbol_values (const char *id, const uint8_t **symbol_data_out, uint32_t *symbol_size_out);
+
+bool
+mono_bundled_resources_get_satellite_assembly_resource_values (const char *id, const uint8_t **data_out, uint32_t *size_out);
+
+bool
+mono_bundled_resources_get_data_resource_values (const char *id, const uint8_t **data_out, uint32_t *size_out);
+
+void
+mono_bundled_resources_add_assembly_resource (const char *id, const char *name, const uint8_t *data, uint32_t size, free_bundled_resource_func free_func, void *free_data);
+
+void
+mono_bundled_resources_add_assembly_symbol_resource (const char *id, const uint8_t *data, uint32_t size, free_bundled_resource_func free_func, void *free_data);
+
+void
+mono_bundled_resources_add_satellite_assembly_resource (const char *id, const char *name, const char *culture, const uint8_t *data, uint32_t size, free_bundled_resource_func free_func, void *free_data);
+
+void
+mono_bundled_resources_add_data_resource (const char *id, const char *name, const uint8_t *data, uint32_t size, free_bundled_resource_func free_func, void *free_data);
+
+bool
+mono_bundled_resources_contains_assemblies (void);
+
+bool
+mono_bundled_resources_contains_satellite_assemblies (void);
+
+#endif /* __MONO_METADATA_BUNDLED_RESOURCES_INTERNALS_H__ */