Add support for virtual_specs checks before installation (#809)

* Add support for virtual_specs checks before installation * ignore shellcheck here * run inside PKG too (for cases other than __osx) * pre-commit * add news * update docs * reveal exceptions just to make sure * Make sure we fail for the right reason * return the process * ignore errors, process tuple * fix test on windows * Pass as unescaped variable * adjust error message for .pkg * Adjust docs * validate virtual_specs * use <= for osx * add comment about solver=classic * typo * Parse , (AND) in __osx Co-authored-by: Marco Esters <mesters@anaconda.com> * Fix PKG os-version control * pre-commit * Add Bash-only checks for SH installers * fix tests * make it posix * pre-commit * adjust expected output in test * pre-commit * fix dict access? * always provide a value even if falsey * Fix TypeError * better test? * add workaround for musl * star this too * use find --------- Co-authored-by: Marco Esters <mesters@anaconda.com>
conda · Aug 7, 2024 · 0d275c1 · 0d275c1
1 parent 53c93c5
commit 0d275c1
Show file tree

Hide file tree

Showing 15 changed files with 299 additions and 32 deletions.
diff --git a/CONSTRUCT.md b/CONSTRUCT.md
@@ -113,6 +113,18 @@ for example, if `python=3.6` is included, then conda will always seek versions
 of packages compatible with Python 3.6. If this is option is not provided, it
 will be set equal to the value of `specs`.
 
+### `virtual_specs`
+
+_required:_ no<br/>
+_type:_ list<br/>
+
+A list of virtual packages that must be satisfied at install time. Virtual
+packages must start with `__`. For example, `__osx>=11` or `__glibc>=2.24`.
+These specs are dry-run solved offline by the bundled `--conda-exe` binary.
+In SH installers, `__glibc>=x.y` and `__osx>=x.y` specs can be checked with
+Bash only. In PKG installers, `__osx` specs can be checked natively without
+the solver being involved as long as only `>=`, `<` or `,` are used.
+
 ### `exclude`
 
 _required:_ no<br/>
@@ -810,7 +822,7 @@ _required:_ no<br/>
 _type:_ list<br/>
 
 Temporary files that could be referenced in the installation process (i.e. customized
- `welcome_file` and `conclusion_file` (see above)) . Should be a list of
+`welcome_file` and `conclusion_file` (see above)) . Should be a list of
 file paths, relative to the directory where `construct.yaml` is. In Windows, these
 files will be copied into a temporary folder, the NSIS `$PLUGINSDIR`, during
 install process (Windows only).

diff --git a/constructor/construct.py b/constructor/construct.py
@@ -81,6 +81,15 @@
 for example, if `python=3.6` is included, then conda will always seek versions
 of packages compatible with Python 3.6. If this is option is not provided, it
 will be set equal to the value of `specs`.
+'''),
+
+    ('virtual_specs',          False, list, '''
+A list of virtual packages that must be satisfied at install time. Virtual
+packages must start with `__`. For example, `__osx>=11` or `__glibc>=2.24`.
+These specs are dry-run solved offline by the bundled `--conda-exe` binary.
+In SH installers, `__glibc>=x.y` and `__osx>=x.y` specs can be checked with
+Bash only. In PKG installers, `__osx` specs can be checked natively without
+the solver being involved as long as only `>=`, `<` or `,` are used.
 '''),
 
     ('exclude',                False, list, '''
@@ -594,7 +603,7 @@
 
     ('temp_extra_files', False, list, '''
 Temporary files that could be referenced in the installation process (i.e. customized
- `welcome_file` and `conclusion_file` (see above)) . Should be a list of
+`welcome_file` and `conclusion_file` (see above)) . Should be a list of
 file paths, relative to the directory where `construct.yaml` is. In Windows, these
 files will be copied into a temporary folder, the NSIS `$PLUGINSDIR`, during
 install process (Windows only).

diff --git a/constructor/header.sh b/constructor/header.sh
@@ -21,6 +21,40 @@ if ! echo "$0" | grep '\.sh$' > /dev/null; then
     return 1
 fi
 
+#if osx and min_osx_version
+min_osx_version="__MIN_OSX_VERSION__"
+system_osx_version=$(SYSTEM_VERSION_COMPAT=0 sw_vers -productVersion)
+# shellcheck disable=SC2183 disable=SC2046
+int_min_osx_version="$(printf "%02d%02d%02d" $(echo "$min_osx_version" | sed 's/\./ /g'))"
+# shellcheck disable=SC2183 disable=SC2046
+int_system_osx_version="$(printf "%02d%02d%02d" $(echo "$system_osx_version" | sed 's/\./ /g'))"
+if [  "$int_system_osx_version" -lt "$int_min_osx_version" ]; then
+    echo "Installer requires macOS >=${min_osx_version}, but system has ${system_osx_version}."
+    exit 1
+fi
+#endif
+#if linux and min_glibc_version
+min_glibc_version="__MIN_GLIBC_VERSION__"
+case "$(ldd --version 2>&1)" in
+    *musl*)
+        # musl ldd will report musl version; call ld.so directly
+        system_glibc_version=$($(find /lib/ /lib64/ -name 'ld-linux-*.so*' 2>/dev/null | head -1) --version | awk 'NR==1{ sub(/\.$/, ""); print $NF}')
+    ;;
+    *)
+        # ldd reports glibc in the last field of the first line
+        system_glibc_version=$(ldd --version | awk 'NR==1{print $NF}')
+    ;;
+esac
+# shellcheck disable=SC2183 disable=SC2046
+int_min_glibc_version="$(printf "%02d%02d%02d" $(echo "$min_glibc_version" | sed 's/\./ /g'))"
+# shellcheck disable=SC2183 disable=SC2046
+int_system_glibc_version="$(printf "%02d%02d%02d" $(echo "$system_glibc_version" | sed 's/\./ /g'))"
+if [  "$int_system_glibc_version" -lt "$int_min_glibc_version" ]; then
+    echo "Installer requires GLIBC >=${min_glibc_version}, but system has ${system_glibc_version}."
+    exit 1
+fi
+#endif
+
 # Export variables to make installer metadata available to pre/post install scripts
 # NOTE: If more vars are added, make sure to update the examples/scripts tests too
 
@@ -423,6 +457,17 @@ export TMP_BACKUP="${TMP:-}"
 export TMP="$PREFIX/install_tmp"
 mkdir -p "$TMP"
 
+# Check whether the virtual specs can be satisfied
+# We need to specify CONDA_SOLVER=classic for conda-standalone
+# to work around this bug in conda-libmamba-solver:
+# https://github.com/conda/conda-libmamba-solver/issues/480
+# shellcheck disable=SC2050
+if [ "__VIRTUAL_SPECS__" != "" ]; then
+    CONDA_QUIET="$BATCH" \
+    CONDA_SOLVER="classic" \
+    "$CONDA_EXEC" create --dry-run --prefix "$PREFIX" --offline __VIRTUAL_SPECS__
+fi
+
 # Create $PREFIX/.nonadmin if the installation didn't require superuser permissions
 if [ "$(id -u)" -ne 0 ]; then
     touch "$PREFIX/.nonadmin"

diff --git a/constructor/main.py b/constructor/main.py
@@ -115,11 +115,18 @@ def main_build(dir_path, output_dir='.', platform=cc_platform,
     elif info.get("signing_certificate"):
         info["windows_signing_tool"] = "signtool"
 
-    for key in 'specs', 'packages':
+    for key in 'specs', 'packages', 'virtual_specs':
         if key not in info:
             continue
         if isinstance(info[key], str):
             info[key] = list(yield_lines(join(dir_path, info[key])))
+        if key == "virtual_specs":
+            for value in info[key]:
+                if not value.startswith("__"):
+                    raise ValueError(
+                        "'virtual_specs' can only include virtual package names like '__name', "
+                        f"but you supplied: {value}."
+                    )
 
     # normalize paths to be copied; if they are relative, they must be to
     # construct.yaml's parent (dir_path)
@@ -137,7 +144,7 @@ def main_build(dir_path, output_dir='.', platform=cc_platform,
                 new_extras.append({orig: dest})
         info[extra_type] = new_extras
 
-    for key in 'channels', 'specs', 'exclude', 'packages', 'menu_packages':
+    for key in 'channels', 'specs', 'exclude', 'packages', 'menu_packages', 'virtual_specs':
         if key in info:
             # ensure strings in those lists are stripped
             info[key] = [line.strip() for line in info[key]]

diff --git a/constructor/nsis/main.nsi.tmpl b/constructor/nsis/main.nsi.tmpl
@@ -1132,6 +1132,21 @@ Section "Install"
     System::Call 'kernel32::SetEnvironmentVariable(t,t)i("INSTALLER_PLAT", "${PLATFORM}").r0'
     System::Call 'kernel32::SetEnvironmentVariable(t,t)i("INSTALLER_TYPE", "EXE").r0'
 
+    ${If} '@VIRTUAL_SPECS@' != ''
+    # We need to specify CONDA_SOLVER=classic for conda-standalone
+    # to work around this bug in conda-libmamba-solver:
+    # https://github.com/conda/conda-libmamba-solver/issues/480
+    System::Call 'kernel32::SetEnvironmentVariable(t,t)i("CONDA_SOLVER", "classic").r0'
+    SetDetailsPrint TextOnly
+    DetailPrint "Checking virtual specs..."
+    push '"$INSTDIR\_conda.exe" create --dry-run --prefix "$INSTDIR" --offline @VIRTUAL_SPECS@'
+    push 'Failed to check virtual specs: @VIRTUAL_SPECS@'
+    push 'WithLog'
+    call AbortRetryNSExecWait
+    SetDetailsPrint both
+    System::Call 'kernel32::SetEnvironmentVariable(t,t)i("CONDA_SOLVER", "").r0'
+    ${EndIf}
+
     @PKG_COMMANDS@
 
     SetDetailsPrint TextOnly

diff --git a/constructor/osx/prepare_installation.sh b/constructor/osx/prepare_installation.sh
@@ -31,6 +31,17 @@ chmod +x "$CONDA_EXEC"
 mkdir -p "$PREFIX/conda-meta"
 touch "$PREFIX/conda-meta/history"
 
+# Check whether the virtual specs can be satisfied
+# We need to specify CONDA_SOLVER=classic for conda-standalone
+# to work around this bug in conda-libmamba-solver:
+# https://github.com/conda/conda-libmamba-solver/issues/480
+# shellcheck disable=SC2050
+if [ "__VIRTUAL_SPECS__" != "" ]; then
+    CONDA_QUIET="$BATCH" \
+    CONDA_SOLVER="classic" \
+    "$CONDA_EXEC" create --dry-run --prefix "$PREFIX" --offline __VIRTUAL_SPECS__
+fi
+
 # Create $PREFIX/.nonadmin if the installation didn't require superuser permissions
 if [ "$(id -u)" -ne 0 ]; then
     touch "$PREFIX/.nonadmin"

diff --git a/constructor/osxpkg.py b/constructor/osxpkg.py
@@ -1,5 +1,6 @@
 import logging
 import os
+import shlex
 import shutil
 import sys
 import xml.etree.ElementTree as ET
@@ -18,6 +19,7 @@
     explained_check_call,
     fill_template,
     get_final_channels,
+    parse_virtual_specs,
     preprocess,
     rm_rf,
     shortcuts_flags,
@@ -188,6 +190,18 @@ def modify_xml(xml_path, info):
         )
         root.append(readme)
 
+    # -- __osx virtual package checks -- #
+    # Reference: https://developer.apple.com/library/archive/documentation/DeveloperTools/Reference/DistributionDefinitionRef/Chapters/Distribution_XML_Ref.html  # noqa
+    osx_versions = parse_virtual_specs(info).get("__osx")
+    if osx_versions:
+        if "min" not in osx_versions:
+            raise ValueError("Specifying __osx requires a lower bound with `>=`")
+        allowed_os_versions = ET.Element("allowed-os-versions")
+        allowed_os_versions.append(ET.Element("os-version", osx_versions))
+        volume_check = ET.Element("volume-check")
+        volume_check.append(allowed_os_versions)
+        root.append(volume_check)
+
     # See below for an explanation of the consequences of this
     # customLocation value.
     for options in root.findall('options'):
@@ -327,6 +341,7 @@ def move_script(src, dst, info, ensure_shebang=False, user_script_type=None):
         'SHORTCUTS': shortcuts_flags(info),
         'ENABLE_SHORTCUTS': str(info['_enable_shortcuts']).lower(),
         'REGISTER_ENVS': str(info.get("register_envs", True)).lower(),
+        'VIRTUAL_SPECS': shlex.join(info.get("virtual_specs", ())),
     }
     data = preprocess(data, ppd)
     custom_variables = info.get('script_env_variables', {})

diff --git a/constructor/shar.py b/constructor/shar.py
@@ -6,6 +6,7 @@
 
 import logging
 import os
+import shlex
 import shutil
 import stat
 import tarfile
@@ -23,6 +24,7 @@
     fill_template,
     get_final_channels,
     hash_files,
+    parse_virtual_specs,
     preprocess,
     read_ascii_only,
     shortcuts_flags,
@@ -92,10 +94,17 @@ def get_header(conda_exec, tarball, info):
         'SHORTCUTS': shortcuts_flags(info),
         'REGISTER_ENVS': str(info.get("register_envs", True)).lower(),
         'TOTAL_INSTALLATION_SIZE_KB': str(approx_size_kb(info, "total")),
+        'VIRTUAL_SPECS': shlex.join(info.get("virtual_specs", ()))
     }
     if has_license:
         replace['LICENSE'] = read_ascii_only(info['license_file'])
 
+    virtual_specs = parse_virtual_specs(info)
+    min_osx_version = virtual_specs.get("__osx", {}).get("min") or ""
+    replace['MIN_OSX_VERSION'] = ppd['min_osx_version'] = min_osx_version
+    min_glibc_version = virtual_specs.get("__glibc", {}).get("min") or ""
+    replace['MIN_GLIBC_VERSION'] = ppd['min_glibc_version'] = min_glibc_version
+
     data = read_header_template()
     data = preprocess(data, ppd)
     custom_variables = info.get('script_env_variables', {})

diff --git a/constructor/utils.py b/constructor/utils.py
@@ -282,3 +282,30 @@ def check_required_env_vars(env_vars):
         raise RuntimeError(
             f"Missing required environment variables {', '.join(missing_vars)}."
         )
+
+
+def parse_virtual_specs(info) -> dict:
+    from .conda_interface import MatchSpec  # prevent circular import
+
+    specs = {"__osx": {}, "__glibc": {}}
+    for spec in info.get("virtual_specs", ()):
+        spec = MatchSpec(spec)
+        if spec.name not in ("__osx", "__glibc"):
+            continue
+        if not spec.version:
+            continue
+        if "|" in spec.version.spec_str:
+            raise ValueError("Can't process `|`-joined versions. Only `,` is allowed.")
+        versions = spec.version.tup if "," in spec.version.spec_str else (spec.version,)
+        for version in versions:
+            operator = version.operator_func.__name__
+            if operator == "ge":
+                specs[spec.name]["min"] = str(version.matcher_vo)
+            elif operator == "lt" and spec.name == "__osx":
+                specs[spec.name]["before"] = str(version.matcher_vo)
+            else:
+                raise ValueError(
+                    f"Invalid version operator for {spec}. "
+                    "__osx only supports `<` or `>=`; __glibc only supports `>=`."
+                )
+    return specs
diff --git a/constructor/winexe.py b/constructor/winexe.py
@@ -374,6 +374,8 @@ def make_nsi(
             else ''
         ),
         ('@TEMP_EXTRA_FILES@', '\n    '.join(insert_tempfiles_commands(temp_extra_files))),
+        ('@VIRTUAL_SPECS@', " ".join([f'"{spec}"' for spec in info.get("virtual_specs", ())])),
+
     ]:
         data = data.replace(key, value)
 

diff --git a/docs/source/construct-yaml.md b/docs/source/construct-yaml.md
@@ -113,6 +113,18 @@ for example, if `python=3.6` is included, then conda will always seek versions
 of packages compatible with Python 3.6. If this is option is not provided, it
 will be set equal to the value of `specs`.
 
+### `virtual_specs`
+
+_required:_ no<br/>
+_type:_ list<br/>
+
+A list of virtual packages that must be satisfied at install time. Virtual
+packages must start with `__`. For example, `__osx>=11` or `__glibc>=2.24`.
+These specs are dry-run solved offline by the bundled `--conda-exe` binary.
+In SH installers, `__glibc>=x.y` and `__osx>=x.y` specs can be checked with
+Bash only. In PKG installers, `__osx` specs can be checked natively without
+the solver being involved as long as only `>=`, `<` or `,` are used.
+
 ### `exclude`
 
 _required:_ no<br/>
@@ -810,7 +822,7 @@ _required:_ no<br/>
 _type:_ list<br/>
 
 Temporary files that could be referenced in the installation process (i.e. customized
- `welcome_file` and `conclusion_file` (see above)) . Should be a list of
+`welcome_file` and `conclusion_file` (see above)) . Should be a list of
 file paths, relative to the directory where `construct.yaml` is. In Windows, these
 files will be copied into a temporary folder, the NSIS `$PLUGINSDIR`, during
 install process (Windows only).

diff --git a/examples/virtual_specs/construct.yaml b/examples/virtual_specs/construct.yaml
@@ -0,0 +1,22 @@
+name: virtual_specs
+
+version: 0.0.1
+
+keep_pkgs: True
+
+channels:
+  - conda-forge
+
+specs:
+  - ca-certificates
+
+virtual_specs:
+  - __osx>=30,<31  # [osx]
+  - __glibc>=20    # [linux]
+  - __win<0        # [win]
+
+initialize_by_default: false
+register_python: false
+check_path_spaces: false
+check_path_length: false
+installer_type: all
diff --git a/news/809-virtual-specs b/news/809-virtual-specs
@@ -0,0 +1,19 @@
+### Enhancements
+
+* A new setting `virtual_specs` allows the installer to run some solver checks before the installation proceeds. Useful for checking whether certain virtual package versions can be satisfied. (#809)
+
+### Bug fixes
+
+* <news item>
+
+### Deprecations
+
+* <news item>
+
+### Docs
+
+* <news item>
+
+### Other
+
+* <news item>