Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

admin/ohpc-filesystem: include custom requires/provides plugin files for RPM dependency analysis #645

Merged
merged 4 commits into from
Jan 12, 2018
Merged

admin/ohpc-filesystem: include custom requires/provides plugin files for RPM dependency analysis #645

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9ae517c
add rpm dependency plugin files
koomie Jan 5, 2018
36eedc1
update symlink_exclude_path
koomie Jan 5, 2018
71d6f7f
script cleanup: remove use of TMPFILE and include searchpath as argument
koomie Jan 10, 2018
c1ac656
admin/ohpc-filesystem: fix typo in -z test
koomie Jan 11, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions components/OHPC_macros
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,8 @@ Requires: llvm-compilers%{PROJ_DELIM}

%endif

%global __libsymlink_exclude_path %{OHPC_HOME}/.*$
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A comment what this for would be nice.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in #648


# MPI dependencies
%if 0%{?ohpc_mpi_dependent} == 1
%if "%{mpi_family}" == "impi"
Expand Down
31 changes: 31 additions & 0 deletions components/admin/ohpc-filesystem/SOURCES/ohpc-find-provides
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#!/bin/bash
#
# ohpc-find-provides
#-----------------------------------------------------------------------
# Licensed under the Apache License, Version 2.0 (the "License"); you
# may not use this file except in compliance with the License. You may
# obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied. See the License for the specific language governing
# permissions and limitations under the License.
#-----------------------------------------------------------------------

IFS=$'\n'

# Get the list of files.
filelist=`sed "s/[]['\"*?{}]/\\\\\&/g"`

# Use standard elfdeps tool and append additional (ohpc) color delimiter

if [ -x /usr/lib/rpm/elfdeps -a -n "$filelist" ];then

provideList=$(echo $filelist | /usr/lib/rpm/elfdeps -P)
for provide in $provideList; do
echo "$provide(ohpc)"
done
fi
74 changes: 74 additions & 0 deletions components/admin/ohpc-filesystem/SOURCES/ohpc-find-requires
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
#!/bin/bash
#
# ohpc-find-requires
#-----------------------------------------------------------------------
# Licensed under the Apache License, Version 2.0 (the "License"); you
# may not use this file except in compliance with the License. You may
# obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied. See the License for the specific language governing
# permissions and limitations under the License.
#-----------------------------------------------------------------------

IFS=$'\n'

# First argument is buildroot
buildroot="$1"
if [ ! -d "$buildroot" ]; then
>&2 echo "Invalid buildroot"
exit 1
fi

# Second argument is top-level search path
searchPath="$2"
if [ -z "$searchPath" ];then
>&2echo "Required search path argument not provided"
exit 1
fi

if [ ! -x /usr/lib/rpm/elfdeps ]; then
>&2 echo "Required /usr/lib/rpm/elfdeps binary not available locally"
exit 1
fi

# Get the list of files.
filelist=`sed "s/[]['\"*?{}]/\\\\\&/g"`

if [ -z "$filelist" ]; then exit 0; fi


# Step 1: use standard elfdeps analysis and cache results
requireList=$(echo ${filelist} | /usr/lib/rpm/elfdeps -R)

# Step 2: append additional color delimiter for ohpc provided packages (that
# install into $searchPath)

for require in ${requireList}; do

# Check if this is owned by ohpc pre-requisite
package=$(rpm -q --queryformat '%{NAME}\n' --whatprovides "$require(ohpc)")
if [ $? -eq 0 ];then
echo "$require(ohpc)"
else

# check if this requirement is housed in ${buildroot}/opt/ohpc.
# If so, we append an (ohpc) color designation, otherwise we
# leave the requirement as is.

libname=${require%%(*} # strip off () to get libname
match=$(find ${buildroot}/${searchPath} -name ${libname})

if [ -n "${match}" ];then
echo "$require(ohpc)"
else
echo "$require"
fi

fi

done
7 changes: 7 additions & 0 deletions components/admin/ohpc-filesystem/SOURCES/ohpc.attr
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
%__ohpc_provides /usr/lib/rpm/ohpc-find-provides
%__ohpc_requires /usr/lib/rpm/ohpc-find-requires %{buildroot} %{OHPC_HOME}
Copy link
Member

@adrianreber adrianreber Jan 12, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The usage of %{OHPC_HOME} in this file makes me a bit nervous. At what time is the file evaluated. After including OHPC_macros? But with happens with RPMs that do not define %{OHPC_HOME}. It then probably resolves to literally '%{OHPC_HOME}'. Is that a problem. Main question: Does this break building non-OHPC RPMS?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point...let me look into that further and test.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like standard macro syntax works fine for the path designations, so can guard those with the following:

%__ohpc_path            ^%{!?OHPC_HOME:/opt/ohpc}
%__elf_exclude_path     ^%{!?OHPC_HOME:/opt/ohpc}

The she-bang macro test does not look to work in the requires stanza, so working on a slightly amended solution for that. Will double check with ohpc and non-ohpc package builds and follow up with newer patches...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update in PR #648. See what you think. I built a non-ohpc package with the updated config and didn't see any issues (and get the same Provides/Requires results you get without the extra ohpc dependency files enabled).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, I take back the fact that the macro checks in the .attr file work as expected after kicking off a number of builds - I should have tested more with non-ohpc packages that include elf binaries. I have updated again via #650 and this approach simply calls out the /opt/ohpc path directly instead of relying on macro definitions.

I was previously concerned with trying to allow for someone to relocate the top-level path for all ohpc packages (say customized environment were someone is rebuilding everything from .src rpms). However, in that case, a user will likely need to modify OHPC_macros to set the desired top-level path, so I just leveraged those settings in the creation of the ohpc.attr file. If they want to customize, they can rebuild ohpc-buildroot to update the paths to suit.

%__ohpc_path ^%{OHPC_HOME}
%__elf_exclude_path ^%{OHPC_HOME}

%__ohpc_magic ^ELF (32|64)-bit.*$
%__ohpc_flags magic_and_path
22 changes: 20 additions & 2 deletions components/admin/ohpc-filesystem/SPECS/ohpc-filesystem.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,15 @@ Group: ohpc/admin
License: ASL 2.0
Source0: OHPC_setup_compiler
Source1: OHPC_setup_mpi
Source2: ohpc.attr
Source3: ohpc-find-requires
Source4: ohpc-find-provides

BuildArch: noarch

%description
This administrative package is used to define top level OpenHPC installation
directories and is utilized by most packages that do not install into system
directories. It is utilized by most packages that do not install into system
default paths.

%package -n ohpc-buildroot
Expand All @@ -22,15 +25,25 @@ Requires: lmod-ohpc
Requires: ohpc-filesystem

%description -n ohpc-buildroot
Common compiler and MPI family convenience scripts used during OpenHPC builds.

This administrative package is used to provide RPM dependency analysis tools
and common compiler and MPI family convenience scripts used during OpenHPC
builds.

%install
# The ohpc-filesystems owns all the common directories
mkdir -p $RPM_BUILD_ROOT/opt/ohpc/pub/{apps,doc,compiler,libs,moduledeps,modulefiles,mpi}
mkdir -p $RPM_BUILD_ROOT/opt/ohpc/admin/ohpc
mkdir -p $RPM_BUILD_ROOT/usr/lib/rpm/fileattrs

install -p -m 644 %{SOURCE0} $RPM_BUILD_ROOT/opt/ohpc/admin/ohpc
install -p -m 644 %{SOURCE1} $RPM_BUILD_ROOT/opt/ohpc/admin/ohpc

# rpm dependency plugins
install -p -m 755 %{SOURCE2} $RPM_BUILD_ROOT/usr/lib/rpm/fileattrs
install -p -m 755 %{SOURCE3} $RPM_BUILD_ROOT/usr/lib/rpm
install -p -m 755 %{SOURCE4} $RPM_BUILD_ROOT/usr/lib/rpm

%files
%dir /opt/ohpc/
%dir /opt/ohpc/admin/
Expand All @@ -45,8 +58,13 @@ install -p -m 644 %{SOURCE1} $RPM_BUILD_ROOT/opt/ohpc/admin/ohpc

%files -n ohpc-buildroot
%dir /opt/ohpc/admin/ohpc/
%dir /usr/lib/rpm/
%dir /usr/lib/rpm/fileattrs/
/opt/ohpc/admin/ohpc/OHPC_setup_compiler
/opt/ohpc/admin/ohpc/OHPC_setup_mpi
/usr/lib/rpm/ohpc-find-provides
/usr/lib/rpm/ohpc-find-requires
/usr/lib/rpm/fileattrs/ohpc.attr

%changelog
* Mon May 8 2017 Karl W Schulz <karl.w.schulz@intel.com> - 1.3
Expand Down