Update llpc from commit 4487a957

[Continuations] Remove `CONTINUATIONS_USE_DUMMY_RET_ADDR` define
[Continuations] Remove references to `_cont_SetupRayGen`
[RT] Fix IR verifier error when reading GPURT
SPIRVReader: Add LayoutMode::None, remove isParentPointer, m_typeMap, make recordTypeWithPad return void
[LGC] Remove descriptor type argument from getDescPtr
Fix broken spvasm tests
Add support for SPV_EXT_fragment_shader_interlock
compilerutils: Put TypeLowering into a namespace
Remove RT lit test dependency on SPIR-V binary
Fix and cleanup barrier translation
Add an check of dual source blend in full pipeline
[LGC] Fix deprecation warnings
SPIRVReader: Unify OpType{Runtime}Array conversion
Fix OpControlBarrier mapping
Fix compiling warning of switch cases
[LGC] Handle load and build buffer descriptor in PatchBufferOp
SPIRVReader: refactor image support
[Continuations] Pass padding and payload as argument
[Continuations] Fix irreducible cycle during continuation transform
Unify the namespace compilerutils with CompilerUtils
Rework mesh shader export slot assignment
[Continuations] Remove RegisterBuffer pass.
[RT] Rayquery refactor to lgc
[LGC] Fix cast error when forceRawView buffer load is used
[Continuations] Remove `DISABLE_CPS_PAYLOADREG` define.
Stop replacing atomicrmw fadd instructions with intrinsics
[LGC] Missing index for LoadTfeOp structured load
[Continations] Remove comment on register buffer
Remove assertion and update tests to align with recent LLVM changes
amdllpc: Fix amdllpc test failure PipelineRays_TestStaticCompile.pipe
[Continuations] Move `handleGetFuncAddr`
amdllpc: Flush SPIR-V output to aid in debugging
Make readlane call mangling-agnostic
[LGC] Fix deprecated StringRef::equals warning
[LGC] Remove use of RepeatedPass
[LGC] Include FormatVariadic.h
[Continuations] Reformat code
SPIRVReader: workaround for some invalid SPIR-V
Fix traversal module pop back
Extract LowerRayQuery from LGC and move to RT
[Continuations] Dialectify `continuation.continue` and `continuation.waitContinue`.
Update intrinsic names for upstream change
Fix use-after-free in LowerRayQuery refactor
Add debug printer for ModuleBunch
Stop replacing atomicrmw fmin/fmax instructions with intrinsics
Update tests for non-i8 gep of mul change
[Continuations] Tidy up assertion in LowerRaytracingPipeline
Prefer mdconst::extract over mdconst::dyn_extract
Update includes and fwd decls
Fix some Clang warnings
Enable intrinsic instead of using named call
Use range-based set() for SmallBitVector
[LGC] Add dereferenceable info for descriptor load
[Continuations] Ignore single unreachable in block
[CompilerUtils] Handle phi in replaceAllPointerUses
Use CreateIntrinsic for building intrinsic calls
llvmraytracing: Add getShaderSubtypeForRtShaderStage
[Continuations] Refactor metadata handling
[RT] Support GpurtGetKnown(Un)SetRayFlagsOp, and add a lit test
Fix build and test issue when bumping LLPC version to 73
Move the Dx9MulZero match from llpcSpirvLowerMath into PatchMulDx9Zero which is added after Scalarize Pass
llvmraytracing: Add getFuncReturnPtrElementType
[RT] Avoid unnecessary compilations for traversal module
Fix RTIP-overriding for GPURT rayQuery functions
Fix #include file ordering
[LGC] Set new .uses_cps PAL metadata where applicable
[CompilerUtils] Simplify pointee types metadata
[Continuations] Use `lgc.cps.jump` in `LegacyCleanupContinuations`
[CompilerUtils] Move promotePointerArguments etc to compilerutils
Use function linkage setting from spirv
[CompilerUtils] Fix cross-module-inliner tests, and move to subdir
[LGC] Add option to force user data spilling
Improve continuations heuristic
[Continuations] Fix clang-format
Remove SpirvLowerRayQuery from lower passes
[Continuations] Cleanup continuations analyses, remove registercount + statesize metadata
[Continuations] Prevent zero-size `lgc.cps.free` calls in `CleanupContinuations`
[LGC] Replace threadMaskedSelect function to use inverseballot
[LGC] Only insert non null components in reassembleOutputExportCalls
[Continuations] Use new `replaceVarArgs` helper from dialects
[Continuations] Add constant jump 'inliner' pass
[Continuations] Basic Continuations 'linter' pass
[Continuations] Introduce lgc.cps.jump in `LowerRaytracingPipeline`
Remove explicit LLVM benchmark targets from CMake
[Continuations] Handle _AmdValue* intrinsics
[LGC] Fix issues of FS without exports
Fix dEQP-VK.subgroups.arithmetic.ray_tracing* crash
Update llvm-dialects submodule
Pack mesh shader outputs in LDS space

CMakeLists.txt

@@ -189,8 +189,12 @@ if(CMAKE_CXX_COMPILER_ID STREQUAL "MSVC")
 
         set_property(TARGET all-targets PROPERTY FOLDER Misc)
         set_property(TARGET AMDGPU PROPERTY FOLDER Misc)
-        set_property(TARGET benchmark PROPERTY FOLDER Misc)
-        set_property(TARGET benchmark_main PROPERTY FOLDER Misc)
+        if (TARGET benchmark)
+          set_property(TARGET benchmark PROPERTY FOLDER Misc)
+        endif()
+        if (TARGET benchmark_main)
+          set_property(TARGET benchmark_main PROPERTY FOLDER Misc)
+        endif()
         set_property(TARGET distribution PROPERTY FOLDER Misc)
         set_property(TARGET Engine PROPERTY FOLDER Misc)
         set_property(TARGET install-distribution PROPERTY FOLDER Misc)
@@ -210,8 +214,14 @@ if(CMAKE_CXX_COMPILER_ID STREQUAL "MSVC")
             set_property(TARGET CONFIGURE_LLVM_NATIVE PROPERTY FOLDER Misc)
             set_property(TARGET CREATE_LLVM_NATIVE PROPERTY FOLDER Misc)
         endif()
+#if _WIN32
+        if (MSVC)
+            # We can't use LLVM_OPTIMIZED_TABLEGEN on Windows, and the 32-bit llvm-tblgen can easily
+            # to run out of memory. Tell the linker to allow addresses larger than 2GB.
+            set_property(TARGET llvm-tblgen PROPERTY LINK_FLAGS "/LARGEADDRESSAWARE")
+        endif()
+#endif
         if (LLPC_BUILD_TESTS)
-            set_property(TARGET check-all PROPERTY FOLDER Tests)
             if(NOT LLPC_IS_STANDALONE)
                 set_property(TARGET check-amber PROPERTY FOLDER "LLPC Tests")
             endif()

compilerutils/CMakeLists.txt

@@ -12,8 +12,10 @@ function(set_compiler_options PROJECT_NAME)
 endfunction()
 
 add_llvm_library(LLVMCompilerUtils
+  lib/ArgPromotion.cpp
   lib/CompilerUtils.cpp
   lib/TypeLowering.cpp
+  lib/TypesMetadata.cpp
 
   DEPENDS
   intrinsics_gen

llvmraytracing/include/llvmraytracing/TypesMetadata.h → compilerutils/include/compilerutils/ArgPromotion.h

@@ -10,8 +10,8 @@
  *  sell copies of the Software, and to permit persons to whom the Software is
  *  furnished to do so, subject to the following conditions:
  *
- *  The above copyright notice and this permission notice shall be included in
- *all copies or substantial portions of the Software.
+ *  The above copyright notice and this permission notice shall be included in all
+ *  copies or substantial portions of the Software.
  *
  *  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
  *  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
@@ -23,27 +23,25 @@
  *
  **********************************************************************************************************************/
 
-//===- TypesMetadata.h - Pointee type metadata for processing DXIL ---------==//
+// Promotion of pointer args to by-value.
 
 #pragma once
 
-#include "llvm/Bitcode/BitcodeReader.h"
-
 namespace llvm {
+class Function;
+class SmallBitVector;
+} // namespace llvm
 
-/// Return element type of a function argument resolving opaque pointers
-/// via !types metadata where appropriate.
-/// Returns nullptr for non-pointers.
-Type *getFuncArgPtrElementType(const Argument *Arg);
+namespace CompilerUtils {
 
-/// Return element type of a function argument resolving opaque pointers
-/// via !types metadata where appropriate.
-/// Returns nullptr for non-pointers.
-Type *getFuncArgPtrElementType(const Function *F, int ArgNo);
+// Replace struct return type with its first element type.
+llvm::Function *unpackStructReturnType(llvm::Function *Fn);
 
-/// LLVM parser callback which adds !types metadata during DXIL parsing
-void DXILValueTypeMetadataCallback(Value *V, unsigned TypeID,
-                                   GetTypeByIDTy GetTypeByID,
-                                   GetContainedTypeIDTy GetContainedTypeID);
+// Turn StructRet argument into return type, modifying pointee type metadata as appropriate.
+llvm::Function *lowerStructRetArgument(llvm::Function *Fn);
 
-} // namespace llvm
+// Promote pointer (by-ref) arguments to by-value, according to PromotionMask
+// and using pointee type metadata.
+llvm::Function *promotePointerArguments(llvm::Function *Fn, const llvm::SmallBitVector &PromotionMask);
+
+} // namespace CompilerUtils

compilerutils/include/compilerutils/CompilerUtils.h

@@ -129,7 +129,6 @@ class CrossModuleInliner {
 // The caller has to handle the erasure afterwards.
 void replaceAllPointerUses(llvm::IRBuilder<> *builder, llvm::Value *oldPointerValue, llvm::Value *newPointerValue,
                            llvm::SmallVectorImpl<llvm::Instruction *> &toBeRemoved);
-
 } // namespace CompilerUtils
 
 namespace llvm {
@@ -141,6 +140,31 @@ namespace llvm {
 //       fixed.
 PointerType *getWithSamePointeeType(PointerType *ptrTy, unsigned addressSpace);
 
+/// Free-standing helpers.
+
+// Helper to visit all calls of a function.
+// Expected type for Callback:
+//  void(CallInst &)
+template <typename CallbackTy> void forEachCall(Function &F, CallbackTy Callback) {
+  static_assert(std::is_invocable_v<CallbackTy, CallInst &>);
+  for (auto &Use : make_early_inc_range(F.uses())) {
+    if (auto *CInst = dyn_cast<CallInst>(Use.getUser()))
+      if (CInst->isCallee(&Use))
+        Callback(*CInst);
+  }
+}
+
+// For each basic block in Func, find the terminator. If it is contained in
+// TerminatorOpcodes, then apply the callback on the terminator.
+template <typename CallbackTy, typename = std::enable_if<std::is_invocable_v<CallbackTy, llvm::Instruction &>>>
+void forEachTerminator(Function *Func, ArrayRef<unsigned> TerminatorOpcodes, CallbackTy Callback) {
+  for (auto &BB : *Func) {
+    auto *Terminator = BB.getTerminator();
+    if (llvm::find(TerminatorOpcodes, Terminator->getOpcode()) != TerminatorOpcodes.end())
+      Callback(*Terminator);
+  }
+}
+
 } // namespace llvm
 
 #endif

compilerutils/include/compilerutils/LoweringPointerTupleMap.h

@@ -37,7 +37,7 @@
 #include <type_traits>
 #include <vector>
 
-namespace compilerutils {
+namespace CompilerUtils {
 
 /// @brief A key-value map from pointer keys to tuples of pointers that is optimized for value and type lowering uses
 ///
@@ -252,4 +252,4 @@ template <typename KeyT, typename ValueT, bool TrackReverse> class LoweringPoint
   }
 };
 
-} // namespace compilerutils
+} // namespace CompilerUtils

compilerutils/include/compilerutils/TypeLowering.h

@@ -61,6 +61,8 @@
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/IR/IRBuilder.h"
 
+namespace CompilerUtils {
+
 class TypeLowering;
 
 /// Given a type, check if it should be replaced.
@@ -169,14 +171,16 @@ class TypeLowering {
   llvm::SmallVector<std::function<ConstantTypeLoweringFn>> m_constantRules;
 
   /// Cache mappings of types (including no-op mappings).
-  compilerutils::LoweringPointerTupleMap<llvm::Type *, llvm::Type *, false> m_typeConversions;
+  CompilerUtils::LoweringPointerTupleMap<llvm::Type *, llvm::Type *, false> m_typeConversions;
 
   llvm::IRBuilder<> m_builder;
 
   /// Map original values to type-converted values.
-  compilerutils::LoweringPointerTupleMap<llvm::Value *, llvm::Value *, true> m_valueMap;
+  CompilerUtils::LoweringPointerTupleMap<llvm::Value *, llvm::Value *, true> m_valueMap;
 
   std::vector<std::pair<llvm::PHINode *, llvm::SmallVector<llvm::PHINode *>>> m_phis;
   std::vector<llvm::Instruction *> m_instructionsToErase;
   llvm::SmallVector<llvm::Function *> m_functionsToErase;
 };
+
+} // namespace CompilerUtils

compilerutils/include/compilerutils/TypesMetadata.h

@@ -0,0 +1,117 @@
+/*
+ ***********************************************************************************************************************
+ *
+ *  Copyright (c) 2024 Advanced Micro Devices, Inc. All Rights Reserved.
+ *
+ *  Permission is hereby granted, free of charge, to any person obtaining a copy
+ *  of this software and associated documentation files (the "Software"), to
+ *  deal in the Software without restriction, including without limitation the
+ *  rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ *  sell copies of the Software, and to permit persons to whom the Software is
+ *  furnished to do so, subject to the following conditions:
+ *
+ *  The above copyright notice and this permission notice shall be included in all
+ *  copies or substantial portions of the Software.
+ *
+ *  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ *  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ *  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ *  AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ *  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ *  FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ *  IN THE SOFTWARE.
+ *
+ **********************************************************************************************************************/
+
+//===- TypesMetadata.h - Pointee type metadata for processing DXIL ---------==//
+
+#pragma once
+
+#include "llvm/Bitcode/BitcodeReader.h"
+
+namespace llvm {
+
+class MDTuple;
+
+// A function argument type and pointee type.
+class TypedArgTy {
+private:
+  Type *ArgTy = nullptr;
+  Type *ElemTy = nullptr;
+
+public:
+  TypedArgTy() {}
+  TypedArgTy(Type *Arg);
+  TypedArgTy(Type *Arg, Type *Elem);
+
+  static TypedArgTy get(const Argument *Arg);
+  static TypedArgTy get(const Function *F, const unsigned ArgNo);
+
+  Type *asType() const { return ArgTy; }
+  Type *getPointerElementType() const;
+
+  bool isPointerTy() const;
+  bool isVoidTy() const;
+  Metadata *getTypeMetadata(LLVMContext &Context);
+
+  bool operator==(const TypedArgTy &RHS) const { return (ArgTy == RHS.ArgTy) && (ElemTy == RHS.ElemTy); }
+};
+
+// A wrapper round FunctionType and metadata for the pointee type(s) of any pointer return type and parameters.
+class TypedFuncTy {
+public:
+  TypedFuncTy() {}
+
+  // Construct a TypedFuncTy for the given result type and arg types.
+  // This constructs the !pointeetys metadata; that can then be attached to a function
+  // using writeMetadata().
+  TypedFuncTy(TypedArgTy ResultTy, ArrayRef<TypedArgTy> ArgTys);
+
+  // Get a TypedFuncTy for the given Function, looking up the !pointeetys metadata.
+  static TypedFuncTy get(const Function *F);
+
+  // Get the IR FunctionType.
+  FunctionType *asFunctionType() const { return FuncTy; }
+
+  // Get a TypedArgTy for the return type.
+  TypedArgTy getReturnType() const;
+
+  // Get a TypedArgTy for a parameter type.
+  TypedArgTy getParamType(unsigned Idx) const;
+
+  // Push a TypedArgTy for each parameter onto the supplied vector.
+  void getParamTypes(SmallVectorImpl<TypedArgTy> &ArgTys) const;
+
+  // Write the metadata (if any) onto the specified function. Typically used when creating a new function
+  // and using our constructor that takes TypedArgTy for return type and arg types.
+  void writeMetadata(Function *F) const;
+
+  static constexpr const char *MDTypesName = "pointeetys";
+
+private:
+  // Shared code for getReturnType and getParamType. This decodes the !pointeetys metadata.
+  Type *getPointeeType(Type *Ty, unsigned Idx) const;
+
+  FunctionType *FuncTy = nullptr;
+  MDTuple *Meta = nullptr;
+};
+
+/// Return element type of a function argument resolving opaque pointers
+/// via !pointeetys metadata where appropriate.
+/// Returns nullptr for non-pointers.
+Type *getFuncArgPtrElementType(const Argument *Arg);
+
+/// Return element type of a function argument resolving opaque pointers
+/// via !pointeetys metadata where appropriate.
+/// Returns nullptr for non-pointers.
+Type *getFuncArgPtrElementType(const Function *F, int ArgNo);
+
+/// Get element type of function return type resolving opaque pointers
+/// via !pointeetys metadata where appropriate.
+Type *getFuncReturnPtrElementType(const Function *F);
+
+/// LLVM parser callback which adds !pointeetys metadata during DXIL parsing
+void DXILValueTypeMetadataCallback(Value *V, unsigned TypeID, GetTypeByIDTy GetTypeByID,
+                                   GetContainedTypeIDTy GetContainedTypeID);
+
+} // namespace llvm