[AArch64] Add better fcmp costs for expanded predicates #147940
+720
−708
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Certain fcmp predicates need to be expanded into multiple operations and or'd together. This adds some more accurate core modelling for them based on the predicate. Unsupported operations are given the cost of a libcall and the latency is set to 2 as that seemed to be fairly common between different CPUs.
davemgreen
requested review from
c-rhodes,
hassnaaHamdi,
nasherm and
sjoerdmeijer
llvmbot
added
backend:AArch64
llvm:analysis
|
@llvm/pr-subscribers-backend-aarch64 @llvm/pr-subscribers-llvm-transforms Author: David Green (davemgreen) ChangesCertain fcmp predicates need to be expanded into multiple operations and or'd together. This adds some more accurate core modelling for them based on the predicate. Unsupported operations are given the cost of a libcall and the latency is set to 2 as that seemed to be fairly common between different CPUs. See https://llvm.godbolt.org/z/PsEsbfozG for example codegen, and some SVE compares should improve with #145473. Patch is 208.16 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/147940.diff 7 Files Affected:
diff --git a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp b/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
index 20e7726558117..7ad9e1f9344bd 100644
--- a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
+++ b/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
@@ -4264,10 +4264,9 @@ InstructionCost AArch64TTIImpl::getCmpSelInstrCost(
unsigned Opcode, Type *ValTy, Type *CondTy, CmpInst::Predicate VecPred,
TTI::TargetCostKind CostKind, TTI::OperandValueInfo Op1Info,
TTI::OperandValueInfo Op2Info, const Instruction *I) const {
- int ISD = TLI->InstructionOpcodeToISD(Opcode);
// We don't lower some vector selects well that are wider than the register
// width. TODO: Improve this with different cost kinds.
- if (isa<FixedVectorType>(ValTy) && ISD == ISD::SELECT) {
+ if (isa<FixedVectorType>(ValTy) && Opcode == Instruction::Select) {
// We would need this many instructions to hide the scalarization happening.
const int AmortizationCost = 20;
@@ -4297,55 +4296,72 @@ InstructionCost AArch64TTIImpl::getCmpSelInstrCost(
return LT.first;
}
- static const TypeConversionCostTblEntry
- VectorSelectTbl[] = {
- { ISD::SELECT, MVT::v2i1, MVT::v2f32, 2 },
- { ISD::SELECT, MVT::v2i1, MVT::v2f64, 2 },
- { ISD::SELECT, MVT::v4i1, MVT::v4f32, 2 },
- { ISD::SELECT, MVT::v4i1, MVT::v4f16, 2 },
- { ISD::SELECT, MVT::v8i1, MVT::v8f16, 2 },
- { ISD::SELECT, MVT::v16i1, MVT::v16i16, 16 },
- { ISD::SELECT, MVT::v8i1, MVT::v8i32, 8 },
- { ISD::SELECT, MVT::v16i1, MVT::v16i32, 16 },
- { ISD::SELECT, MVT::v4i1, MVT::v4i64, 4 * AmortizationCost },
- { ISD::SELECT, MVT::v8i1, MVT::v8i64, 8 * AmortizationCost },
- { ISD::SELECT, MVT::v16i1, MVT::v16i64, 16 * AmortizationCost }
- };
+ static const TypeConversionCostTblEntry VectorSelectTbl[] = {
+ {Instruction::Select, MVT::v2i1, MVT::v2f32, 2},
+ {Instruction::Select, MVT::v2i1, MVT::v2f64, 2},
+ {Instruction::Select, MVT::v4i1, MVT::v4f32, 2},
+ {Instruction::Select, MVT::v4i1, MVT::v4f16, 2},
+ {Instruction::Select, MVT::v8i1, MVT::v8f16, 2},
+ {Instruction::Select, MVT::v16i1, MVT::v16i16, 16},
+ {Instruction::Select, MVT::v8i1, MVT::v8i32, 8},
+ {Instruction::Select, MVT::v16i1, MVT::v16i32, 16},
+ {Instruction::Select, MVT::v4i1, MVT::v4i64, 4 * AmortizationCost},
+ {Instruction::Select, MVT::v8i1, MVT::v8i64, 8 * AmortizationCost},
+ {Instruction::Select, MVT::v16i1, MVT::v16i64, 16 * AmortizationCost}};
EVT SelCondTy = TLI->getValueType(DL, CondTy);
EVT SelValTy = TLI->getValueType(DL, ValTy);
if (SelCondTy.isSimple() && SelValTy.isSimple()) {
- if (const auto *Entry = ConvertCostTableLookup(VectorSelectTbl, ISD,
+ if (const auto *Entry = ConvertCostTableLookup(VectorSelectTbl, Opcode,
SelCondTy.getSimpleVT(),
SelValTy.getSimpleVT()))
return Entry->Cost;
}
}
- if (isa<FixedVectorType>(ValTy) && ISD == ISD::SETCC) {
- Type *ValScalarTy = ValTy->getScalarType();
- if ((ValScalarTy->isHalfTy() && !ST->hasFullFP16()) ||
- ValScalarTy->isBFloatTy()) {
- auto *ValVTy = cast<FixedVectorType>(ValTy);
-
- // Without dedicated instructions we promote [b]f16 compares to f32.
- auto *PromotedTy =
- VectorType::get(Type::getFloatTy(ValTy->getContext()), ValVTy);
-
- InstructionCost Cost = 0;
- // Promote operands to float vectors.
- Cost += 2 * getCastInstrCost(Instruction::FPExt, PromotedTy, ValTy,
- TTI::CastContextHint::None, CostKind);
- // Compare float vectors.
+ if (Opcode == Instruction::FCmp) {
+ // Without dedicated instructions we promote f16 + bf16 compares to f32.
+ if ((!ST->hasFullFP16() && ValTy->getScalarType()->isHalfTy()) ||
+ ValTy->getScalarType()->isBFloatTy()) {
+ Type *PromotedTy =
+ ValTy->getWithNewType(Type::getFloatTy(ValTy->getContext()));
+ InstructionCost Cost =
+ getCastInstrCost(Instruction::FPExt, PromotedTy, ValTy,
+ TTI::CastContextHint::None, CostKind);
+ if (!Op1Info.isConstant() && !Op2Info.isConstant())
+ Cost *= 2;
Cost += getCmpSelInstrCost(Opcode, PromotedTy, CondTy, VecPred, CostKind,
Op1Info, Op2Info);
- // During codegen we'll truncate the vector result from i32 to i16.
- Cost +=
- getCastInstrCost(Instruction::Trunc, VectorType::getInteger(ValVTy),
- VectorType::getInteger(PromotedTy),
- TTI::CastContextHint::None, CostKind);
+ if (ValTy->isVectorTy())
+ Cost += getCastInstrCost(
+ Instruction::Trunc, VectorType::getInteger(cast<VectorType>(ValTy)),
+ VectorType::getInteger(cast<VectorType>(PromotedTy)),
+ TTI::CastContextHint::None, CostKind);
return Cost;
}
+
+ auto LT = getTypeLegalizationCost(ValTy);
+ // Model unknown fp compares as a libcall.
+ if (LT.second.getScalarType() != MVT::f64 &&
+ LT.second.getScalarType() != MVT::f32 &&
+ LT.second.getScalarType() != MVT::f16)
+ return LT.first * getCallInstrCost(/*Function*/ nullptr, ValTy,
+ {ValTy, ValTy}, CostKind);
+
+ // Some comparison operators require expanding to multiple compares + or.
+ unsigned Factor = 1;
+ if (!CondTy->isVectorTy() &&
+ (VecPred == FCmpInst::FCMP_ONE || VecPred == FCmpInst::FCMP_UEQ))
+ Factor = 2; // fcmp with 2 selects
+ else if (isa<FixedVectorType>(ValTy) &&
+ (VecPred == FCmpInst::FCMP_ONE || VecPred == FCmpInst::FCMP_UEQ ||
+ VecPred == FCmpInst::FCMP_ORD))
+ Factor = 3; // fcmxx+fcmyy+or
+ else if (isa<ScalableVectorType>(ValTy) &&
+ (VecPred == FCmpInst::FCMP_ONE || VecPred == FCmpInst::FCMP_UEQ))
+ Factor = 3; // fcmxx+fcmyy+or
+
+ return Factor * (CostKind == TTI::TCK_Latency ? 2 : LT.first);
}
// Treat the icmp in icmp(and, 0) or icmp(and, -1/1) when it can be folded to
@@ -4353,7 +4369,7 @@ InstructionCost AArch64TTIImpl::getCmpSelInstrCost(
// comparison is not unsigned. FIXME: Enable for non-throughput cost kinds
// providing it will not cause performance regressions.
if (CostKind == TTI::TCK_RecipThroughput && ValTy->isIntegerTy() &&
- ISD == ISD::SETCC && I && !CmpInst::isUnsigned(VecPred) &&
+ Opcode == Instruction::ICmp && I && !CmpInst::isUnsigned(VecPred) &&
TLI->isTypeLegal(TLI->getValueType(DL, ValTy)) &&
match(I->getOperand(0), m_And(m_Value(), m_Value()))) {
if (match(I->getOperand(1), m_Zero()))
diff --git a/llvm/test/Analysis/CostModel/AArch64/cmp.ll b/llvm/test/Analysis/CostModel/AArch64/cmp.ll
index b55a759a78cc3..270c1e60e7ace 100644
--- a/llvm/test/Analysis/CostModel/AArch64/cmp.ll
+++ b/llvm/test/Analysis/CostModel/AArch64/cmp.ll
@@ -13,14 +13,14 @@ define void @cmps() {
; CHECK-NEXT: Cost Model: Found costs of 1 for: %cv16i8 = icmp slt <16 x i8> undef, undef
; CHECK-NEXT: Cost Model: Found costs of 1 for: %cv8i16 = icmp ult <8 x i16> undef, undef
; CHECK-NEXT: Cost Model: Found costs of 1 for: %cv4i32 = icmp sge <4 x i32> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %cf16 = fcmp oge half undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %cf32 = fcmp ogt float undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %cf64 = fcmp ogt double undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %cbf64 = fcmp ogt bfloat undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:7 CodeSize:4 Lat:4 SizeLat:4 for: %cfv816 = fcmp olt <8 x half> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %cfv432 = fcmp oge <4 x float> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %cfv264 = fcmp oge <2 x double> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:7 CodeSize:4 Lat:4 SizeLat:4 for: %cbfv816 = fcmp olt <8 x bfloat> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:3 CodeSize:3 Lat:4 SizeLat:3 for: %cf16 = fcmp oge half undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %cf32 = fcmp ogt float undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %cf64 = fcmp ogt double undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:3 CodeSize:3 Lat:4 SizeLat:3 for: %cbf64 = fcmp ogt bfloat undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:7 CodeSize:5 Lat:5 SizeLat:5 for: %cfv816 = fcmp olt <8 x half> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %cfv432 = fcmp oge <4 x float> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %cfv264 = fcmp oge <2 x double> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:7 CodeSize:5 Lat:5 SizeLat:5 for: %cbfv816 = fcmp olt <8 x bfloat> undef, undef
; CHECK-NEXT: Cost Model: Found costs of RThru:0 CodeSize:1 Lat:1 SizeLat:1 for: ret void
;
%c8 = icmp slt i8 undef, undef
diff --git a/llvm/test/Analysis/CostModel/AArch64/fcmp.ll b/llvm/test/Analysis/CostModel/AArch64/fcmp.ll
index fb9888d98af6b..0bf195f435fe4 100644
--- a/llvm/test/Analysis/CostModel/AArch64/fcmp.ll
+++ b/llvm/test/Analysis/CostModel/AArch64/fcmp.ll
@@ -4,15 +4,15 @@
define void @fcmp_oeq(i32 %arg) {
; CHECK-LABEL: 'fcmp_oeq'
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %f32 = fcmp oeq float undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %v2f32 = fcmp oeq <2 x float> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %v4f32 = fcmp oeq <4 x float> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:2 CodeSize:1 Lat:1 SizeLat:1 for: %v8f32 = fcmp oeq <8 x float> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %f64 = fcmp oeq double undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %v2f64 = fcmp oeq <2 x double> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:2 CodeSize:1 Lat:1 SizeLat:1 for: %v4f64 = fcmp oeq <4 x double> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %f128 = fcmp oeq fp128 undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:2 CodeSize:1 Lat:1 SizeLat:1 for: %v2f128 = fcmp oeq <2 x fp128> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %f32 = fcmp oeq float undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v2f32 = fcmp oeq <2 x float> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v4f32 = fcmp oeq <4 x float> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of 2 for: %v8f32 = fcmp oeq <8 x float> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %f64 = fcmp oeq double undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v2f64 = fcmp oeq <2 x double> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of 2 for: %v4f64 = fcmp oeq <4 x double> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of 10 for: %f128 = fcmp oeq fp128 undef, undef
+; CHECK-NEXT: Cost Model: Found costs of 20 for: %v2f128 = fcmp oeq <2 x fp128> undef, undef
; CHECK-NEXT: Cost Model: Found costs of RThru:0 CodeSize:1 Lat:1 SizeLat:1 for: ret void
;
%f32 = fcmp oeq float undef, undef
@@ -29,19 +29,19 @@ define void @fcmp_oeq(i32 %arg) {
define void @fcmp_oeq_half(i32 %arg) {
; CHECK-BASE-LABEL: 'fcmp_oeq_half'
-; CHECK-BASE-NEXT: Cost Model: Found costs of 1 for: %f16 = fcmp oeq half undef, undef
-; CHECK-BASE-NEXT: Cost Model: Found costs of 3 for: %v2f16 = fcmp oeq <2 x half> undef, undef
-; CHECK-BASE-NEXT: Cost Model: Found costs of 4 for: %v4f16 = fcmp oeq <4 x half> undef, undef
-; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:7 CodeSize:4 Lat:4 SizeLat:4 for: %v8f16 = fcmp oeq <8 x half> undef, undef
-; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:14 CodeSize:4 Lat:4 SizeLat:4 for: %v16f16 = fcmp oeq <16 x half> undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:3 CodeSize:3 Lat:4 SizeLat:3 for: %f16 = fcmp oeq half undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:3 CodeSize:3 Lat:4 SizeLat:3 for: %v2f16 = fcmp oeq <2 x half> undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:4 CodeSize:4 Lat:5 SizeLat:4 for: %v4f16 = fcmp oeq <4 x half> undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:7 CodeSize:5 Lat:5 SizeLat:5 for: %v8f16 = fcmp oeq <8 x half> undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:14 CodeSize:7 Lat:5 SizeLat:7 for: %v16f16 = fcmp oeq <16 x half> undef, undef
; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:0 CodeSize:1 Lat:1 SizeLat:1 for: ret void
;
; CHECK-FP16-LABEL: 'fcmp_oeq_half'
-; CHECK-FP16-NEXT: Cost Model: Found costs of 1 for: %f16 = fcmp oeq half undef, undef
-; CHECK-FP16-NEXT: Cost Model: Found costs of 1 for: %v2f16 = fcmp oeq <2 x half> undef, undef
-; CHECK-FP16-NEXT: Cost Model: Found costs of 1 for: %v4f16 = fcmp oeq <4 x half> undef, undef
-; CHECK-FP16-NEXT: Cost Model: Found costs of 1 for: %v8f16 = fcmp oeq <8 x half> undef, undef
-; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:2 CodeSize:1 Lat:1 SizeLat:1 for: %v16f16 = fcmp oeq <16 x half> undef, undef
+; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %f16 = fcmp oeq half undef, undef
+; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v2f16 = fcmp oeq <2 x half> undef, undef
+; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v4f16 = fcmp oeq <4 x half> undef, undef
+; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v8f16 = fcmp oeq <8 x half> undef, undef
+; CHECK-FP16-NEXT: Cost Model: Found costs of 2 for: %v16f16 = fcmp oeq <16 x half> undef, undef
; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:0 CodeSize:1 Lat:1 SizeLat:1 for: ret void
;
%f16 = fcmp oeq half undef, undef
@@ -54,11 +54,11 @@ define void @fcmp_oeq_half(i32 %arg) {
define void @fcmp_oeq_bfloat(i32 %arg) {
; CHECK-LABEL: 'fcmp_oeq_bfloat'
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %bf16 = fcmp oeq bfloat undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 3 for: %v2bf16 = fcmp oeq <2 x bfloat> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 4 for: %v4bf16 = fcmp oeq <4 x bfloat> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:7 CodeSize:4 Lat:4 SizeLat:4 for: %v8bf16 = fcmp oeq <8 x bfloat> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:14 CodeSize:4 Lat:4 SizeLat:4 for: %v16bf16 = fcmp oeq <16 x bfloat> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:3 CodeSize:3 Lat:4 SizeLat:3 for: %bf16 = fcmp oeq bfloat undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:3 CodeSize:3 Lat:4 SizeLat:3 for: %v2bf16 = fcmp oeq <2 x bfloat> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:4 CodeSize:4 Lat:5 SizeLat:4 for: %v4bf16 = fcmp oeq <4 x bfloat> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:7 CodeSize:5 Lat:5 SizeLat:5 for: %v8bf16 = fcmp oeq <8 x bfloat> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:14 CodeSize:7 Lat:5 SizeLat:7 for: %v16bf16 = fcmp oeq <16 x bfloat> undef, undef
; CHECK-NEXT: Cost Model: Found costs of RThru:0 CodeSize:1 Lat:1 SizeLat:1 for: ret void
;
%bf16 = fcmp oeq bfloat undef, undef
@@ -71,15 +71,15 @@ define void @fcmp_oeq_bfloat(i32 %arg) {
define void @fcmp_ogt(i32 %arg) {
; CHECK-LABEL: 'fcmp_ogt'
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %f32 = fcmp ogt float undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %v2f32 = fcmp ogt <2 x float> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %v4f32 = fcmp ogt <4 x float> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:2 CodeSize:1 Lat:1 SizeLat:1 for: %v8f32 = fcmp ogt <8 x float> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %f64 = fcmp ogt double undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %v2f64 = fcmp ogt <2 x double> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:2 CodeSize:1 Lat:1 SizeLat:1 for: %v4f64 = fcmp ogt <4 x double> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %f128 = fcmp ogt fp128 undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:2 CodeSize:1 Lat:1 SizeLat:1 for: %v2f128 = fcmp ogt <2 x fp128> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %f32 = fcmp ogt float undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v2f32 = fcmp ogt <2 x float> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v4f32 = fcmp ogt <4 x float> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of 2 for: %v8f32 = fcmp ogt <8 x float> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %f64 = fcmp ogt double undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v2f64 = fcmp ogt <2 x double> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of 2 for: %v4f64 = fcmp ogt <4 x double> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of 10 for: %f128 = fcmp ogt fp128 undef, undef
+; CHECK-NEXT: Cost Model: Found costs of 20 for: %v2f128 = fcmp ogt <2 x fp128> undef, undef
; CHECK-NEXT: Cost Model: Found costs of RThru:0 CodeSize:1 Lat:1 SizeLat:1 for: ret void
;
%f32 = fcmp ogt float undef, undef
@@ -96,19 +96,19 @@ define void @fcmp_ogt(i32 %arg) {
define void @fcmp_ogt_half(i32 %arg) {
; CHECK-BASE-LABEL: 'fcmp_ogt_half'
-; CHECK-BASE-NEXT: Cost Model: Found costs of 1 for: %f16 = fcmp ogt half undef, undef
-; CHECK-BASE-NEXT: Cost Model: Found costs of 3 for: %v2f16 = fcmp ogt <2 x half> undef, undef
-; CHECK-BASE-NEXT: Cost Model: Found costs of 4 for: %v4f16 = fcmp ogt <4 x half> undef, undef
-; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:7 CodeSize:4 Lat:4 SizeLat:4 for: %v8f16 = fcmp ogt <8 x half> undef, undef
-; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:14 CodeSize:4 Lat:4 SizeLat:4 for: %v16f16 = fcmp ogt <16 x half> undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:3 CodeSize:3 Lat:4 SizeLat:3 for: %f16 = fcmp ogt half undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:3 CodeSize:3 Lat:4 SizeLat:3 for: %v2f16 = fcmp ogt <2 x half> undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:4 CodeSize:4 Lat:5 SizeLat:4 for: %v4f16 = fcmp ogt <4 x half> undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:7 CodeSize:5 Lat:5 SizeLat:5 for: %v8f16 = fcmp ogt <8 x half> undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:14 CodeSize:7 Lat:5 SizeLat:7 for: %v16f16 = fcmp ogt <16 x half> undef, undef
; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:0 CodeSize:1 Lat:1 SizeLat:1 for: ret void
;
; CHECK-FP16-LABEL: 'fcmp_ogt_half'
-; CHECK-FP16-NEXT: Cost Model: Found costs of 1 for: %f16 = fcmp ogt half undef, undef
-; CHECK-FP16-NEXT: Cost Model: Found costs of 1 for: %v2f16 = fcmp ogt <2 x half> undef, undef
-; CHECK-FP16-NEXT: Cost Model: Found costs of 1 for: %v4f16 = fcmp ogt <4 x half> undef, undef
-; CHECK-FP16-NEXT: Cost Model: Found costs of 1 for: %v8f16 = fcmp ogt <8 x half> undef, undef
-; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:2 CodeSize:1 Lat:1 SizeLat:1 for: %v16f16 = fcmp ogt <16 x half> undef, undef
+; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %f16 = fcmp ogt half undef, undef
+; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v2f16 = fcmp ogt <2 x half> undef, undef
+; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v4f16 = fcmp ogt <4 x half> undef, undef
+; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 fo...
[truncated]
|
|
@llvm/pr-subscribers-llvm-analysis Author: David Green (davemgreen) ChangesCertain fcmp predicates need to be expanded into multiple operations and or'd together. This adds some more accurate core modelling for them based on the predicate. Unsupported operations are given the cost of a libcall and the latency is set to 2 as that seemed to be fairly common between different CPUs. See https://llvm.godbolt.org/z/PsEsbfozG for example codegen, and some SVE compares should improve with #145473. Patch is 208.16 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/147940.diff 7 Files Affected:
diff --git a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp b/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
index 20e7726558117..7ad9e1f9344bd 100644
--- a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
+++ b/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
@@ -4264,10 +4264,9 @@ InstructionCost AArch64TTIImpl::getCmpSelInstrCost(
unsigned Opcode, Type *ValTy, Type *CondTy, CmpInst::Predicate VecPred,
TTI::TargetCostKind CostKind, TTI::OperandValueInfo Op1Info,
TTI::OperandValueInfo Op2Info, const Instruction *I) const {
- int ISD = TLI->InstructionOpcodeToISD(Opcode);
// We don't lower some vector selects well that are wider than the register
// width. TODO: Improve this with different cost kinds.
- if (isa<FixedVectorType>(ValTy) && ISD == ISD::SELECT) {
+ if (isa<FixedVectorType>(ValTy) && Opcode == Instruction::Select) {
// We would need this many instructions to hide the scalarization happening.
const int AmortizationCost = 20;
@@ -4297,55 +4296,72 @@ InstructionCost AArch64TTIImpl::getCmpSelInstrCost(
return LT.first;
}
- static const TypeConversionCostTblEntry
- VectorSelectTbl[] = {
- { ISD::SELECT, MVT::v2i1, MVT::v2f32, 2 },
- { ISD::SELECT, MVT::v2i1, MVT::v2f64, 2 },
- { ISD::SELECT, MVT::v4i1, MVT::v4f32, 2 },
- { ISD::SELECT, MVT::v4i1, MVT::v4f16, 2 },
- { ISD::SELECT, MVT::v8i1, MVT::v8f16, 2 },
- { ISD::SELECT, MVT::v16i1, MVT::v16i16, 16 },
- { ISD::SELECT, MVT::v8i1, MVT::v8i32, 8 },
- { ISD::SELECT, MVT::v16i1, MVT::v16i32, 16 },
- { ISD::SELECT, MVT::v4i1, MVT::v4i64, 4 * AmortizationCost },
- { ISD::SELECT, MVT::v8i1, MVT::v8i64, 8 * AmortizationCost },
- { ISD::SELECT, MVT::v16i1, MVT::v16i64, 16 * AmortizationCost }
- };
+ static const TypeConversionCostTblEntry VectorSelectTbl[] = {
+ {Instruction::Select, MVT::v2i1, MVT::v2f32, 2},
+ {Instruction::Select, MVT::v2i1, MVT::v2f64, 2},
+ {Instruction::Select, MVT::v4i1, MVT::v4f32, 2},
+ {Instruction::Select, MVT::v4i1, MVT::v4f16, 2},
+ {Instruction::Select, MVT::v8i1, MVT::v8f16, 2},
+ {Instruction::Select, MVT::v16i1, MVT::v16i16, 16},
+ {Instruction::Select, MVT::v8i1, MVT::v8i32, 8},
+ {Instruction::Select, MVT::v16i1, MVT::v16i32, 16},
+ {Instruction::Select, MVT::v4i1, MVT::v4i64, 4 * AmortizationCost},
+ {Instruction::Select, MVT::v8i1, MVT::v8i64, 8 * AmortizationCost},
+ {Instruction::Select, MVT::v16i1, MVT::v16i64, 16 * AmortizationCost}};
EVT SelCondTy = TLI->getValueType(DL, CondTy);
EVT SelValTy = TLI->getValueType(DL, ValTy);
if (SelCondTy.isSimple() && SelValTy.isSimple()) {
- if (const auto *Entry = ConvertCostTableLookup(VectorSelectTbl, ISD,
+ if (const auto *Entry = ConvertCostTableLookup(VectorSelectTbl, Opcode,
SelCondTy.getSimpleVT(),
SelValTy.getSimpleVT()))
return Entry->Cost;
}
}
- if (isa<FixedVectorType>(ValTy) && ISD == ISD::SETCC) {
- Type *ValScalarTy = ValTy->getScalarType();
- if ((ValScalarTy->isHalfTy() && !ST->hasFullFP16()) ||
- ValScalarTy->isBFloatTy()) {
- auto *ValVTy = cast<FixedVectorType>(ValTy);
-
- // Without dedicated instructions we promote [b]f16 compares to f32.
- auto *PromotedTy =
- VectorType::get(Type::getFloatTy(ValTy->getContext()), ValVTy);
-
- InstructionCost Cost = 0;
- // Promote operands to float vectors.
- Cost += 2 * getCastInstrCost(Instruction::FPExt, PromotedTy, ValTy,
- TTI::CastContextHint::None, CostKind);
- // Compare float vectors.
+ if (Opcode == Instruction::FCmp) {
+ // Without dedicated instructions we promote f16 + bf16 compares to f32.
+ if ((!ST->hasFullFP16() && ValTy->getScalarType()->isHalfTy()) ||
+ ValTy->getScalarType()->isBFloatTy()) {
+ Type *PromotedTy =
+ ValTy->getWithNewType(Type::getFloatTy(ValTy->getContext()));
+ InstructionCost Cost =
+ getCastInstrCost(Instruction::FPExt, PromotedTy, ValTy,
+ TTI::CastContextHint::None, CostKind);
+ if (!Op1Info.isConstant() && !Op2Info.isConstant())
+ Cost *= 2;
Cost += getCmpSelInstrCost(Opcode, PromotedTy, CondTy, VecPred, CostKind,
Op1Info, Op2Info);
- // During codegen we'll truncate the vector result from i32 to i16.
- Cost +=
- getCastInstrCost(Instruction::Trunc, VectorType::getInteger(ValVTy),
- VectorType::getInteger(PromotedTy),
- TTI::CastContextHint::None, CostKind);
+ if (ValTy->isVectorTy())
+ Cost += getCastInstrCost(
+ Instruction::Trunc, VectorType::getInteger(cast<VectorType>(ValTy)),
+ VectorType::getInteger(cast<VectorType>(PromotedTy)),
+ TTI::CastContextHint::None, CostKind);
return Cost;
}
+
+ auto LT = getTypeLegalizationCost(ValTy);
+ // Model unknown fp compares as a libcall.
+ if (LT.second.getScalarType() != MVT::f64 &&
+ LT.second.getScalarType() != MVT::f32 &&
+ LT.second.getScalarType() != MVT::f16)
+ return LT.first * getCallInstrCost(/*Function*/ nullptr, ValTy,
+ {ValTy, ValTy}, CostKind);
+
+ // Some comparison operators require expanding to multiple compares + or.
+ unsigned Factor = 1;
+ if (!CondTy->isVectorTy() &&
+ (VecPred == FCmpInst::FCMP_ONE || VecPred == FCmpInst::FCMP_UEQ))
+ Factor = 2; // fcmp with 2 selects
+ else if (isa<FixedVectorType>(ValTy) &&
+ (VecPred == FCmpInst::FCMP_ONE || VecPred == FCmpInst::FCMP_UEQ ||
+ VecPred == FCmpInst::FCMP_ORD))
+ Factor = 3; // fcmxx+fcmyy+or
+ else if (isa<ScalableVectorType>(ValTy) &&
+ (VecPred == FCmpInst::FCMP_ONE || VecPred == FCmpInst::FCMP_UEQ))
+ Factor = 3; // fcmxx+fcmyy+or
+
+ return Factor * (CostKind == TTI::TCK_Latency ? 2 : LT.first);
}
// Treat the icmp in icmp(and, 0) or icmp(and, -1/1) when it can be folded to
@@ -4353,7 +4369,7 @@ InstructionCost AArch64TTIImpl::getCmpSelInstrCost(
// comparison is not unsigned. FIXME: Enable for non-throughput cost kinds
// providing it will not cause performance regressions.
if (CostKind == TTI::TCK_RecipThroughput && ValTy->isIntegerTy() &&
- ISD == ISD::SETCC && I && !CmpInst::isUnsigned(VecPred) &&
+ Opcode == Instruction::ICmp && I && !CmpInst::isUnsigned(VecPred) &&
TLI->isTypeLegal(TLI->getValueType(DL, ValTy)) &&
match(I->getOperand(0), m_And(m_Value(), m_Value()))) {
if (match(I->getOperand(1), m_Zero()))
diff --git a/llvm/test/Analysis/CostModel/AArch64/cmp.ll b/llvm/test/Analysis/CostModel/AArch64/cmp.ll
index b55a759a78cc3..270c1e60e7ace 100644
--- a/llvm/test/Analysis/CostModel/AArch64/cmp.ll
+++ b/llvm/test/Analysis/CostModel/AArch64/cmp.ll
@@ -13,14 +13,14 @@ define void @cmps() {
; CHECK-NEXT: Cost Model: Found costs of 1 for: %cv16i8 = icmp slt <16 x i8> undef, undef
; CHECK-NEXT: Cost Model: Found costs of 1 for: %cv8i16 = icmp ult <8 x i16> undef, undef
; CHECK-NEXT: Cost Model: Found costs of 1 for: %cv4i32 = icmp sge <4 x i32> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %cf16 = fcmp oge half undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %cf32 = fcmp ogt float undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %cf64 = fcmp ogt double undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %cbf64 = fcmp ogt bfloat undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:7 CodeSize:4 Lat:4 SizeLat:4 for: %cfv816 = fcmp olt <8 x half> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %cfv432 = fcmp oge <4 x float> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %cfv264 = fcmp oge <2 x double> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:7 CodeSize:4 Lat:4 SizeLat:4 for: %cbfv816 = fcmp olt <8 x bfloat> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:3 CodeSize:3 Lat:4 SizeLat:3 for: %cf16 = fcmp oge half undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %cf32 = fcmp ogt float undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %cf64 = fcmp ogt double undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:3 CodeSize:3 Lat:4 SizeLat:3 for: %cbf64 = fcmp ogt bfloat undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:7 CodeSize:5 Lat:5 SizeLat:5 for: %cfv816 = fcmp olt <8 x half> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %cfv432 = fcmp oge <4 x float> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %cfv264 = fcmp oge <2 x double> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:7 CodeSize:5 Lat:5 SizeLat:5 for: %cbfv816 = fcmp olt <8 x bfloat> undef, undef
; CHECK-NEXT: Cost Model: Found costs of RThru:0 CodeSize:1 Lat:1 SizeLat:1 for: ret void
;
%c8 = icmp slt i8 undef, undef
diff --git a/llvm/test/Analysis/CostModel/AArch64/fcmp.ll b/llvm/test/Analysis/CostModel/AArch64/fcmp.ll
index fb9888d98af6b..0bf195f435fe4 100644
--- a/llvm/test/Analysis/CostModel/AArch64/fcmp.ll
+++ b/llvm/test/Analysis/CostModel/AArch64/fcmp.ll
@@ -4,15 +4,15 @@
define void @fcmp_oeq(i32 %arg) {
; CHECK-LABEL: 'fcmp_oeq'
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %f32 = fcmp oeq float undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %v2f32 = fcmp oeq <2 x float> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %v4f32 = fcmp oeq <4 x float> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:2 CodeSize:1 Lat:1 SizeLat:1 for: %v8f32 = fcmp oeq <8 x float> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %f64 = fcmp oeq double undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %v2f64 = fcmp oeq <2 x double> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:2 CodeSize:1 Lat:1 SizeLat:1 for: %v4f64 = fcmp oeq <4 x double> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %f128 = fcmp oeq fp128 undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:2 CodeSize:1 Lat:1 SizeLat:1 for: %v2f128 = fcmp oeq <2 x fp128> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %f32 = fcmp oeq float undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v2f32 = fcmp oeq <2 x float> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v4f32 = fcmp oeq <4 x float> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of 2 for: %v8f32 = fcmp oeq <8 x float> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %f64 = fcmp oeq double undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v2f64 = fcmp oeq <2 x double> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of 2 for: %v4f64 = fcmp oeq <4 x double> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of 10 for: %f128 = fcmp oeq fp128 undef, undef
+; CHECK-NEXT: Cost Model: Found costs of 20 for: %v2f128 = fcmp oeq <2 x fp128> undef, undef
; CHECK-NEXT: Cost Model: Found costs of RThru:0 CodeSize:1 Lat:1 SizeLat:1 for: ret void
;
%f32 = fcmp oeq float undef, undef
@@ -29,19 +29,19 @@ define void @fcmp_oeq(i32 %arg) {
define void @fcmp_oeq_half(i32 %arg) {
; CHECK-BASE-LABEL: 'fcmp_oeq_half'
-; CHECK-BASE-NEXT: Cost Model: Found costs of 1 for: %f16 = fcmp oeq half undef, undef
-; CHECK-BASE-NEXT: Cost Model: Found costs of 3 for: %v2f16 = fcmp oeq <2 x half> undef, undef
-; CHECK-BASE-NEXT: Cost Model: Found costs of 4 for: %v4f16 = fcmp oeq <4 x half> undef, undef
-; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:7 CodeSize:4 Lat:4 SizeLat:4 for: %v8f16 = fcmp oeq <8 x half> undef, undef
-; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:14 CodeSize:4 Lat:4 SizeLat:4 for: %v16f16 = fcmp oeq <16 x half> undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:3 CodeSize:3 Lat:4 SizeLat:3 for: %f16 = fcmp oeq half undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:3 CodeSize:3 Lat:4 SizeLat:3 for: %v2f16 = fcmp oeq <2 x half> undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:4 CodeSize:4 Lat:5 SizeLat:4 for: %v4f16 = fcmp oeq <4 x half> undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:7 CodeSize:5 Lat:5 SizeLat:5 for: %v8f16 = fcmp oeq <8 x half> undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:14 CodeSize:7 Lat:5 SizeLat:7 for: %v16f16 = fcmp oeq <16 x half> undef, undef
; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:0 CodeSize:1 Lat:1 SizeLat:1 for: ret void
;
; CHECK-FP16-LABEL: 'fcmp_oeq_half'
-; CHECK-FP16-NEXT: Cost Model: Found costs of 1 for: %f16 = fcmp oeq half undef, undef
-; CHECK-FP16-NEXT: Cost Model: Found costs of 1 for: %v2f16 = fcmp oeq <2 x half> undef, undef
-; CHECK-FP16-NEXT: Cost Model: Found costs of 1 for: %v4f16 = fcmp oeq <4 x half> undef, undef
-; CHECK-FP16-NEXT: Cost Model: Found costs of 1 for: %v8f16 = fcmp oeq <8 x half> undef, undef
-; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:2 CodeSize:1 Lat:1 SizeLat:1 for: %v16f16 = fcmp oeq <16 x half> undef, undef
+; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %f16 = fcmp oeq half undef, undef
+; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v2f16 = fcmp oeq <2 x half> undef, undef
+; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v4f16 = fcmp oeq <4 x half> undef, undef
+; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v8f16 = fcmp oeq <8 x half> undef, undef
+; CHECK-FP16-NEXT: Cost Model: Found costs of 2 for: %v16f16 = fcmp oeq <16 x half> undef, undef
; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:0 CodeSize:1 Lat:1 SizeLat:1 for: ret void
;
%f16 = fcmp oeq half undef, undef
@@ -54,11 +54,11 @@ define void @fcmp_oeq_half(i32 %arg) {
define void @fcmp_oeq_bfloat(i32 %arg) {
; CHECK-LABEL: 'fcmp_oeq_bfloat'
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %bf16 = fcmp oeq bfloat undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 3 for: %v2bf16 = fcmp oeq <2 x bfloat> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 4 for: %v4bf16 = fcmp oeq <4 x bfloat> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:7 CodeSize:4 Lat:4 SizeLat:4 for: %v8bf16 = fcmp oeq <8 x bfloat> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:14 CodeSize:4 Lat:4 SizeLat:4 for: %v16bf16 = fcmp oeq <16 x bfloat> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:3 CodeSize:3 Lat:4 SizeLat:3 for: %bf16 = fcmp oeq bfloat undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:3 CodeSize:3 Lat:4 SizeLat:3 for: %v2bf16 = fcmp oeq <2 x bfloat> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:4 CodeSize:4 Lat:5 SizeLat:4 for: %v4bf16 = fcmp oeq <4 x bfloat> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:7 CodeSize:5 Lat:5 SizeLat:5 for: %v8bf16 = fcmp oeq <8 x bfloat> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:14 CodeSize:7 Lat:5 SizeLat:7 for: %v16bf16 = fcmp oeq <16 x bfloat> undef, undef
; CHECK-NEXT: Cost Model: Found costs of RThru:0 CodeSize:1 Lat:1 SizeLat:1 for: ret void
;
%bf16 = fcmp oeq bfloat undef, undef
@@ -71,15 +71,15 @@ define void @fcmp_oeq_bfloat(i32 %arg) {
define void @fcmp_ogt(i32 %arg) {
; CHECK-LABEL: 'fcmp_ogt'
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %f32 = fcmp ogt float undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %v2f32 = fcmp ogt <2 x float> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %v4f32 = fcmp ogt <4 x float> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:2 CodeSize:1 Lat:1 SizeLat:1 for: %v8f32 = fcmp ogt <8 x float> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %f64 = fcmp ogt double undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %v2f64 = fcmp ogt <2 x double> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:2 CodeSize:1 Lat:1 SizeLat:1 for: %v4f64 = fcmp ogt <4 x double> undef, undef
-; CHECK-NEXT: Cost Model: Found costs of 1 for: %f128 = fcmp ogt fp128 undef, undef
-; CHECK-NEXT: Cost Model: Found costs of RThru:2 CodeSize:1 Lat:1 SizeLat:1 for: %v2f128 = fcmp ogt <2 x fp128> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %f32 = fcmp ogt float undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v2f32 = fcmp ogt <2 x float> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v4f32 = fcmp ogt <4 x float> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of 2 for: %v8f32 = fcmp ogt <8 x float> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %f64 = fcmp ogt double undef, undef
+; CHECK-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v2f64 = fcmp ogt <2 x double> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of 2 for: %v4f64 = fcmp ogt <4 x double> undef, undef
+; CHECK-NEXT: Cost Model: Found costs of 10 for: %f128 = fcmp ogt fp128 undef, undef
+; CHECK-NEXT: Cost Model: Found costs of 20 for: %v2f128 = fcmp ogt <2 x fp128> undef, undef
; CHECK-NEXT: Cost Model: Found costs of RThru:0 CodeSize:1 Lat:1 SizeLat:1 for: ret void
;
%f32 = fcmp ogt float undef, undef
@@ -96,19 +96,19 @@ define void @fcmp_ogt(i32 %arg) {
define void @fcmp_ogt_half(i32 %arg) {
; CHECK-BASE-LABEL: 'fcmp_ogt_half'
-; CHECK-BASE-NEXT: Cost Model: Found costs of 1 for: %f16 = fcmp ogt half undef, undef
-; CHECK-BASE-NEXT: Cost Model: Found costs of 3 for: %v2f16 = fcmp ogt <2 x half> undef, undef
-; CHECK-BASE-NEXT: Cost Model: Found costs of 4 for: %v4f16 = fcmp ogt <4 x half> undef, undef
-; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:7 CodeSize:4 Lat:4 SizeLat:4 for: %v8f16 = fcmp ogt <8 x half> undef, undef
-; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:14 CodeSize:4 Lat:4 SizeLat:4 for: %v16f16 = fcmp ogt <16 x half> undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:3 CodeSize:3 Lat:4 SizeLat:3 for: %f16 = fcmp ogt half undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:3 CodeSize:3 Lat:4 SizeLat:3 for: %v2f16 = fcmp ogt <2 x half> undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:4 CodeSize:4 Lat:5 SizeLat:4 for: %v4f16 = fcmp ogt <4 x half> undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:7 CodeSize:5 Lat:5 SizeLat:5 for: %v8f16 = fcmp ogt <8 x half> undef, undef
+; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:14 CodeSize:7 Lat:5 SizeLat:7 for: %v16f16 = fcmp ogt <16 x half> undef, undef
; CHECK-BASE-NEXT: Cost Model: Found costs of RThru:0 CodeSize:1 Lat:1 SizeLat:1 for: ret void
;
; CHECK-FP16-LABEL: 'fcmp_ogt_half'
-; CHECK-FP16-NEXT: Cost Model: Found costs of 1 for: %f16 = fcmp ogt half undef, undef
-; CHECK-FP16-NEXT: Cost Model: Found costs of 1 for: %v2f16 = fcmp ogt <2 x half> undef, undef
-; CHECK-FP16-NEXT: Cost Model: Found costs of 1 for: %v4f16 = fcmp ogt <4 x half> undef, undef
-; CHECK-FP16-NEXT: Cost Model: Found costs of 1 for: %v8f16 = fcmp ogt <8 x half> undef, undef
-; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:2 CodeSize:1 Lat:1 SizeLat:1 for: %v16f16 = fcmp ogt <16 x half> undef, undef
+; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %f16 = fcmp ogt half undef, undef
+; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v2f16 = fcmp ogt <2 x half> undef, undef
+; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 for: %v4f16 = fcmp ogt <4 x half> undef, undef
+; CHECK-FP16-NEXT: Cost Model: Found costs of RThru:1 CodeSize:1 Lat:2 SizeLat:1 fo...
[truncated]
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Certain fcmp predicates need to be expanded into multiple operations and or'd together. This adds some more accurate core modelling for them based on the predicate. Unsupported operations are given the cost of a libcall and the latency is set to 2 as that seemed to be fairly common between different CPUs.
See https://llvm.godbolt.org/z/PsEsbfozG for example codegen, and some SVE compares should improve with #145473.