Merge branch 'dev' into ck/#69-slackVoltageAngle

# Conflicts: # CHANGELOG.md
ie3-institute · Jun 20, 2022 · 3f186d9 · 3f186d9
2 parents fad3271 + 259d209
commit 3f186d9
Show file tree

Hide file tree

Showing 37 changed files with 1,137 additions and 133 deletions.
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
@@ -8,8 +8,6 @@ updates:
   open-pull-requests-limit: 10
   target-branch: dev
   reviewers:
-  - ckittl
-  - johanneshiry
   - t-ober
   - sensarmad
   - sebastian-peter

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -8,22 +8,51 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 - Implement SQL source for primary data [#34](https://github.com/ie3-institute/simona/issues/34), [#101](https://github.com/ie3-institute/simona/issues/101)
+- Relevant scientific papers have been added to the documentation [#139](https://github.com/ie3-institute/simona/issues/139)
+- Add troubleshooting section to Users guide [#160](https://github.com/ie3-institute/simona/issues/160)
+- Added Kafka sink for results [#24](https://github.com/ie3-institute/simona/issues/24)
 
 ### Changed
 - Re-organizing test resources into their respective packages [#105](https://github.com/ie3-institute/simona/issues/105)
-- BREAKING: Using snapshot version of PSDM
+- BREAKING: Using snapshot version of PSDM and PSU
 - Simplified PrimaryServiceProxy due to changes in PSDM [#120](https://github.com/ie3-institute/simona/issues/120)
 - Improved handling of weights and their sum in determination of weather data [#173](https://github.com/ie3-institute/simona/issues/173)
 - Improving code readability in EvcsAgent by moving FreeLotsRequest to separate methods [#19](https://github.com/ie3-institute/simona/issues/19)
+- Ignore dependabot snapshot dependencies [#27](https://github.com/ie3-institute/simona/issues/27)
 - Sending termination message to external simulation on expected and unexpected shutdowns of SIMONA [#35](https://github.com/ie3-institute/simona/issues/35)
+- Change transformer calculation since changes in PSDM [#99](https://github.com/ie3-institute/simona/issues/99)
+- Adapt to changed PvInputModel of PSDM (elevationAngle) [#100](https://github.com/ie3-institute/simona/issues/100)
+- Consolidate csv parameterization in config [#149](https://github.com/ie3-institute/simona/issues/149)
+- Change weather scheme to COSMO [PR#154](https://github.com/ie3-institute/simona/pull/154)
+- Adapt documentation to changed simonaAPI [#191](https://github.com/ie3-institute/simona/issues/191)
+- Implementing a new plugin framework for external simulations [#195](https://github.com/ie3-institute/simona/issues/195)
 - Improved implementation of `RefSystemParser` [#212](https://github.com/ie3-institute/simona/issues/212)
-- Removed Gradle task puml2png (Converting Plantuml to png / svg files) since it is no longer needed  [#230](https://github.com/ie3-institute/simona/issues/230)
-- Harmonized configuration of csv parameters [#149](https://github.com/ie3-institute/simona/issues/149)
+- Include missing images into Documentation [#151](https://github.com/ie3-institute/simona/issues/151)
+- Changing the export methode for diagrams [#156](https://github.com/ie3-institute/simona/issues/156)
+- Change references implementation in Documentation to bibtex [#174](https://github.com/ie3-institute/simona/issues/174) 
+- Update Model descriptions (Documentation) [#122](https://github.com/ie3-institute/simona/issues/122)
+- Changes of Getting Started Section (Documentation) [#124](https://github.com/ie3-institute/simona/issues/124) 
+- Update gradle [#176](https://github.com/ie3-institute/simona/issues/176)
+- Setting java version to 17 [#58](https://github.com/ie3-institute/simona/issues/58)
+- Made SimonaConfig.BaseRuntimeConfig serializable [#36](https://github.com/ie3-institute/simona/issues/36)
+- Adapt to new simonaAPI snapshot [#95](https://github.com/ie3-institute/simona/issues/95)
+- Update Sphinx to 4.5.0 as well as extensions [#214](https://github.com/ie3-institute/simona/issues/214)
 
 ### Fixed
-- Location of `vn_simona` test grid (was partially in Berlin and Dortmund)
-- Let `ParticipantAgent` die after failed registration with secondary services (prevents stuck simulation)
+- Location of `vn_simona` test grid (was partially in Berlin and Dortmund) [#72](https://github.com/ie3-institute/simona/issues/72)
+- Let `ParticipantAgent` die after failed registration with secondary services (prevents stuck simulation) [#76](https://github.com/ie3-institute/simona/issues/76)
 - Fix default resolution of weather source wrapper [#78](https://github.com/ie3-institute/simona/issues/78)
-- Respect for voltage angle in DBFS slack voltage exchange protocol [#61](https://github.com/ie3-institute/simona/issues/61)
+- Fix invalid thread allocation in GridAgent [#111](https://github.com/ie3-institute/simona/issues/111)
+- Fixed config auto-generation [#130](https://github.com/ie3-institute/simona/issues/130)
+- Fixed genConfigSample gradle task[#148](https://github.com/ie3-institute/simona/issues/148)
+- Fixed some unreachable code [#167](https://github.com/ie3-institute/simona/issues/167)
+- Fix treatment of non-InitializeTrigger triggers in initialization within SimScheduler [#237](https://github.com/ie3-institute/simona/issues/237)
+- Fix breaking SIMONA caused by introducing temperature dependant load profiles in PSDM [#255](https://github.com/ie3-institute/simona/issues/255)
+- Respect for voltage angle in DBFS slack voltage exchange protocol [#69](https://github.com/ie3-institute/simona/issues/69)
+
+### Removed
+- Remove workaround for tscfg tmp directory [#178](https://github.com/ie3-institute/simona/issues/178)
+- Removed Gradle task puml2png (Converting Plantuml to png / svg files) since it is no longer needed  [#228](https://github.com/ie3-institute/simona/issues/228)
+- Remove RocketChat notification from Jenkinsfile [#234](https://github.com/ie3-institute/simona/issues/234)
 
 [Unreleased]: https://github.com/ie3-institute/simona/compare/a14a093239f58fca9b2b974712686b33e5e5f939...HEAD
diff --git a/Dockerfile b/Dockerfile
@@ -0,0 +1,25 @@
+FROM openjdk:17-slim
+
+# USAGE:
+# build with ARG version and if applicable with ARG snapshot suffix
+    # e.g.: docker build --build-arg version=2.1.0 --build-arg snapshotSuffix=-SNAPSHOT -t simona .
+# run by mounting directory
+    # e.g. docker run -v `realpath input`:/input --rm simona
+        # note: this does not work for windows so you have to enter the absolute path manually and escape the \'s
+
+ARG version
+# snapshot suffix for jar files is "-SNAPSHOT"
+ARG snapshotSuffix=""
+
+ENV jarFile="simona-${version}${snapshotSuffix}-all.jar"
+ENV config=""
+
+RUN mkdir exec \
+  && mkdir input
+
+# copy simona fat jar into container
+COPY build/libs/$jarFile /exec/
+# inputData is mounted upon running
+VOLUME /input
+
+ENTRYPOINT ["sh", "-c", "java -jar exec/${jarFile} --config=${config}"]
diff --git a/README.md b/README.md
@@ -34,6 +34,7 @@ CONTRIBUTING.md file in the root directory of this repository.
 For all SIMONA related questions please feel free to contact people involved in the development and maintenance of SIMONA.
 For the moment, these people are:
 
+- Feismann, Daniel - [daniel.feismann@tu-dortmund.de](mailto:daniel.feismann@tu-dortmund.de)
 - Peter, Sebastian - [sebastian.peter@tu-dortmund.de](mailto:sebastian.peter@tu-dortmund.de)
 - Oberließen, Thomas - [thomas.oberliessen@tu-dortmund.de](mailto:thomas.oberliessen@tu-dortmund.de)
 - Sen Sarma, Debopama - [debopama-sen.sarma@tu-dortmund.de](mailto:debopama-sen.sarma@tu-dortmund.de)

diff --git a/build.gradle b/build.gradle
@@ -7,13 +7,13 @@ plugins {
   id 'signing'
   id 'maven-publish' // publish to a maven repo (local or mvn central, has to be defined)
   id 'pmd' // code check, working on source code
-  id 'com.diffplug.spotless' version '6.6.1'// code format
+  id 'com.diffplug.spotless' version '6.7.2'// code format
   id 'com.github.onslip.gradle-one-jar' version '1.0.6' // pack a self contained jar
   id "com.github.ben-manes.versions" version '0.42.0'
   id "de.undercouch.download" version "5.1.0" // downloads plugin
   id "kr.motd.sphinx" version "2.10.1" // documentation generation
   id "com.github.johnrengelman.shadow" version "7.1.2" // fat jar
-  id "org.sonarqube" version "3.3" // sonarqube
+  id "org.sonarqube" version "3.4.0.2513" // sonarqube
   id "org.scoverage" version "7.0.0" // scala code coverage scoverage
   id "com.github.maiflai.scalatest" version "0.32" // run scalatest without specific spec task
   id 'org.hidetake.ssh' version '2.10.1'
@@ -30,7 +30,7 @@ ext {
   tscfgVersion = '0.9.998'
   scapegoatVersion = '1.4.14'
 
-  testContainerVersion = '0.40.7'
+  testContainerVersion = '0.40.8'
 
   scriptsLocation = 'gradle' + File.separator + 'scripts' + File.separator // location of script plugins
 }
@@ -60,6 +60,7 @@ repositories {
   mavenCentral() // searches in Sonatype's central repository
   maven { url 'https://www.jitpack.io' } // allows github repos as dependencies
   maven { url 'https://oss.sonatype.org/content/repositories/snapshots' } // sonatype snapshot repo
+  maven { url "https://packages.confluent.io/maven" } // confluent repo (kafka)
 }
 
 dependencies {
@@ -70,7 +71,7 @@ dependencies {
   }
 
   // ie³ internal repository
-  implementation('com.github.ie3-institute:PowerSystemUtils:1.6') {
+  implementation('com.github.ie3-institute:PowerSystemUtils:2.0-SNAPSHOT') {
     exclude group: 'org.apache.logging.log4j'
     exclude group: 'org.slf4j'
     /* Exclude our own nested dependencies */
@@ -97,13 +98,13 @@ dependencies {
   }
 
   /* logging */
-  implementation "com.typesafe.scala-logging:scala-logging_${scalaVersion}:3.9.4" // akka scala logging
+  implementation "com.typesafe.scala-logging:scala-logging_${scalaVersion}:3.9.5" // akka scala logging
   implementation "ch.qos.logback:logback-classic:1.2.11"
 
   /* testing */
   testImplementation 'org.spockframework:spock-core:2.1-groovy-3.0'
   testImplementation 'org.scalatestplus:mockito-3-4_2.13:3.2.10.0'
-  implementation 'org.mockito:mockito-core:4.5.1' // mocking framework
+  implementation 'org.mockito:mockito-core:4.6.1' // mocking framework
   testImplementation "org.scalatest:scalatest_${scalaVersion}:3.2.12"
   testRuntimeOnly 'com.vladsch.flexmark:flexmark-all:0.64.0' //scalatest html output
   testImplementation group: 'org.pegdown', name: 'pegdown', version: '1.6.0'
@@ -112,6 +113,7 @@ dependencies {
   // testcontainers
   testImplementation "com.dimafeng:testcontainers-scala-scalatest_${scalaVersion}:${testContainerVersion}"
   testImplementation "com.dimafeng:testcontainers-scala-postgresql_${scalaVersion}:${testContainerVersion}"
+  testImplementation "com.dimafeng:testcontainers-scala-kafka_${scalaVersion}:${testContainerVersion}"
 
   /* --- Scala libs --- */
   /* CORE Scala */
@@ -141,6 +143,11 @@ dependencies {
   implementation "com.sksamuel.scapegoat:scalac-scapegoat-plugin_${scalaBinaryVersion}:${scapegoatVersion}"
   scalaCompilerPlugin "com.sksamuel.scapegoat:scalac-scapegoat-plugin_${scalaBinaryVersion}:${scapegoatVersion}"
 
+  /* Kafka */
+  implementation group: 'org.apache.kafka', name: 'kafka-clients', version: '3.2.0'
+  implementation 'io.confluent:kafka-streams-avro-serde:7.1.1'
+  implementation "com.sksamuel.avro4s:avro4s-core_${scalaVersion}:4.1.0"
+
   implementation 'org.apache.commons:commons-math3:3.6.1' // apache commons math3
   implementation 'org.apache.poi:poi-ooxml:5.2.2' // used for FilenameUtils
   implementation 'javax.measure:unit-api:2.1.3'
@@ -156,6 +163,14 @@ tasks.withType(JavaCompile) {
   options.encoding = 'UTF-8'
 }
 
+jar {
+  manifest {
+    attributes(
+        'Main-Class': 'edu.ie3.simona.main.RunSimonaStandalone'
+        )
+  }
+}
+
 //////////////////////////////////////////////////////////////////////
 // Build akka'able fat jar using the gradle shadow plugin
 // see http://www.sureshpw.com/2015/10/building-akka-bundle-with-all.html

diff --git a/input/samples/vn_simona/fullGrid/load_input.csv b/input/samples/vn_simona/fullGrid/load_input.csv
@@ -1,4 +1,4 @@
-"uuid","cos_phi_rated","dsm","e_cons_annual","id","node","operates_from","operates_until","operator","q_characteristics","s_rated","standard_load_profile"
+"uuid","cos_phi_rated","dsm","e_cons_annual","id","node","operates_from","operates_until","operator","q_characteristics","s_rated","load_profile"
 c2402412-97fa-4ca4-aa66-e6e04d010001,0.9700000286102295,false,4000.0,NS_NET126_L_F1_(36),ca3391eb-ca94-4945-ac72-e116f396f82c,,,,cosPhiFixed:{(0.00,1.00)},2.0618600845336914,h0
 fa8ef266-5b15-4fdd-a145-71ba95e3463d,0.949999988079071,false,4000.0,NS_NET146_L_F3_(17),0f3ba59d-a9ce-4669-aa12-bebec42238b7,,,,cosPhiFixed:{(0.00,1.00)},2.3157899379730225,h0
 4dd0785a-482c-47e3-bb82-e315083684d1,0.9700000286102295,false,4000.0,NS_NET116_L_S3_2(6),550ebca7-1455-44eb-9431-ffbf08e58bd4,,,,cosPhiFixed:{(0.00,1.00)},4.1237101554870605,h0

diff --git a/src/main/resources/config/config-template.conf b/src/main/resources/config/config-template.conf
@@ -70,6 +70,20 @@ PrimaryDataCsvParams {
   timePattern: string | "yyyy-MM-dd'T'HH:mm:ss[.S[S][S]]'Z'" # default pattern from PSDM:TimeBasedSimpleValueFactory
 }
 
+#@define abstract
+KafkaParams {
+  runId: string
+  bootstrapServers: string
+  schemaRegistryUrl: string
+  linger: int // in ms
+}
+
+#@define extends KafkaParams
+ResultKafkaParams {
+  base: KafkaParams
+  topicNodeRes = string
+}
+
 #@define
 BaseOutputConfig {
   notifier: string           # Result event notifier
@@ -201,6 +215,9 @@ simona.output.sink.influxDb1x {
   database = string
 }
 
+#@optional
+simona.output.sink.kafka = ResultKafkaParams
+
 simona.output.grid = GridOutputConfig
 simona.output.participant = {
   defaultConfig = BaseOutputConfig

diff --git a/src/main/scala/edu/ie3/simona/config/ConfigFailFast.scala b/src/main/scala/edu/ie3/simona/config/ConfigFailFast.scala
@@ -9,14 +9,21 @@ package edu.ie3.simona.config
 import com.typesafe.config.{Config, ConfigException}
 import com.typesafe.scalalogging.LazyLogging
 import edu.ie3.simona.config.SimonaConfig.Simona.Output.Sink.InfluxDb1x
-import edu.ie3.simona.config.SimonaConfig.{BaseOutputConfig, RefSystemConfig}
+import edu.ie3.simona.config.SimonaConfig.{
+  BaseOutputConfig,
+  RefSystemConfig,
+  ResultKafkaParams
+}
 import edu.ie3.simona.exceptions.InvalidConfigParameterException
 import edu.ie3.simona.io.result.ResultSinkType
 import edu.ie3.simona.model.participant.load.{LoadModelBehaviour, LoadReference}
 import edu.ie3.simona.service.primary.PrimaryServiceProxy
 import edu.ie3.simona.service.weather.WeatherSource
 import edu.ie3.simona.util.CollectionUtils
-import edu.ie3.simona.util.ConfigUtil.DatabaseConfigUtil.checkInfluxDb1xParams
+import edu.ie3.simona.util.ConfigUtil.DatabaseConfigUtil.{
+  checkInfluxDb1xParams,
+  checkKafkaParams
+}
 import edu.ie3.simona.util.ConfigUtil.{CsvConfigUtil, NotifierIdentifier}
 import edu.ie3.util.scala.ReflectionTools
 import edu.ie3.util.{StringUtils, TimeUtil}
@@ -138,7 +145,7 @@ case object ConfigFailFast extends LazyLogging {
     */
   private def checkDataSink(sink: SimonaConfig.Simona.Output.Sink): Unit = {
     // ensures failure if new output sinks are added to enforce adaptions of the check sink method as well
-    val supportedSinks = Set("influxdb1x", "csv")
+    val supportedSinks = Set("influxdb1x", "csv", "kafka")
     if (
       !sink.productElementNames
         .map(_.trim.toLowerCase)
@@ -147,7 +154,7 @@ case object ConfigFailFast extends LazyLogging {
     )
       throw new InvalidConfigParameterException(
         s"Newly added sink(s) " +
-          s"'${sink.productElementNames.toSet.removedAll(supportedSinks)}' detected! " +
+          s"'${sink.productElementNames.map(_.toLowerCase).toSet.removedAll(supportedSinks)}' detected! " +
           s"Please adapt 'ConfigFailFast' accordingly! Currently supported sinks: ${supportedSinks.mkString(", ")}."
       )
 
@@ -172,15 +179,16 @@ case object ConfigFailFast extends LazyLogging {
           "one sink is configured!"
       )
 
-    // if this is db sink, check the connection
     sinkConfigs.find(_.isDefined) match {
       case Some(Some(influxDb1x: InfluxDb1x)) =>
         checkInfluxDb1xParams(
           "Sink",
           ResultSinkType.buildInfluxDb1xUrl(influxDb1x),
           influxDb1x.database
         )
-      case _ => // no db connection, do nothing
+      case Some(Some(kafka: ResultKafkaParams)) =>
+        checkKafkaParams(kafka, Seq(kafka.topicNodeRes))
+      case _ => // do nothing
     }
 
   }

diff --git a/src/main/scala/edu/ie3/simona/config/SimonaConfig.scala b/src/main/scala/edu/ie3/simona/config/SimonaConfig.scala
@@ -290,6 +290,13 @@ object SimonaConfig {
 
   }
 
+  sealed abstract class KafkaParams(
+      val bootstrapServers: java.lang.String,
+      val linger: scala.Int,
+      val runId: java.lang.String,
+      val schemaRegistryUrl: java.lang.String
+  )
+
   final case class LoadRuntimeConfig(
       override val calculateMissingReactivePowerWithModel: scala.Boolean,
       override val scaling: scala.Double,
@@ -554,6 +561,63 @@ object SimonaConfig {
 
   }
 
+  final case class ResultKafkaParams(
+      override val bootstrapServers: java.lang.String,
+      override val linger: scala.Int,
+      override val runId: java.lang.String,
+      override val schemaRegistryUrl: java.lang.String,
+      topicNodeRes: java.lang.String
+  ) extends KafkaParams(bootstrapServers, linger, runId, schemaRegistryUrl)
+  object ResultKafkaParams {
+    def apply(
+        c: com.typesafe.config.Config,
+        parentPath: java.lang.String,
+        $tsCfgValidator: $TsCfgValidator
+    ): SimonaConfig.ResultKafkaParams = {
+      SimonaConfig.ResultKafkaParams(
+        topicNodeRes = $_reqStr(parentPath, c, "topicNodeRes", $tsCfgValidator),
+        bootstrapServers =
+          $_reqStr(parentPath, c, "bootstrapServers", $tsCfgValidator),
+        linger = $_reqInt(parentPath, c, "linger", $tsCfgValidator),
+        runId = $_reqStr(parentPath, c, "runId", $tsCfgValidator),
+        schemaRegistryUrl =
+          $_reqStr(parentPath, c, "schemaRegistryUrl", $tsCfgValidator)
+      )
+    }
+    private def $_reqInt(
+        parentPath: java.lang.String,
+        c: com.typesafe.config.Config,
+        path: java.lang.String,
+        $tsCfgValidator: $TsCfgValidator
+    ): scala.Int = {
+      if (c == null) 0
+      else
+        try c.getInt(path)
+        catch {
+          case e: com.typesafe.config.ConfigException =>
+            $tsCfgValidator.addBadPath(parentPath + path, e)
+            0
+        }
+    }
+
+    private def $_reqStr(
+        parentPath: java.lang.String,
+        c: com.typesafe.config.Config,
+        path: java.lang.String,
+        $tsCfgValidator: $TsCfgValidator
+    ): java.lang.String = {
+      if (c == null) null
+      else
+        try c.getString(path)
+        catch {
+          case e: com.typesafe.config.ConfigException =>
+            $tsCfgValidator.addBadPath(parentPath + path, e)
+            null
+        }
+    }
+
+  }
+
   final case class VoltLvlConfig(
       id: java.lang.String,
       vNom: java.lang.String
@@ -1511,7 +1575,8 @@ object SimonaConfig {
 
       final case class Sink(
           csv: scala.Option[SimonaConfig.Simona.Output.Sink.Csv],
-          influxDb1x: scala.Option[SimonaConfig.Simona.Output.Sink.InfluxDb1x]
+          influxDb1x: scala.Option[SimonaConfig.Simona.Output.Sink.InfluxDb1x],
+          kafka: scala.Option[SimonaConfig.ResultKafkaParams]
       )
       object Sink {
         final case class Csv(
@@ -1618,6 +1683,16 @@ object SimonaConfig {
                     $tsCfgValidator
                   )
                 )
+              else None,
+            kafka =
+              if (c.hasPathOrNull("kafka"))
+                scala.Some(
+                  SimonaConfig.ResultKafkaParams(
+                    c.getConfig("kafka"),
+                    parentPath + "kafka.",
+                    $tsCfgValidator
+                  )
+                )
               else None
           )
         }