diff --git a/docs/explanations/1-runtime-features.md b/docs/explanations/1-runtime-features.md new file mode 100644 index 000000000..ffa8dc5e9 --- /dev/null +++ b/docs/explanations/1-runtime-features.md @@ -0,0 +1,3 @@ + + +Challenge: Submit an issue or PR with an improvement \ No newline at end of file diff --git a/docs/explanations/2-compile-time-features.md b/docs/explanations/2-compile-time-features.md new file mode 100644 index 000000000..ffa8dc5e9 --- /dev/null +++ b/docs/explanations/2-compile-time-features.md @@ -0,0 +1,3 @@ + + +Challenge: Submit an issue or PR with an improvement \ No newline at end of file diff --git a/docs/explanations/3-checking-of-correctness.md b/docs/explanations/3-checking-of-correctness.md new file mode 100644 index 000000000..a5ef0a7d0 --- /dev/null +++ b/docs/explanations/3-checking-of-correctness.md @@ -0,0 +1,4 @@ +Printing of generated codecs with macro options (implicit val printCodec: CodecMakerConfig.PrintCodec = new CodecMakerConfig.PrintCodec {}) +Using Java decompilers (CFR) + +Challenge: Find a missing test coverage for some condition or hidden feature and submit an issue (or a PR with a fix) \ No newline at end of file diff --git a/docs/explanations/4-integrations-with-others.md b/docs/explanations/4-integrations-with-others.md new file mode 100644 index 000000000..e69de29bb diff --git a/docs/explanations/5-performance-benchmarks.md b/docs/explanations/5-performance-benchmarks.md new file mode 100644 index 000000000..8bda74dee --- /dev/null +++ b/docs/explanations/5-performance-benchmarks.md @@ -0,0 +1,6 @@ +How to run JVM and Scala.js benchmarks + +Synthetic (for focusing on parsing/serialization of different value types and collections) + +Real world format/API samples + +Challenge: \ No newline at end of file diff --git a/docs/explanations/6-impact-on-others.md b/docs/explanations/6-impact-on-others.md new file mode 100644 index 000000000..ef5c60779 --- /dev/null +++ b/docs/explanations/6-impact-on-others.md @@ -0,0 +1,3 @@ +framework benchmarks +safety improvements in play-json, spray-json, upickle +fast performance feedback for jackson-module-scala (and underlying jackson-core), circe \ No newline at end of file diff --git a/docs/how-tos/1-how-to-double-check-safety-and-correctness.md b/docs/how-tos/1-how-to-double-check-safety-and-correctness.md new file mode 100644 index 000000000..4bc3e27d9 --- /dev/null +++ b/docs/how-tos/1-how-to-double-check-safety-and-correctness.md @@ -0,0 +1,3 @@ +- Check if some security sensitive options of compile-time and runtime configuration are safe for your use cases +- Print and analyze sources of generated codecs (check number of anonymous classes, size of generated methods, etc.) +- Patch jsoniter-scala sources with some instrumentation, build and publish locally (as example to check number of instantiated codecs) diff --git a/docs/how-tos/2-how-to-squize-more-performance.md b/docs/how-tos/2-how-to-squize-more-performance.md new file mode 100644 index 000000000..f338e5923 --- /dev/null +++ b/docs/how-tos/2-how-to-squize-more-performance.md @@ -0,0 +1,24 @@ +Tools to check: +- Load test with realistic or synthetic input using async-profiler in different modes (-e cpu, alloc, wall) +- Write JMH benchmarks with realistic or synthetic input using different profilers (gc, perfnorm, perfasm) +- Decompile byte code with CFR decompiler to Java + +Options to consider: +- Tune sizes of internal buffers (do not rely on defaults) +- Do pretty printing or use exception stack traces only for debugging +- Use reading and writing from/to pre-allocated sub-arrays +- Use short up to 8 field names per case class (not more than 64 characters in total) +- Use ASCII field and class names +- Turn off checking of field duplication (when parsing on client side) +- Avoid using string types for data that could be immediately parsed to UUID, data-time, and numbers +- Prefer arrays (or immutable arrays in Scala 3) for JSON arrays, especially when they have primitive type values +- Prefer discriminator keys and avoid discriminator fields +- Use JSON arrays for some stable data structures with all required values (2D/3D coordinates, price/quantity, etc.) +- In product types define required fields first, then most frequently used optional fields +- In enumeration and sum-types define most frequently used classes/objects first +- Turn off hex dumps for exceptions +- Generate decoders/encoders only (reduce code size for Scala.js) + +Anti-patterns: +- Generation of codecs for intermediate (non-top level) data structures (as an example, overuse of `derives` keyword) +- Generation and wide usage of codecs for primitives, strings, etc. diff --git a/docs/how-tos/3-how-to-reduce-maintenance-costs.md b/docs/how-tos/3-how-to-reduce-maintenance-costs.md new file mode 100644 index 000000000..1155fc586 --- /dev/null +++ b/docs/how-tos/3-how-to-reduce-maintenance-costs.md @@ -0,0 +1,6 @@ +Approaches to consider: +- Switch to Scala 3 (safer and faster generation of smaller codecs) +- Use data-structures that follow JSON representation as close as possible (don't forget compile-time configuration during derivation or using annotations) +- Use 2 data models for huge projects with different API versions or 3-rd party data structures (use chimney or ducktape for transformation between them) +- Use GraphQL (caliban) for reach and highly customized requests +- Use Smithy (smithy4s-json) for model first approach with cross-language APIs diff --git a/docs/how-tos/4-how-to-extract-and-serialize-raw-JSON-representation.md b/docs/how-tos/4-how-to-extract-and-serialize-raw-JSON-representation.md new file mode 100644 index 000000000..7b2d05868 --- /dev/null +++ b/docs/how-tos/4-how-to-extract-and-serialize-raw-JSON-representation.md @@ -0,0 +1 @@ +Example: api-key signing \ No newline at end of file diff --git a/docs/how-tos/5-how-to-wrap-JSON-representation-into-JSON-string.md b/docs/how-tos/5-how-to-wrap-JSON-representation-into-JSON-string.md new file mode 100644 index 000000000..c2ad12bac --- /dev/null +++ b/docs/how-tos/5-how-to-wrap-JSON-representation-into-JSON-string.md @@ -0,0 +1 @@ +Example: OpenRTB native \ No newline at end of file diff --git a/docs/how-tos/6-how-to-validate-schemaless-JSON.md b/docs/how-tos/6-how-to-validate-schemaless-JSON.md new file mode 100644 index 000000000..49120f3e2 --- /dev/null +++ b/docs/how-tos/6-how-to-validate-schemaless-JSON.md @@ -0,0 +1 @@ +Example: jsontier-scala-examples/example_02.sc \ No newline at end of file diff --git a/docs/how-tos/7-how-to-parse-and-serialize-schemaless-JSON.md b/docs/how-tos/7-how-to-parse-and-serialize-schemaless-JSON.md new file mode 100644 index 000000000..be3e7b782 --- /dev/null +++ b/docs/how-tos/7-how-to-parse-and-serialize-schemaless-JSON.md @@ -0,0 +1 @@ +dijon, jsoniter-scala-circe, play-json-jsoniter diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 000000000..772f414d4 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,19 @@ +1. Goals + +- Security +- Correctness +- Performance +- Productivity + +2. Features + +3. Tutorials + +4. How-tos + +5. API docs + https://www.javadoc.io/doc/com.github.plokhotnyuk.jsoniter-scala + +6. Known-issues + +7. Blog posts \ No newline at end of file diff --git a/docs/tutorials/1-getting-started.md b/docs/tutorials/1-getting-started.md new file mode 100644 index 000000000..84d9d4b33 --- /dev/null +++ b/docs/tutorials/1-getting-started.md @@ -0,0 +1,202 @@ +# Getting started +Let's start our adventure with jsoniter-scala from parsing and serialization of some complex data structure of nested +collections and case classes. + +## Prerequisites +In all tutorials we will use [scala-cli](https://scala-cli.virtuslab.org) for running Scala scripts, so you'll need to install it upfront. + +You can use any text editor to copy and paste provided code snippets. Please, also, check Scala CLI Cookbooks +documentation if you use [VSCode](https://scala-cli.virtuslab.org/docs/cookbooks/ide/vscode), [Intellij IDEA](https://scala-cli.virtuslab.org/docs/cookbooks/ide/intellij), or [emacs](https://scala-cli.virtuslab.org/docs/cookbooks/ide/emacs) editors. + +Support of Scala CLI by IDEs is improving over time, so that [IntelliJ IDEA](https://www.jetbrains.com/idea/) with the latest version of Scala +plugin you can try its [improved support for Scala CLI projects](https://blog.jetbrains.com/scala/2024/11/13/intellij-scala-plugin-2024-3-is-out/#scala-cli). + +Latest versions of jsoniter-scala libraries require JDK 11 or above. You can use any of its distributions. + +In all tutorials we will start from a file with the following dependencies and imports: + +```scala +//> using scala 3 +//> using dep "com.github.plokhotnyuk.jsoniter-scala::jsoniter-scala-core::2.31.3" +//> using compileOnly.dep "com.github.plokhotnyuk.jsoniter-scala::jsoniter-scala-macros::2.31.3" + +import com.github.plokhotnyuk.jsoniter_scala.core._ +import com.github.plokhotnyuk.jsoniter_scala.macros._ +``` + +Please just copy and paste code snippets from the subsequent steps and run by pressing some `Run` button/keystroke in +your IDE or run it using `scala-cli ` command from the terminal. + +Make sure to preserve indentation when copying and pasting because we will use indenting syntax of Scala 3. + +## Definition of a complex data structure + +Let's imagine that we need to generate a JSON representation of some report for a US shop. + +Below is a definition of its data structures and instantiation of some sample data: + +```scala +enum Category extends Enum[Category]: + case Electronics, Fashion, HomeGoods + +case class Product(id: Long, name: String, category: Category, price: BigDecimal, description: String) + +enum OrderStatus extends Enum[OrderStatus]: + case Pending, Shipped, Delivered, Cancelled + +case class OrderItem(product: Product, quantity: Int) + +case class Order(id: Long, customer: Customer, items: Seq[OrderItem], status: OrderStatus) + +case class Customer(id: Long, name: String, email: String, address: Address) + +case class Address(street: String, city: String, state: String, zip: String) + +enum PaymentMethod: + case CreditCard(cardNumber: Long, validThru: java.time.YearMonth) extends PaymentMethod + case PayPal(id: String) extends PaymentMethod + +case class Payment(method: PaymentMethod, amount: BigDecimal, timestamp: java.time.Instant) + +case class OrderPayment(order: Order, payment: Payment) + +val product1 = Product( + id = 1L, + name = "Apple iPhone 16", + category = Category.Electronics, + price = BigDecimal(999.99), + description = "A high-end smartphone with advanced camera and AI capabilities" +) +val product2 = Product( + id = 2L, + name = "Nike Air Max 270", + category = Category.Fashion, + price = BigDecimal(129.99), + description = "A stylish and comfortable sneaker with a full-length air unit" +) +val product3 = Product( + id = 3L, + name = "KitchenAid Stand Mixer", + category = Category.HomeGoods, + price = BigDecimal(299.99), + description = "A versatile and powerful stand mixer for baking and cooking" +) +val customer1 = Customer( + id = 1L, + name = "John Doe", + email = "john.doe@example.com", + address = Address( + street = "123 Main St", + city = "Anytown", + state = "CA", + zip = "12345" + ) +) +val customer2 = Customer( + id = 2L, + name = "Jane Smith", + email = "jane.smith@example.com", + address = Address( + street = "456 Elm St", + city = "Othertown", + state = "NY", + zip = "67890" + ) +) +val order1 = Order( + id = 1L, + customer = customer1, + items = Seq( + OrderItem(product1, 1), + OrderItem(product2, 2) + ), + status = OrderStatus.Pending +) +val order2 = Order( + id = 2L, + customer = customer2, + items = Seq( + OrderItem(product3, 1) + ), + status = OrderStatus.Shipped +) +val paymentMethod1 = PaymentMethod.CreditCard( + cardNumber = 1234_5678_9012_3456L, + validThru = java.time.YearMonth.parse("2026-12") +) +val paymentMethod2 = PaymentMethod.PayPal( + id = "jane.smith@example.com" +) +val payment1 = Payment( + method = paymentMethod1, + amount = BigDecimal(1259.97), + timestamp = java.time.Instant.parse("2025-01-03T12:30:45Z") +) +val payment2 = Payment( + method = paymentMethod2, + amount = BigDecimal(299.99), + timestamp = java.time.Instant.parse("2025-01-15T19:10:55Z") +) +val orderPayment1 = OrderPayment( + order = order1, + payment = payment1 +) +val orderPayment2 = OrderPayment( + order = order2, + payment = payment2 +) +val report = Seq( + orderPayment1, + orderPayment2 +) +``` + +## Defining the codec + +Now we need to derive a codec for the report type (`List[OrderPayment]` type in our case). We will use +`JsonCodecMaker.make` macros for that: +```scala +given JsonValueCodec[Seq[OrderPayment]] = JsonCodecMaker.make +``` + +An instance of this codec (also known as a type-class instance) is getting to be visible in the scope of subsequent +calls of parsing and serialization methods. + +## Serialization + +Now we are ready to serialize the report. For that we need to define some entry point method and call `writeToString`. +We will also print resulting JSON to the system output to see it as an output: + +```scala +@main def gettingStarted: Unit = + val json = writeToString(report) + println(json) +``` + +From this moment you can run the script and see a long JSON string on the screen. + +## Parsing + +Having the JSON string in memory you can parse it using following lines that should be pasted to the end of the +`gettingStarted` method: +```scala + val parsedReport = readFromString(json) + println(parsedReport) +``` + +Let's rerun the script and get additionally printed `toString` representation of a report parsed from JSON string. + +If something gone wrong you can pick and run [the final version of a script for this tutorial](1-getting-started.scala). + +## Challenge +Experiment with the script to use different types of collections instead of `Seq` and try to find any collection type +from the standard Scala library that is not supported by `JsonCodecMaker.make` macros to derive codec for serialization +and parsing. + +## Recap +In this tutorial we learned basics for parsing and serialization of complex nested data structures using `scala-cli`. +Having definition of data structures and their instances in memory we need to: +1. Add dependency usage and package imports of `core` and `macros` modules of jsoniter-scala +2. Define `given` type-class instance of `JsonValueCodec` for top-level data structure +3. Call `writeToString` to serialize an instance of complex data structure to JSON representation +4. Call `readFromString` to parse a complex data structure from JSON representation diff --git a/docs/tutorials/1-getting-started.scala b/docs/tutorials/1-getting-started.scala new file mode 100644 index 000000000..97822313b --- /dev/null +++ b/docs/tutorials/1-getting-started.scala @@ -0,0 +1,128 @@ +//> using scala 3 +//> using dep "com.github.plokhotnyuk.jsoniter-scala::jsoniter-scala-core::2.31.3" +//> using compileOnly.dep "com.github.plokhotnyuk.jsoniter-scala::jsoniter-scala-macros::2.31.3" + +import com.github.plokhotnyuk.jsoniter_scala.core._ +import com.github.plokhotnyuk.jsoniter_scala.macros._ + +enum Category extends Enum[Category]: + case Electronics, Fashion, HomeGoods + +case class Product(id: Long, name: String, category: Category, price: BigDecimal, description: String) + +enum OrderStatus extends Enum[OrderStatus]: + case Pending, Shipped, Delivered, Cancelled + +case class OrderItem(product: Product, quantity: Int) + +case class Order(id: Long, customer: Customer, items: Seq[OrderItem], status: OrderStatus) + +case class Customer(id: Long, name: String, email: String, address: Address) + +case class Address(street: String, city: String, state: String, zip: String) + +enum PaymentMethod: + case CreditCard(cardNumber: Long, validThru: java.time.YearMonth) extends PaymentMethod + case PayPal(id: String) extends PaymentMethod + +case class Payment(method: PaymentMethod, amount: BigDecimal, timestamp: java.time.Instant) + +case class OrderPayment(order: Order, payment: Payment) + +val product1 = Product( + id = 1L, + name = "Apple iPhone 16", + category = Category.Electronics, + price = BigDecimal(999.99), + description = "A high-end smartphone with advanced camera and AI capabilities" +) +val product2 = Product( + id = 2L, + name = "Nike Air Max 270", + category = Category.Fashion, + price = BigDecimal(129.99), + description = "A stylish and comfortable sneaker with a full-length air unit" +) +val product3 = Product( + id = 3L, + name = "KitchenAid Stand Mixer", + category = Category.HomeGoods, + price = BigDecimal(299.99), + description = "A versatile and powerful stand mixer for baking and cooking" +) +val customer1 = Customer( + id = 1L, + name = "John Doe", + email = "john.doe@example.com", + address = Address( + street = "123 Main St", + city = "Anytown", + state = "CA", + zip = "12345" + ) +) +val customer2 = Customer( + id = 2L, + name = "Jane Smith", + email = "jane.smith@example.com", + address = Address( + street = "456 Elm St", + city = "Othertown", + state = "NY", + zip = "67890" + ) +) +val order1 = Order( + id = 1L, + customer = customer1, + items = Seq( + OrderItem(product1, 1), + OrderItem(product2, 2) + ), + status = OrderStatus.Pending +) +val order2 = Order( + id = 2L, + customer = customer2, + items = Seq( + OrderItem(product3, 1) + ), + status = OrderStatus.Shipped +) +val paymentMethod1 = PaymentMethod.CreditCard( + cardNumber = 1234_5678_9012_3456L, + validThru = java.time.YearMonth.parse("2026-12") +) +val paymentMethod2 = PaymentMethod.PayPal( + id = "jane.smith@example.com" +) +val payment1 = Payment( + method = paymentMethod1, + amount = BigDecimal(1259.97), + timestamp = java.time.Instant.parse("2025-01-03T12:30:45Z") +) +val payment2 = Payment( + method = paymentMethod2, + amount = BigDecimal(299.99), + timestamp = java.time.Instant.parse("2025-01-15T19:10:55Z") +) +val orderPayment1 = OrderPayment( + order = order1, + payment = payment1 +) +val orderPayment2 = OrderPayment( + order = order2, + payment = payment2 +) +val report = Seq( + orderPayment1, + orderPayment2 +) + +given JsonValueCodec[Seq[OrderPayment]] = JsonCodecMaker.make + +@main def gettingStarted: Unit = + val json = writeToString(report) + println(json) + val parsedReport = readFromString(json) + println(parsedReport) diff --git a/docs/tutorials/2-use-compile-time-configuration.md b/docs/tutorials/2-use-compile-time-configuration.md new file mode 100644 index 000000000..448746065 --- /dev/null +++ b/docs/tutorials/2-use-compile-time-configuration.md @@ -0,0 +1,9 @@ +1. CodecMakerConfiguration (renaming, stringification, enabling of recursion, disabling generation of decoding or encoding implementations, etc.) +2. Static annotations at field and class definitions and their priority over CodecMakerConfiguration +3. Codec injection using implicits or givens + +## Challenge +Experiment with different combinations of compile-time options and find those that are not supported. +Will compiler errors explain why codecs cannot be generated for such options? + +## Recap diff --git a/docs/tutorials/3-use-runtime-configuration.md b/docs/tutorials/3-use-runtime-configuration.md new file mode 100644 index 000000000..2d51b3611 --- /dev/null +++ b/docs/tutorials/3-use-runtime-configuration.md @@ -0,0 +1,11 @@ +Examples with runtime configuration options for: +- Pretty printing and stack traces for debugging +- Max buffer sizes for safety when parsing from `java.io.InputStream` (or `java.nio.DirectByteBuffer`) +- Buffer sizes and disabling of exception hex dump for performance + +## Challenge +Serialize the following string to JSON representation using `writeToString` and then convert resulting string to +byte array using some non UTF-8 charset and then parse it back using `readFromByteArray`. Will jsoniter-scala +parser throw an error with a helpful message? + +## Recap \ No newline at end of file diff --git a/docs/tutorials/4-derive-codecs-for-algebraic-data-types.md b/docs/tutorials/4-derive-codecs-for-algebraic-data-types.md new file mode 100644 index 000000000..8d69ac21a --- /dev/null +++ b/docs/tutorials/4-derive-codecs-for-algebraic-data-types.md @@ -0,0 +1,6 @@ +Scala 3 enums with case classes types +Compile-time configuration of discriminator encoding and mapping of type names + +## Challenge + +## Recap \ No newline at end of file diff --git a/docs/tutorials/5-use-derives-keyword.md b/docs/tutorials/5-use-derives-keyword.md new file mode 100644 index 000000000..f48c6faec --- /dev/null +++ b/docs/tutorials/5-use-derives-keyword.md @@ -0,0 +1,8 @@ +Use inlined CodecMakerConfiguration +Use static annotations and their priority over CodecMakerConfiguration +Use derives on top-level data structures +Use custom codec injections using implicits or givens + +## Challenge + +## Recap \ No newline at end of file diff --git a/docs/tutorials/6-parse-streaming-JSON-values-and-huge-JSON-arrays.md b/docs/tutorials/6-parse-streaming-JSON-values-and-huge-JSON-arrays.md new file mode 100644 index 000000000..6c49686f9 --- /dev/null +++ b/docs/tutorials/6-parse-streaming-JSON-values-and-huge-JSON-arrays.md @@ -0,0 +1,7 @@ +Generate ~10GB JSON of streamed values +Parse and handle parsed data without loading the whole input in memory +Do the same with huge JSON array + +## Challenge + +## Recap \ No newline at end of file diff --git a/docs/tutorials/7-write-custom-value-and-key-codecs.md b/docs/tutorials/7-write-custom-value-and-key-codecs.md new file mode 100644 index 000000000..c9f95462b --- /dev/null +++ b/docs/tutorials/7-write-custom-value-and-key-codecs.md @@ -0,0 +1,11 @@ +Examples: +1. Custom value codec for Base64 +2. Custom value codec JS compatible `Long` values +3. Custom value codec for type constructors +4. Custom value codec for union types +5. Custom key codec for enums +6. Custom key codec for tuples + +## Challenge + +## Recap diff --git a/release.sbt b/release.sbt index 2d7b901d7..27b9a1e73 100644 --- a/release.sbt +++ b/release.sbt @@ -22,6 +22,11 @@ lazy val updateVersionInReadmeAndExamples: ReleaseStep = { st: State => s"git add $path" !! st.log } + Seq( + "README.md", + "docs/tutorials/1-getting-started.md", + "docs/tutorials/1-getting-started.scala", + ).foreach(updateFile) updateFile("README.md") (1 to 3).foreach(n => updateFile(s"jsoniter-scala-examples/example0$n.sc"))