DEVELOPMENT.md

Development Guide

Table of Contents

• Development Guide
  • Prerequisites
  • Checkout source code
  • How code is organized
  • See what tasks are available
  • Generating Intellij's Run Configurations
  • Updating FlatBuffers Java entities
  • Updating Web3 Contract Wrappers
  • Processing using Postgres
    • Performing Migrations
    • Updating Jooq Entities
  • Processing into Kafka
  • How to update tests
  • Run tests
  • Formatting source

Prerequisites

1. Java 11 or higher (AdoptOpenJDK, Zulu Community or OpenJDK)
2. Docker
3. Docker Compose
4. direnv
5. (Optional but highly recommendable) IntelliJ IDEA Community or Ultimate

Code organization

The codebase can be broken down as follows:

├── buildSrc          # Custom gradle tasks that aids in development.
├── docker            # Custom Dockerfiles
│   ├── exflo         #   - Base Dockerfile to generate docker releases of Exflo  
│   └── flatbuffers   #   - Base Dockerfile to generate flatbuffer specific images
├── domain            # Shared domain entities across different modules (mainly related to flatbuffers)
├── gradle            # Configuration of main gradle dependencies
├── ingestion         # Core of Exflo
│   ├── base          #   - Common logic to all plugins
│   ├── kafka         #   - Specific logic related to Kafka implementation
│   └── postgres      #   - Specific logic related to Postgres implementation
├── plugin            # Common entry point plugin definitions for Besu
├── postman           # JSON RPC Postman collection to interact with the node
├── testutil          # Test utilities and helpers related to testing Exflo
└── truffle           # Some tests require the interaction with the Blockchain, here are located Solidity tests

See what tasks are available

To see all of the gradle tasks that are available:

$ cd exflo
$ ./gradlew tasks

Generating Intellij's Run Configurations

If you plan to use the IDE shortcuts (which we strongly recommend), after cloning the repository just issue the following on the terminal:

$ ./gradlew generateIntellijRunConfigs

This gradle task will auto create required XMLs that are stored inside .idea/runConfigurations folder. If you want to customize options, you can edit the YAML run-configs.yaml file found on intellij folder.

Updating FlatBuffers Java entities

If you want to update the flatbuffers entities found on domain submodule:

$ ./gradlew :domain:compileFlatBuffers

It will place updated entities on domain/src/generated/java folder.

Updating Web3 Contract Wrappers

If you want to update the flatbuffers entities, just write the following:

$ ./gradlew generateContractWrappers

Processing using Postgres

Type the following on your terminal in order to start docker with postgres:

$ docker-compose -f docker-compose.exflo-postgres.yml up -d

Alternatively you can do the same by running the Run configuration named DOCKER | Postgres.

After a couple of seconds postgres will be running on your machine, to verify, just navigate to http://localhost:8082/ on your browser and pgweb should welcome you with an empty database.

The final step is to execute the Run configuration named ROPSTEN | Postgres | Run for Ropsten network or DEV | Postgres | Run for a private dev network.

Performing Migrations

By default, ExfloPostgresPlugin will try to apply migrations automatically.

If for whatever reason you want to run the migrations yourself you can use the following:

$ ./gradlew flywayMigrate

To drop all data and structure in the DB you can use:

$ ./gradlew flywayClean

We have added two Run configurations as shortcuts as well:

• GRADLE | Postgres > FlywayMigrate
• GRADLE | Postgres > FlywayClean

Updating Jooq Entities

If you have added db migrations and applied them, you will need to regenerate the jooq entities for interacting with the db. In order to do so, first ensure that:

1. Postgres is running.
2. Migrations have been applied correctly.

In the terminal:

$ ./gradlew jooq-codegen-primary

We have added another shortcut named GRADLE | Postgres > JooqCodeGen.

Processing into Kafka

The steps are performed in the same way as have been describing before for Postgres but with small variations:

$ docker-compose -f docker-compose.exflo-kafka.yml up -d

Alternatively you can do the same by running the Run configuration named DOCKER | Kafka.

After a couple of seconds, kafka will be running on your machine, to verify, you can check the logs using the following command:

$ docker-compose -f docker-compose.exflo-kafka.yml logs

The final step is to execute the Run configuration named ROPSTEN | Kafka | Run.

How to update tests

We are using KotlinTest for verifying that everything works as intended.

Inside testutil module you will find a series of helper classes. Currently the majority of tests are performed on ingestion/base module (although we will cover postgres and kafka modules as well).

We are using an exported test.blocks blockchain that the testing framework ingests in order to perform tests. That way we can combine some tests that involves the usage of Solidity code (like testing for Tokens). If you want to update some tests on truffle/:

$ cd truffle/
$ bin/capture-test-blocks

This will auto-generate the required test.blocks and test-report.json files.

Run tests

All the unit tests are run as part of the build, but can be explicitly triggered with:

$ ./gradlew test

Formatting source

To format the code you can run:

$ ./gradlew ktlintFormat