How do I connect Trino to Iceberg Rest catalog?

[paulpaul1076]

Hello guys, could you please explain how to connect Trino to Iceberg Rest catalog?

Here's a properties file for it:

// /etc/catalog/iceberg_rest.properties
connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=http://ip:8181/ 
iceberg.rest-catalog.warehouse=s3a://obs-zdp-warehouse-stage-mz/
hive.s3.aws-access-key=redacted
hive.s3.aws-secret-key=redacted
hive.s3.endpoint=redacted
hive.s3.region=redacted
hive.s3.path-style-access=true
hive.s3.ssl.enabled=true

Is it possible to connect to both Hive and Rest catalog at the same time in Stackable Trino?

I'm guessing this example works only for the Hive metastore? https://docs.stackable.tech/home/stable/trino/usage-guide/catalogs/iceberg

[sbernauer]

Hi @paulpaul1076, the Iceberg REST catalog is not yet supported natively by Stackable.
Until then you need to enter some bogus Hive Metastore config (as it is mandatory), and then use configOverrides to set iceberg.catalog.type=rest and additional settings you need.
A non-tested example could look like

apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCatalog
metadata:
  name: iceberg
  labels:
    trino: simple-trino
spec:
  connector:
    iceberg:
      metastore:
        configMap: my-hive # This configmap needs to exist and contain the key `HIVE` with e.g. `thrift://hive-iceberg.default.svc.cluster.local:9083`
      s3:
        inline:
          host: test-minio
          port: 9000
          accessStyle: Path
          credentials:
            secretClass: minio-credentials
		  tls:
		    verification:
		      server:
		        caCert:
		          webPki: {}
  configOverrides:
    iceberg.catalog.type: rest
    iceberg.rest-catalog.uri: http://ip:8181/
    iceberg.rest-catalog.warehouse: s3a://obs-zdp-warehouse-stage-mz/

[sbernauer]

Hi @paulpaul1076 I gave this a try. We have a generic connector, which you can use to pull in arbitrary connectors which we don't support natively yet.
Until we support the REST catalog natively, you can use that as follows to configure it:

apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCatalog
metadata:
  name: iceberg
  labels:
    trino: trino
spec:
  connector:
    generic:
      connectorName: iceberg
      properties:
        iceberg.catalog.type:
          value: rest
        iceberg.rest-catalog.uri:
          value: http://ip:8181/
        iceberg.rest-catalog.warehouse:
          value: s3a://obs-zdp-warehouse-stage-mz/
        hive.s3.aws-access-key:
          valueFromSecret:
            name: minio-s3-credentials
            key: accessKey
        hive.s3.aws-secret-key:
          valueFromSecret:
            name: minio-s3-credentials
            key: secretKey
        hive.s3.region:
          value: us-east-1
        hive.s3.path-style-access:
          value: "true"
        hive.s3.ssl.enabled:
          value: "true"
---
apiVersion: v1
kind: Secret
metadata:
  name: minio-s3-credentials
stringData:
  accessKey: admin
  secretKey: adminadmin

Does this work for you?

[paulpaul1076]

@sbernauer thanks a lot! This worked like a charm.

Btw @sbernauer @lfrancke if you guys plan to add more catalogs to stackable Trino, besides Rest please consider Nessie also: https://github.com/projectnessie/nessie

It's a pretty big project, people definitely use it, it's got cool features, where you can basically work with iceberg tables as if they were git repositories: create branches, merge one branch into another, etc.

[paulpaul1076]

@lfrancke a few days later, I can say that this catalog is some kind of experimentation catalog after all. We ran it in prod for 4 hours and it crashed with some exceptions related to connecting to postgres. It turned out the the rest catalog which tabular (the company behind iceberg) talks about is some kind of catalog available only in their paid platform. And the only opensource iceberg catalog which is actively supported is nessie: https://github.com/projectnessie/nessie

That catalog https://github.com/tabular-io/iceberg-rest-image which I talked about is basically just a JdbcCatalog on top of which the github repo adds a rest server.

So, we're gonna experiment with nessie for now and I will tell you my thoughts.

[sbernauer]

I just wanted to mention that there is also Gravitino, which you could give a try.
It was announced on the Iceberg mailing list

[paulpaul1076]

@sbernauer hello, thanks! I know of Gravitino, I've seen it before. They are thinking of making it the default community catalog right now, I'm also subscribed to their mailing list.

However we decided to go with Nessie, because it seems like a more mature project and they also have a very helpful chat where you can ask questions and they help you tremendously.

I'll be keeping an eye out on gravitino. I think you guys should add support for Nessie and Gravitino.