A native implementation of the Delta protocol in go. This library started as a port of delta-rs.
This project is in alpha and the API is under development.
Current implementation is designed for highly concurrent writes; reads are not yet supported.
Our use case is to ingest data, write it to the Delta table folder as a Parquet file using a Parquet go library, and then add the Parquet file to the Delta table using delta-go.
| Storage | Status | Comment |
|---|---|---|
| Local |  |
|
| S3 - AWS | |
Requires lock for concurrent writes |
| Operation | Status | Description |
|---|---|---|
| Create | |
Create a new table |
| Append | |
Append data in a Parquet file to a table |
| Checkpoint | |
Create a V1 checkpoint for a table. Note that the optional log cleanup has not been fully tested. |
| Writer Version | Requirement | Status |
|---|---|---|
| Version 2 | Append Only Tables | |
| Version 2 | Column Invariants | |
| Version 3 | Enforce delta.checkpoint.writeStatsAsJson |
|
| Version 3 | Enforce delta.checkpoint.writeStatsAsStruct |
|
| Version 3 | CHECK constraints | |
| Version 4 | Change Data Feed | |
| Version 4 | Generated Columns | |
| Version 5 | Column Mapping | |
| Version 6 | Identity Columns | |
| Version 7 | Table Features |
| Reader Version | Requirement | Status |
|---|---|---|
| Version 2 | Column Mapping | |
| Version 3 | Table Features (requires reader V7) |
Create a table in S3. This table is configured to use DynamoDB LogStore locking to enable multi-cluster S3 support.
store, err := s3store.New(s3Client, baseURI)
logStore, err := dynamodblogstore.New(dynamodblogstore.Options{Client: dynamoDBClient, TableName: deltaLogStoreTableName})
table := delta.NewTableWithLogStore(store, nillock.New(), logStore)
metadata := delta.NewTableMetaData("Test Table", "test description", new(delta.Format).Default(), schema, []string{}, make(map[string]string))
err := table.Create(*metadata, new(delta.Protocol).Default(), delta.CommitInfo{}, []delta.Add{})Append data to the table. The data is in a parquet file located at parquetRelativePath; the path is relative to the baseURI.
add, _, err := delta.NewAdd(store, storage.NewPath(parquetRelativePath), make(map[string]string))
transaction := table.CreateTransaction(delta.NewTransactionOptions())
transaction.AddActions([]deltalib.Action{add})
operation := delta.Write{Mode: delta.Append}
appMetaData := make(map[string]any)
appMetaData["isBlindAppend"] = true
transaction.SetAppMetadata(appMetaData)
transaction.SetOperation(operation)
v, err := transaction.CommitLogStore()There are also some simple examples available in the examples/ folder.
If delta-go and other client(s) are being used to write to the same Delta table on S3, then it is important to configure all clients to use multi-cluster LogStore to avoid write conflicts.