github-as-s3

This post is an educational exploration into the S3 protocol and a practical guide on how one might implement it using Git as the underlying storage mechanism. Please be mindful of the terms of service and code of conduct for any Git hosting provider you choose to use with these concepts; I am not responsible for any actions taken against your accounts.

This project contains the source code for the Github+S3 proxy.

The intended use is to proxy object storage onto Git itself. You can read more about this here

The following routes are implemented

• PutObject
• GetObject
• HeadObject
• DeleteObject
• ListObjectsV2
• CreateBucket
• DeleteBucket
• ListBuckets

Getting started

Run it with Go

cp .env.example .env
go run ./cmd/cli/

The environment values require the following:

# required - replace with your own username and token
GITHUB_TOKEN=
GITHUB_OWNER=ktunprasert

# optional
# hosting related
GHS3_PORT=
GHS3_ADDRESS=
# committer's information
# these are the defaults defined in application.go
GIT_USERNAME=GHS3
GIT_EMAIL=bot@ghs3.com

The token requires 2 permissions: repo, delete_repo

Run it with Docker

docker build . -t ghs3
docker run --rm -t ghs3 --env "GITHUB_OWNER=ktunprasert" --env "GITHUB_TOKEN=$TOKEN"

Run it with docker-compose

# configure your compose file with proper environment values
docker-compose up -d

Development

air

Hot reloading with air is configured

Compatible & Tested applications

Tool	Tested
rclone	cp, sync, delete, mkdir, purge, ls, deletefile, touch, lsd, rmdir	✅
aws s3	mb, cp, ls, rm	✅
pocketbase	creating and restoring back up + deleting files	✅
pocketbase	using as file storage	❓

FAQ

How to set up my rclone to read to it?

[ghs3]
type = s3
provider = Other
endpoint = http://localhost:8080
list_version = 2

This was all I needed to start moving files

rclone mkdir ghs3:/test-repo/
rclone copy hello-world.txt ghs3:/test-repo/