feat: Multi-database and AWS S3 storage support #100
Conversation
- Replace POSTGRES_* env vars with generic DB_* naming convention - Add support for MySQL alongside PostgreSQL and SQLite - Add DB_TYPE environment variable to specify database engine - Configure optional dependencies for pg and mysql2 drivers - Maintain backward compatibility with SQLite as fallback
- Implement StorageManager with automatic S3/local detection - Support S3 storage for assets, collections, and storage.json - Add CloudFront integration and URL handling fixes - Maintain full backward compatibility with local storage
…Move storage files to dedicated directory, rename S3Storage to AwsS3Storage, remove redundant files, integrate CloudStorage into StorageManager, fix built-in collections asset processing, unify S3/local logic, improve collection listing format consistency, fix URL issues, add detailed logging
- Add STORAGE_TYPE environment variable for explicit storage configuration - Support local file system and AWS S3 storage with configurable switching - Add multi-database support with PostgreSQL, MySQL, and SQLite fallback - Update clean-world.mjs script to support all database types - Improve environment variable documentation with clear configuration sections - Add proper validation for required S3 configuration when using AWS storage - Maintain backward compatibility with existing configurations
.env.example
Outdated
| S3_ASSETS_PREFIX= | ||
| S3_COLLECTIONS_PREFIX= | ||
| S3_STORAGE_PREFIX= |
There was a problem hiding this comment.
These changes would default to postgres for new people attempting to quickly spin up new worlds and I think we should continue to stick to sqlite as the default, so that people don't have to spend a whole lot of time launching a postgres docker container etc.
Additionally these are a lot of new environment variables that (might) be simplified by just using URI's like this:
# by default set to empty which uses sqlite in the world folder
DB_TYPE=
DB_URL=
# optionally configured to connect to a remote postgres database
DB_TYPE=pg
DB_URL=https://username:password@localhost:5432/database?schema=public
I don't think many people use mysql so i'd be happier to just not support that right now.
There was a problem hiding this comment.
I find your suggestion fine and I've updated both the code and the .env.example file to simplify the database setup as you recommended. Now, only two environment variables are used:
-
DB_TYPE (leave empty for SQLite, or set to pg for Postgres)
-
DB_URL (leave empty for SQLite, or provide a Postgres connection URI)
MySQL support and all the old related variables have been removed. This makes the default experience much simpler for new users, who can now spin up worlds with SQLite out of the box.
scripts/clean-world-s3.mjs
Outdated
| * - For SQLite: Uses local ./[world]/db.sqlite file | ||
| * | ||
| * Storage configuration: | ||
| * - STORAGE_TYPE must be set to 'aws' |
There was a problem hiding this comment.
S3 has become a bit of a standard, for example CloudFlare R2 can actually use S3-specified libraries for managing files on R2. So perhaps 'aws' should just be called 's3'?
There was a problem hiding this comment.
You're right!
I've updated the codebase to use s3 instead of aws for the storage type and environment variables. This makes the storage integration more generic and compatible with any S3-compatible provider (such as CloudFlare R2, DigitalOcean Spaces, etc.), not just AWS S3.
All relevant code, environment variables, and documentation have been updated accordingly.
scripts/clean-world-s3.mjs
Outdated
|
|
||
| // Close database connection before exiting | ||
| await db.destroy() | ||
| process.exit() |
There was a problem hiding this comment.
Maybe this could be done in the future, but maintaing two separate world clean scripts might be better abstracted so that there are functions like deleteFileS3() and deleteFileLocal() so that any changes to the general logic in a shared world-clean script would automatically maintain both.
There was a problem hiding this comment.
I've refactored the clean-world scripts to use shared utilities with functions like deleteFileS3() and deleteFileLocal().
Now both storage types stay in sync automatically when we update the cleanup logic - no more maintaining two separate scripts! The code is much cleaner and easier to maintain.
The scripts are organized in a nice modular structure (scripts/clean-world/) but still work exactly the same way with npm run world:clean.
Thanks for the feedback - this is definitely a much better approach!
src/server/storage/AwsS3Storage.js
Outdated
There was a problem hiding this comment.
Again highly likely that this will work with Cloudflare R2 as its S3 compatible, so we could just call this S3Storage.js etc and drop the AWS naming.
|
Is it possible to rebase this branch so that it doesn't have a bunch of dev commits merged back on top of it? This helps a lot to keep git history clean |
…to sqlite, remove MySQL support and old env vars. Update .env.example accordingly.
… for S3-compatible storage support (CloudFlare R2, DigitalOcean Spaces, etc.)
… and modular structure - Create clean-world/ directory with modular scripts - Add clean-world-utils.mjs with shared functions (deleteFileLocal, deleteFileS3, etc.) - Refactor clean-world-local.mjs and clean-world-s3.mjs to use shared utilities - Update main clean-world.mjs to dynamically import appropriate script - Add README.md with documentation for the new structure - Fix path resolution issues for world directories - Maintain backward compatibility with npm run world:clean This addresses the reviewer's suggestion to avoid code duplication and ensure both storage types stay in sync when cleanup logic changes.
…Move storage files to dedicated directory, rename S3Storage to AwsS3Storage, remove redundant files, integrate CloudStorage into StorageManager, fix built-in collections asset processing, unify S3/local logic, improve collection listing format consistency, fix URL issues, add detailed logging
- Replace POSTGRES_* env vars with generic DB_* naming convention - Add support for MySQL alongside PostgreSQL and SQLite - Add DB_TYPE environment variable to specify database engine - Configure optional dependencies for pg and mysql2 drivers - Maintain backward compatibility with SQLite as fallback
…to sqlite, remove MySQL support and old env vars. Update .env.example accordingly.
… for S3-compatible storage support (CloudFlare R2, DigitalOcean Spaces, etc.)
… and modular structure - Create clean-world/ directory with modular scripts - Add clean-world-utils.mjs with shared functions (deleteFileLocal, deleteFileS3, etc.) - Refactor clean-world-local.mjs and clean-world-s3.mjs to use shared utilities - Update main clean-world.mjs to dynamically import appropriate script - Add README.md with documentation for the new structure - Fix path resolution issues for world directories - Maintain backward compatibility with npm run world:clean This addresses the reviewer's suggestion to avoid code duplication and ensure both storage types stay in sync when cleanup logic changes.
…mes/numinia-hyperfy2 into feat/full-storage-and-db-support # Conflicts: # .env.example # scripts/clean-world.mjs # src/server/storage/StorageManager.js
…alizing the storage manager
|
I did a rebase of the branch with dev and I think I didn't mess up anything 🤞! |
Description
This PR introduces comprehensive database and storage flexibility to the Hyperfy server infrastructure. The changes add multi-database support (PostgreSQL, MySQL, SQLite) and AWS S3 storage capabilities, improving deployment flexibility and scalability options.
Components affected:
Key improvements:
DB_TYPEconfiguration supporting PostgreSQL, MySQL, and SQLiteSTORAGE_TYPEconfiguration for local file system or AWS S3 storageFixes # (issue)
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Please describe your tests:
Test Configuration:
Checklist: