-
Notifications
You must be signed in to change notification settings - Fork 37
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge remote-tracking branch 'origin/main' into new-token-ci-support-…
…den-prod # Conflicts: # .github/workflows/local_tests_den_dev.yaml # runhouse/rns/rns_client.py # runhouse/servers/cluster_servlet.py # runhouse/servers/http/auth.py # runhouse/servers/obj_store.py
- Loading branch information
Showing
36 changed files
with
678 additions
and
308 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
## LoRA Fine-Tuning Class with Example of Notebook Usage | ||
In this example, we define a Fine Tuner class (LoraFineTuner.py) in **regular Python** and launch remote GPU compute to do the fine-tuning. | ||
|
||
In particular, we show how you can start the fine tuning and interact with the fine-tuning class (a remote object) through regular Python or a Notebook. Runhouse lets you work *locally* with *remote objects* defined by regular code and edited locally, compared to tooling like hosted notebooks which let you *work locally while SSH'ed into a remote setting.* This offers a few distinct advantages: | ||
* **Real compute and real data:** ML Engineers and data scientists do not need to launch projects on toy compute offered in a research environment. | ||
* **Real code:** Rather than working on Notebooks (because they have to), your team is writing code and developing locally just like a normal software team. The only difference is dispatching the work for remote computation since the local machine doesn't have the right hardware. | ||
* **Fast research to production:** The work done while writing and testing the class is essentially enough to bring the work to production as well. There is no costly rebuilding of the same code a second time to work in a Pipeline. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
40 changes: 40 additions & 0 deletions
40
examples/torch-training/airflow-multicloud/DataProcessing.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
import os | ||
|
||
import boto3 | ||
|
||
|
||
# Download data from S3 | ||
def download_folder_from_s3(bucket_name, s3_folder_prefix, local_folder_path): | ||
s3 = boto3.client("s3") | ||
|
||
paginator = s3.get_paginator("list_objects_v2") | ||
for page in paginator.paginate(Bucket=bucket_name, Prefix=s3_folder_prefix): | ||
if "Contents" in page: | ||
for obj in page["Contents"]: | ||
s3_key = obj["Key"] | ||
relative_path = os.path.relpath(s3_key, s3_folder_prefix) | ||
local_path = os.path.join(local_folder_path, relative_path) | ||
|
||
os.makedirs(os.path.dirname(local_path), exist_ok=True) | ||
s3.download_file(bucket_name, s3_key, local_path) | ||
print(f"Downloaded {s3_key} to {local_path}") | ||
|
||
|
||
# download_folder_from_s3('rh-demo-external', 'your/s3/folder/prefix', '/path/to/local/folder', 'your-access-key-id', 'your-secret-access-key') | ||
|
||
|
||
# Upload data to S3 bucket | ||
def upload_folder_to_s3(local_folder_path, bucket_name, s3_folder_prefix): | ||
s3 = boto3.client("s3") | ||
|
||
for root, dirs, files in os.walk(local_folder_path): | ||
for file in files: | ||
local_path = os.path.join(root, file) | ||
relative_path = os.path.relpath(local_path, local_folder_path) | ||
s3_path = os.path.join(s3_folder_prefix, relative_path) | ||
|
||
s3.upload_file(local_path, bucket_name, s3_path) | ||
print(f"Uploaded {local_path} to s3://{bucket_name}/{s3_path}") | ||
|
||
|
||
# upload_folder_to_s3('/path/to/local/folder', 'rh-demo-external', 'your/s3/folder/prefix', 'your-access-key-id', 'your-secret-access-key') |
Oops, something went wrong.