Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

536 add compute resource usage to processingjson #1063

Merged
merged 31 commits into from
Oct 3, 2024

Conversation

dbirman
Copy link
Member

@dbirman dbirman commented Sep 13, 2024

Adds a new class ResourceUsage that can optionally be attached to a data process. Actual usage is stored in ResourceTimestamped objects which track percent usage along with timestamps. See examples:

timestamped_cpu_usage = [
    ResourceTimestamped(timestamp=date(2024, 9, 13), usage=0.75),
    ResourceTimestamped(timestamp=date(2024, 9, 13), usage=0.80),
]

timestamped_gpu_usage = [
    ResourceTimestamped(timestamp=date(2024, 9, 13), usage=0.6),
    ResourceTimestamped(timestamp=date(2024, 9, 13), usage=0.66),
]

timestamped_ram_usage = [
    ResourceTimestamped(timestamp=date(2024, 9, 13), usage=0.7),
    ResourceTimestamped(timestamp=date(2024, 9, 13), usage=0.725),
]

timestamped_file_io_usage = [
    ResourceTimestamped(timestamp=date(2024, 9, 13), usage=5.5),
    ResourceTimestamped(timestamp=date(2024, 9, 13), usage=6.0),
]

DataProcess(
    name=ProcessName.IMAGE_TILE_FUSING,
    software_version="0.0.1",
    start_date_time=t,
    end_date_time=t,
    input_location="/path/to/inputs",
    output_location="/path/to/outputs",
    code_version="0.1",
    code_url="https://github.com/abcd",
    parameters={"size": 7},
    resources=ResourceUsage(
        os=OperatingSystem.UBUNTU_20_04,
        architecture=CPUArchitecture.X86_64,
        cpu="Intel Core i7",
        cpu_cores=8,
        gpu="NVIDIA GeForce RTX 3080",
        memory=MemoryValue(value=32.0, unit=MemoryUnit.GB),
        ram=MemoryValue(value=16.0, unit=MemoryUnit.GB),
        cpu_usage=timestamped_cpu_usage,
        gpu_usage=timestamped_gpu_usage,
        ram_usage=timestamped_ram_usage,
        file_io_usage=timestamped_file_io_usage,
        file_io_unit=MemoryUnit.MB,
    ),
)

@dbirman dbirman linked an issue Sep 13, 2024 that may be closed by this pull request
Copy link
Collaborator

@saskiad saskiad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

couple questions

Copy link
Collaborator

@saskiad saskiad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as other one - let's use the AwareDatetimeWithDefault. I clearly never pointed this out before... sorry about that.

examples/processing.py Outdated Show resolved Hide resolved
src/aind_data_schema/core/processing.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@saskiad saskiad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we wanted to change the value_with_unit thing, right? couple other small things as well

@saskiad saskiad self-requested a review October 3, 2024 21:28
@dbirman dbirman added this pull request to the merge queue Oct 3, 2024
Merged via the queue into dev with commit cb97a80 Oct 3, 2024
5 checks passed
@dbirman dbirman deleted the 536-add-compute-resource-usage-to-processingjson branch October 3, 2024 21:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add compute resource usage to processing.json
3 participants