Skip to content

Commit

Permalink
Added fuctionality for new "Release" feature in FIM API
Browse files Browse the repository at this point in the history 
New "Release" feature added to the FIM API. This feature will allow for automated FIM, CatFIM, and relevant metrics to be generated when a new FIM Version is released. See #373 for more detailed steps that take place in this feature.

Additions
- Added new window to the UI in api/frontend/gui/templates/index.html.
- Added new job type to api/node/connector/connector.py to allow these release jobs to run.
- Added additional logic in api/node/updater/updater.py to run the new eval and CatFIM scripts used in the release feature.

Changes
- Updated api/frontend/output_handler/output_handler.py to allow for copying more broad ranges of file paths instead of only the /data/outputs directory.

Resolves Issues #264 #278 #307 and #373
  • Loading branch information
@nickchadwick-noaa
nickchadwick-noaa authored May 7, 2021
1 parent 22ba0df commit b5da4b4
Show file tree
Hide file tree
Showing 5 changed files with 486 additions and 154 deletions.
13 changes: 13 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,18 @@
All notable changes to this project will be documented in this file.
We follow the [Semantic Versioning 2.0.0](http://semver.org/) format.
<br/><br/>
## v3.0.16.0 - 2021-05-07 - [PR #378](https://github.com/NOAA-OWP/cahaba/pull/378)

New "Release" feature added to the FIM API. This feature will allow for automated FIM, CatFIM, and relevant metrics to be generated when a new FIM Version is released. See [#373](https://github.com/NOAA-OWP/cahaba/issues/373) for more detailed steps that take place in this feature.

## Additions
- Added new window to the UI in `api/frontend/gui/templates/index.html`.
- Added new job type to `api/node/connector/connector.py` to allow these release jobs to run.
- Added additional logic in `api/node/updater/updater.py` to run the new eval and CatFIM scripts used in the release feature.

## Changes
- Updated `api/frontend/output_handler/output_handler.py` to allow for copying more broad ranges of file paths instead of only the `/data/outputs` directory.

<br/><br/>
## v3.0.15.10 - 2021-05-06 - [PR #375](https://github.com/NOAA-OWP/cahaba/pull/375)

Expand Down
158 changes: 124 additions & 34 deletions api/frontend/gui/templates/index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@

color: black;
background-color: white;
font-size: 0.9rem;

transition: all 0.1s linear;

Expand Down Expand Up @@ -137,6 +138,27 @@
background-color: #093568;
}

#release-button {
display: flex;
width: 100%;
height: 100%;

align-items: center;
justify-content: center;

font-size: 1.2rem;
color: white;
background-color: #28be99;
cursor: pointer;

transition: all 0.5s linear;
pointer-events: inherit;
}

#release-button:hover, #release-button.active {
background-color: #09685b;
}

.form-field {
display: flex;
width: 0;
Expand Down Expand Up @@ -224,15 +246,40 @@
<!-- <h1>FIM Run API</h1> -->
<div id="request-container">
<div id="request-tabs-container">
<div class="request-tab" onclick="selectForm(event, 'release')">Release</div>
<div class="request-tab active" onclick="selectForm(event, 'fim-run')">FIM Run</div>
<div class="request-tab" onclick="selectForm(event, 'calibration')">Calibration</div>
<div class="request-tab" onclick="selectForm(event, 'preprocessing')">Pre-processing</div>
</div>
<div id="release" class="request-form" style="display: None;">
<div class="form-content">
<h3 style="margin: 0;">FIM Version</h3>
<div class="container-horizontal" style="width: 10rem;">
<span>FIM_3_</span>
<input type="text" id="release-job-version-major" class="form-field" placeholder="0"></input>
<span>_</span>
<input type="text" id="release-job-version-minor" class="form-field" placeholder="0"></input>
<span>_</span>
<input type="text" id="release-job-version-patch" class="form-field" placeholder="0"></input>
</div>
<h3 style="margin: 0;">Previous Major FIM Version</h3>
<div class="container-horizontal" style="width: 10rem;">
<span>FIM_3_</span>
<input type="text" id="prev-release-job-version-major" class="form-field" placeholder="0"></input>
<span>_</span>
<input type="text" id="prev-release-job-version-minor" class="form-field" placeholder="0"></input>
<span>_</span>
<input type="text" id="prev-release-job-version-patch" class="form-field" placeholder="0"></input>
</div>
<div class="error-list" id="release-errors"></div>
</div>
<div class="start-button not-connected" onclick="submitRequest('release')">Not Connected</div>
</div>
<div id="fim-run" class="request-form">
<div class="form-content">
<h3 style="margin: 0;">Basic</h3>
<div class="container-horizontal">
<input type="text" id="job-name" class="form-field" placeholder="Job Name"></input>
<input type="text" id="fim-run-job-name" class="form-field" placeholder="Job Name"></input>
<select id="presets-list" class="form-field" onchange="presetListSelected(event)"></select>
<input type="text" id="hucs" class="form-field" placeholder="HUC(s)" style="display: none;"></input>
<input type="text" id="git-branch" class="form-field" placeholder="Git Branch"></input>
Expand Down Expand Up @@ -271,7 +318,7 @@ <h3 style="margin: 0;">Extent</h3>

</div>
</div>
<div class="start-button not-connected" onclick="submitRequest('fim-run')">Not Connected</div>
<div class="start-button not-connected" onclick="submitRequest('fim_run')">Not Connected</div>
</div>
<div id="calibration" class="request-form" style="display: None;">
Calibration stuff goes here
Expand Down Expand Up @@ -370,7 +417,7 @@ <h3 style="margin: 0;">Extent</h3>
socket.on('job_added', job_type => {
document.querySelector('#request-container').classList.remove('loading')
if (job_type === 'fim_run') {
document.querySelector('#job-name').value = ''
document.querySelector('#fim-run-job-name').value = ''
document.querySelector('#hucs').value = ''
document.querySelector('#git-branch').value = ''
document.querySelector("input[name='dev-run']").checked = false
Expand All @@ -380,46 +427,89 @@ <h3 style="margin: 0;">Extent</h3>
document.querySelector("input[name='extent'][value='FR']").checked = true
document.querySelector("input[name='extent'][value='MS']").checked = false
document.querySelector('#fim-run-errors').innerHTML = ''
} else if (job_type === 'release') {
console.log('release job has been added')
document.querySelector('#release-job-version-major').value = ''
document.querySelector('#release-job-version-minor').value = ''
document.querySelector('#release-job-version-patch').value = ''
document.querySelector('#prev-release-job-version-major').value = ''
document.querySelector('#prev-release-job-version-minor').value = ''
document.querySelector('#prev-release-job-version-patch').value = ''
document.querySelector('#release-errors').innerHTML = ''
}
})
socket.on('job_canceled', () => {
console.log("Job has been cancelled")
})
})

// Fim_run related code
const submitRequest = requestName => {
let validation_errors = []
const job_name = document.querySelector('#job-name').value
const preset = document.querySelector('#presets-list').value
const hucs = document.querySelector('#hucs').value
const git_branch = document.querySelector('#git-branch').value
const dev_run = document.querySelector("input[name='dev-run']").checked
const viz_run = document.querySelector("input[name='viz-run']").checked
const configuration = Array.from(document.querySelectorAll("input[name='configuration']")).filter(c => c.checked)[0].value
const extents = Array.from(document.querySelectorAll("input[name='extent']")).filter(c => c.checked).map(c => c.value)

if (job_name === '') validation_errors.push('Job Name Cannot Be Empty')
if (preset === 'custom' && hucs === '') validation_errors.push('Huc(s) Cannot Be Empty')
if (git_branch === '') validation_errors.push('Git Branch Cannot Be Empty')

if (validation_errors.length > 0){
document.querySelector('#fim-run-errors').innerHTML = validation_errors.map(e => `<span>${e}</span>`).join('')
return
}
if (requestName === 'fim_run'){
// Fim_run related code
let validation_errors = []
const job_name = document.querySelector('#fim-run-job-name').value
const preset = document.querySelector('#presets-list').value
const hucs = document.querySelector('#hucs').value
const git_branch = document.querySelector('#git-branch').value
const dev_run = document.querySelector("input[name='dev-run']").checked
const viz_run = document.querySelector("input[name='viz-run']").checked
const configuration = Array.from(document.querySelectorAll("input[name='configuration']")).filter(c => c.checked)[0].value
const extents = Array.from(document.querySelectorAll("input[name='extent']")).filter(c => c.checked).map(c => c.value)

if (job_name === '') validation_errors.push('Job Name Cannot Be Empty')
if (preset === 'custom' && hucs === '') validation_errors.push('Huc(s) Cannot Be Empty')
if (git_branch === '') validation_errors.push('Git Branch Cannot Be Empty')

if (validation_errors.length > 0){
document.querySelector('#fim-run-errors').innerHTML = validation_errors.map(e => `<span>${e}</span>`).join('')
return
}

document.querySelector('#request-container').classList.add('loading')

socket.emit('new_job', {
job_name,
preset,
hucs,
git_branch,
dev_run,
viz_run,
configuration,
extents
})
document.querySelector('#request-container').classList.add('loading')

socket.emit('new_job', {
job_type: 'fim_run',
job_name,
preset,
hucs,
git_branch,
dev_run,
viz_run,
configuration,
extents
})
} else if (requestName === 'release'){
// Release related code
let validation_errors = []
const job_version_major = document.querySelector('#release-job-version-major').value
const job_version_minor = document.querySelector('#release-job-version-minor').value
const job_version_patch = document.querySelector('#release-job-version-patch').value
const prev_job_version_major = document.querySelector('#prev-release-job-version-major').value
const prev_job_version_minor = document.querySelector('#prev-release-job-version-minor').value
const prev_job_version_patch = document.querySelector('#prev-release-job-version-patch').value

if (job_version_major === '' || job_version_minor === '' || job_version_patch === '')
validation_errors.push('Fim Version must be completely filled out')
if (prev_job_version_major === '' || prev_job_version_minor === '' || prev_job_version_patch === '')
validation_errors.push('Previous Major Fim Version must be completely filled out')

if (validation_errors.length > 0){
document.querySelector('#release-errors').innerHTML = validation_errors.map(e => `<span>${e}</span>`).join('')
return
}

document.querySelector('#request-container').classList.add('loading')

socket.emit('new_job', {
job_type: 'release',
job_version_major,
job_version_minor,
job_version_patch,
prev_job_version_major,
prev_job_version_minor,
prev_job_version_patch
})
}
}

const disableDevRun = e => {
Expand Down
6 changes: 2 additions & 4 deletions api/frontend/output_handler/output_handler.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
SOCKET_URL = os.environ.get('SOCKET_URL')

def handle_outputs(data):
nice_name = data['nice_name']
job_name = data['job_name']
directory_path = data['directory_path']
file_name = data['file_name']
Expand All @@ -18,16 +17,15 @@ def handle_outputs(data):
if chunk_index == 0:
mode = 'wb'
try:
os.makedirs(f"/data/outputs/{nice_name}/{directory_path}")
os.makedirs(directory_path)
except:
pass

# Write binary data to file
with open(f"/data/outputs/{nice_name}/{directory_path}/{file_name}", mode) as binary_file:
with open(f"{directory_path}/{file_name}", mode) as binary_file:
print(f"Writing chunk {chunk_index} for file {directory_path}/{file_name}")
binary_file.write(file_chunk)

# TODO: Write an emit that will trigger the next chunk to be sent.
sio.emit('output_handler_finished_file_chunk', {'job_name': job_name, 'file_path': f"{directory_path}/{file_name}"})

sio = socketio.Client()
Expand Down
Loading

0 comments on commit b5da4b4

Please sign in to comment.