[Feature/agent_framework] Deprovision API #271

dbwiddis · 2023-12-10T09:39:16Z

Description

Deprovisions resources created by a workflow.

Issues Resolved

Fixes #89
Fixes #32

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

dbwiddis · 2023-12-11T18:38:35Z

Current REST response (since we aren't undeploying model it blocks delete model and delete connector:

{
    "error": "Failed to deprovision some resources: [model_id gHcrWowBNatcpxJRFcCN, connector_id fncrWowBNatcpxJRE8BQ]."
}

Current output:

[2023-12-11T10:37:23,535][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Deprovisioning steps: root_agent_deprovision, sub_agent_deprovision, gpt-3.5-model_deprovision, openAI_connector_deprovision
[2023-12-11T10:37:23,536][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting root_agent_deprovision.
[2023-12-11T10:37:23,604][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Successful root_agent_deprovision for agent_id g3crWowBNatcpxJRLMBU
[2023-12-11T10:37:23,604][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Finished root_agent_deprovision.
[2023-12-11T10:37:23,605][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting sub_agent_deprovision.
[2023-12-11T10:37:23,663][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Finished sub_agent_deprovision.
[2023-12-11T10:37:23,663][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Successful sub_agent_deprovision for agent_id gncrWowBNatcpxJRK8Cd
[2023-12-11T10:37:23,682][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting gpt-3.5-model_deprovision.
[2023-12-11T10:37:23,707][ERROR][o.o.f.w.DeleteModelStep  ] [88665a43e3fa] Failed to delete model
[2023-12-11T10:37:23,708][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Failed gpt-3.5-model_deprovision for model_id gHcrWowBNatcpxJRFcCN: org.opensearch.flowframework.exception.FlowFrameworkException: Model cannot be deleted in deploying or deployed state. Try undeploy model first then delete
[2023-12-11T10:37:23,708][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting openAI_connector_deprovision.
[2023-12-11T10:37:23,719][ERROR][o.o.m.a.c.DeleteConnectorTransportAction] [88665a43e3fa] 1 models are still using this connector, please delete or update the models first!
[2023-12-11T10:37:23,720][ERROR][o.o.f.w.DeleteConnectorStep] [88665a43e3fa] Failed to delete connector
[2023-12-11T10:37:23,720][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Failed openAI_connector_deprovision for connector_id fncrWowBNatcpxJRE8BQ: org.opensearch.flowframework.exception.FlowFrameworkException: 1 models are still using this connector, please delete or update the models first: [gHcrWowBNatcpxJRFcCN]
[2023-12-11T10:37:23,721][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting gpt-3.5-model_deprovision.
[2023-12-11T10:37:23,723][ERROR][o.o.f.w.DeleteModelStep  ] [88665a43e3fa] Failed to delete model
[2023-12-11T10:37:23,724][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Failed gpt-3.5-model_deprovision for model_id gHcrWowBNatcpxJRFcCN: org.opensearch.flowframework.exception.FlowFrameworkException: Model cannot be deleted in deploying or deployed state. Try undeploy model first then delete
[2023-12-11T10:37:23,724][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting openAI_connector_deprovision.
[2023-12-11T10:37:23,726][ERROR][o.o.m.a.c.DeleteConnectorTransportAction] [88665a43e3fa] 1 models are still using this connector, please delete or update the models first!
[2023-12-11T10:37:23,727][ERROR][o.o.f.w.DeleteConnectorStep] [88665a43e3fa] Failed to delete connector
[2023-12-11T10:37:23,727][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Failed openAI_connector_deprovision for connector_id fncrWowBNatcpxJRE8BQ: org.opensearch.flowframework.exception.FlowFrameworkException: 1 models are still using this connector, please delete or update the models first: [gHcrWowBNatcpxJRFcCN]

dbwiddis · 2023-12-12T17:50:58Z

Successful deprovision log:

[2023-12-12T09:48:22,139][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Deprovisioning steps: root_agent_deprovision, sub_agent_deprovision, deployed-gpt-3.5-model_deprovision, gpt-3.5-model_deprovision, openAI_connector_deprovision
[2023-12-12T09:48:22,140][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting root_agent_deprovision.
[2023-12-12T09:48:22,180][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Successful root_agent_deprovision for agent_id 6UkkX4wBMUfZGiR3qNj7
[2023-12-12T09:48:22,180][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Finished root_agent_deprovision.
[2023-12-12T09:48:22,302][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting sub_agent_deprovision.
[2023-12-12T09:48:22,342][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Finished sub_agent_deprovision.
[2023-12-12T09:48:22,342][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Successful sub_agent_deprovision for agent_id 6EkkX4wBMUfZGiR3qNhx
[2023-12-12T09:48:22,447][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting deployed-gpt-3.5-model_deprovision.
[2023-12-12T09:48:22,447][INFO ][o.o.m.a.u.TransportUndeployModelsAction] [88665a43e3fa] Undeploying model: dhrubo
[2023-12-12T09:48:22,448][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Finished deployed-gpt-3.5-model_deprovision.
[2023-12-12T09:48:22,448][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Successful deployed-gpt-3.5-model_deprovision for model_id 5kkkX4wBMUfZGiR3ktik
[2023-12-12T09:48:22,552][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting gpt-3.5-model_deprovision.
[2023-12-12T09:48:22,595][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Successful gpt-3.5-model_deprovision for model_id 5kkkX4wBMUfZGiR3ktik
[2023-12-12T09:48:22,595][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Finished gpt-3.5-model_deprovision.
[2023-12-12T09:48:22,717][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting openAI_connector_deprovision.
[2023-12-12T09:48:22,719][ERROR][o.o.m.a.c.DeleteConnectorTransportAction] [88665a43e3fa] 1 models are still using this connector, please delete or update the models first!
[2023-12-12T09:48:22,719][ERROR][o.o.f.w.DeleteConnectorStep] [88665a43e3fa] Failed to delete connector
[2023-12-12T09:48:22,719][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Failed openAI_connector_deprovision for connector_id 5EkkX4wBMUfZGiR3kNjR: org.opensearch.flowframework.exception.FlowFrameworkException: 1 models are still using this connector, please delete or update the models first: [5kkkX4wBMUfZGiR3ktik]
[2023-12-12T09:48:23,723][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting openAI_connector_deprovision.
[2023-12-12T09:48:23,766][INFO ][o.o.m.a.c.DeleteConnectorTransportAction] [88665a43e3fa] Completed Delete Connector Request, connector id:5EkkX4wBMUfZGiR3kNjR deleted
[2023-12-12T09:48:23,766][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Finished openAI_connector_deprovision.
[2023-12-12T09:48:23,766][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Successful openAI_connector_deprovision for connector_id 5EkkX4wBMUfZGiR3kNjR
[2023-12-12T09:48:24,961][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] updated workflow 3UkjX4wBMUfZGiR319gi state to NOT_STARTED

Workflow stat was reset. Provisioned and deprovisioned multiple times. :)

dbwiddis · 2023-12-12T21:02:27Z

While the delete model API returns immediately with a true DeleteResponse, it takes a bit of time for the deleted model to register when checking its existence for delete connector. Intentionally using a shortened timeout shows what happens if a resource doesn't get deleted:

[2023-12-12T12:55:22,970][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Deprovisioning steps: root_agent_deprovision, sub_agent_deprovision, deployed-gpt-3.5-model_deprovision, gpt-3.5-model_deprovision, openAI_connector_deprovision
[2023-12-12T12:55:22,971][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting root_agent_deprovision.
[2023-12-12T12:55:23,035][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Finished root_agent_deprovision.
[2023-12-12T12:55:23,036][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Successful root_agent_deprovision for agent_id NA7PX4wBfVtHp98y7Kyw
[2023-12-12T12:55:23,056][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting sub_agent_deprovision.
[2023-12-12T12:55:23,095][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Successful sub_agent_deprovision for agent_id Mw7PX4wBfVtHp98y66z6
[2023-12-12T12:55:23,095][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Finished sub_agent_deprovision.
[2023-12-12T12:55:23,114][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting deployed-gpt-3.5-model_deprovision.
[2023-12-12T12:55:23,117][INFO ][o.o.m.a.u.TransportUndeployModelsAction] [88665a43e3fa] Undeploying model: dhrubo
[2023-12-12T12:55:23,122][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Finished deployed-gpt-3.5-model_deprovision.
[2023-12-12T12:55:23,122][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Successful deployed-gpt-3.5-model_deprovision for model_id MQ7PX4wBfVtHp98y1ax3
[2023-12-12T12:55:23,122][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting gpt-3.5-model_deprovision.
[2023-12-12T12:55:23,291][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Successful gpt-3.5-model_deprovision for model_id MQ7PX4wBfVtHp98y1ax3
[2023-12-12T12:55:23,291][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Finished gpt-3.5-model_deprovision.
[2023-12-12T12:55:23,291][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting openAI_connector_deprovision.
[2023-12-12T12:55:23,302][ERROR][o.o.m.a.c.DeleteConnectorTransportAction] [88665a43e3fa] 1 models are still using this connector, please delete or update the models first!
[2023-12-12T12:55:23,303][ERROR][o.o.f.w.DeleteConnectorStep] [88665a43e3fa] Failed to delete connector
[2023-12-12T12:55:23,303][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Failed openAI_connector_deprovision for connector_id Lw7PX4wBfVtHp98y06wV: org.opensearch.flowframework.exception.FlowFrameworkException: 1 models are still using this connector, please delete or update the models first: [MQ7PX4wBfVtHp98y1ax3]
[2023-12-12T12:55:23,305][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting openAI_connector_deprovision.
[2023-12-12T12:55:23,307][ERROR][o.o.m.a.c.DeleteConnectorTransportAction] [88665a43e3fa] 1 models are still using this connector, please delete or update the models first!
[2023-12-12T12:55:23,307][ERROR][o.o.f.w.DeleteConnectorStep] [88665a43e3fa] Failed to delete connector
[2023-12-12T12:55:23,307][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Failed openAI_connector_deprovision for connector_id Lw7PX4wBfVtHp98y06wV: org.opensearch.flowframework.exception.FlowFrameworkException: 1 models are still using this connector, please delete or update the models first: [MQ7PX4wBfVtHp98y1ax3]
[2023-12-12T12:55:23,308][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Resources remaining: [resources_Created [workflow_step_name= create_connector, workflow_step_id= openAI_connector, resource_id= Lw7PX4wBfVtHp98y06wV]]
[2023-12-12T12:55:23,460][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] updated workflow Lg7PX4wBfVtHp98yoqxN state to COMPLETED

Response from delete API

{
    "error": "Failed to deprovision some resources: [connector_id Lw7PX4wBfVtHp98y06wV]."
}

Status API shows this:

{
    "workflow_id": "Lg7PX4wBfVtHp98yoqxN",
    "state": "COMPLETED",
    "resources_created": [
        {
            "workflow_step_name": "create_connector",
            "workflow_step_id": "openAI_connector",
            "connector_id": "Lw7PX4wBfVtHp98y06wV"
        }
    ]
}

Deprovisioning again successfully deletes the resource:

[2023-12-12T12:55:56,188][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Deprovisioning steps: openAI_connector_deprovision
[2023-12-12T12:55:56,188][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Starting openAI_connector_deprovision.
[2023-12-12T12:55:56,230][INFO ][o.o.m.a.c.DeleteConnectorTransportAction] [88665a43e3fa] Completed Delete Connector Request, connector id:Lw7PX4wBfVtHp98y06wV deleted
[2023-12-12T12:55:56,231][INFO ][o.o.f.w.ProcessNode      ] [88665a43e3fa] Finished openAI_connector_deprovision.
[2023-12-12T12:55:56,231][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Successful openAI_connector_deprovision for connector_id Lw7PX4wBfVtHp98y06wV
[2023-12-12T12:55:56,251][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] Resources remaining: []
[2023-12-12T12:55:56,317][INFO ][o.o.f.t.DeprovisionWorkflowTransportAction] [88665a43e3fa] updated workflow Lg7PX4wBfVtHp98yoqxN state to NOT_STARTED

Status API:

{
    "workflow_id": "Lg7PX4wBfVtHp98yoqxN",
    "state": "NOT_STARTED"
}

joshpalis

Overall looks good to me, just a few questions for clarification

src/main/java/org/opensearch/flowframework/transport/DeprovisionWorkflowTransportAction.java

joshpalis

@dbwiddis just a heads up

src/main/java/org/opensearch/flowframework/transport/DeprovisionWorkflowTransportAction.java

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Deprovision REST and Transport Actions Signed-off-by: Daniel Widdis <widdis@gmail.com> * Fix errors you find actually running the code Signed-off-by: Daniel Widdis <widdis@gmail.com> * Add test for Rest deprovision action Signed-off-by: Daniel Widdis <widdis@gmail.com> * Initial copypaste of Deprovision Transport Action Test Signed-off-by: Daniel Widdis <widdis@gmail.com> * Add some delays to let deletions propagate, reset workflow state Signed-off-by: Daniel Widdis <widdis@gmail.com> * Improved deprovisioning results and status updates Signed-off-by: Daniel Widdis <widdis@gmail.com> * Fix bug in resource created parsing Signed-off-by: Daniel Widdis <widdis@gmail.com> * Completed test implementations Signed-off-by: Daniel Widdis <widdis@gmail.com> * Fixes after rebase Signed-off-by: Daniel Widdis <widdis@gmail.com> --------- Signed-off-by: Daniel Widdis <widdis@gmail.com>

dbwiddis force-pushed the deprovision branch 3 times, most recently from a6cdfcb to 34724d5 Compare December 11, 2023 04:10

dbwiddis mentioned this pull request Dec 11, 2023

[BUG] Deploying a model doesn't update the workflow state #272

Closed

dbwiddis force-pushed the deprovision branch from 6b6ba19 to 3c76dc9 Compare December 12, 2023 16:51

dbwiddis marked this pull request as ready for review December 12, 2023 21:17

dbwiddis requested review from owaiskazi19, joshpalis, ohltyler, amitgalitz and jackiehanyang as code owners December 12, 2023 21:17

joshpalis approved these changes Dec 13, 2023

View reviewed changes

src/main/java/org/opensearch/flowframework/transport/DeprovisionWorkflowTransportAction.java Show resolved Hide resolved

src/main/java/org/opensearch/flowframework/transport/DeprovisionWorkflowTransportAction.java Show resolved Hide resolved

joshpalis reviewed Dec 13, 2023

View reviewed changes

src/main/java/org/opensearch/flowframework/transport/DeprovisionWorkflowTransportAction.java Outdated Show resolved Hide resolved

src/main/java/org/opensearch/flowframework/transport/DeprovisionWorkflowTransportAction.java Outdated Show resolved Hide resolved

dbwiddis force-pushed the deprovision branch 2 times, most recently from a18a9c3 to 1cbe20c Compare December 13, 2023 22:17

amitgalitz reviewed Dec 13, 2023

View reviewed changes

src/main/java/org/opensearch/flowframework/transport/DeprovisionWorkflowTransportAction.java Show resolved Hide resolved

dbwiddis force-pushed the deprovision branch from 6704d48 to 0431876 Compare December 14, 2023 02:36

This was linked to issues Dec 14, 2023

[PROPOSAL] Uninstall / Rollback / Overwrite capability for resources #32

Closed

[META] Implement basic manual Undo/Rollback capability #89

Closed

dbwiddis mentioned this pull request Dec 14, 2023

[BUG] Multiple REST APIs ignore the FlowFrameworkException status #293

Closed

dbwiddis added 7 commits December 14, 2023 15:19

Deprovision REST and Transport Actions

97c03d4

Signed-off-by: Daniel Widdis <widdis@gmail.com>

Fix errors you find actually running the code

a47e13c

Signed-off-by: Daniel Widdis <widdis@gmail.com>

Add test for Rest deprovision action

5cfbf54

Signed-off-by: Daniel Widdis <widdis@gmail.com>

Initial copypaste of Deprovision Transport Action Test

921164f

Signed-off-by: Daniel Widdis <widdis@gmail.com>

Add some delays to let deletions propagate, reset workflow state

e6167fe

Signed-off-by: Daniel Widdis <widdis@gmail.com>

Improved deprovisioning results and status updates

2f3082d

Signed-off-by: Daniel Widdis <widdis@gmail.com>

Fix bug in resource created parsing

98a177e

Signed-off-by: Daniel Widdis <widdis@gmail.com>

dbwiddis added 2 commits December 14, 2023 15:19

Completed test implementations

bfdee4b

Signed-off-by: Daniel Widdis <widdis@gmail.com>

Fixes after rebase

fad7b2b

Signed-off-by: Daniel Widdis <widdis@gmail.com>

dbwiddis force-pushed the deprovision branch from 0431876 to fad7b2b Compare December 14, 2023 23:21

amitgalitz approved these changes Dec 15, 2023

View reviewed changes

dbwiddis merged commit b3060f4 into opensearch-project:feature/agent_framework Dec 15, 2023
10 checks passed

dbwiddis deleted the deprovision branch December 15, 2023 01:21

dbwiddis mentioned this pull request Dec 15, 2023

[CHORE] Port Feature Branch commits to main and then 2.x #264

Closed

28 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature/agent_framework] Deprovision API #271

[Feature/agent_framework] Deprovision API #271

dbwiddis commented Dec 10, 2023 •

edited

Loading

dbwiddis commented Dec 11, 2023

dbwiddis commented Dec 12, 2023

dbwiddis commented Dec 12, 2023

joshpalis left a comment

joshpalis left a comment

[Feature/agent_framework] Deprovision API #271

[Feature/agent_framework] Deprovision API #271

Conversation

dbwiddis commented Dec 10, 2023 • edited Loading

Description

Issues Resolved

dbwiddis commented Dec 11, 2023

dbwiddis commented Dec 12, 2023

dbwiddis commented Dec 12, 2023

joshpalis left a comment

Choose a reason for hiding this comment

joshpalis left a comment

Choose a reason for hiding this comment

dbwiddis commented Dec 10, 2023 •

edited

Loading