[Search] Review searches clean up

[Dosant]

We should review how and when our searches are currently cleaned up. This includes a strategy for handling discarded searches and reviewing our timeout settings

related: #106395, #106400

Currently, all sessions (both persisted and non-persisted) are cleaned up in the monitoring task

kibana/x-pack/plugins/data_enhanced/server/search/session/check_non_persiseted_sessions.ts

Lines 67 to 99 in 1c5d548

	if (isSessionStale(session, config)) {
	// delete saved object to free up memory
	// TODO: there's a potential rare edge case of deleting an object and then receiving a new trackId for that same session!
	// Maybe we want to change state to deleted and cleanup later?
	logger.debug(`Deleting stale session | ${session.id}`);
	try {
	deletedSessionIds.push(session.id);
	await savedObjectsClient.delete(SEARCH_SESSION_TYPE, session.id, {
	namespace: session.namespaces?.[0],
	});
	} catch (e) {
	logger.error(
	`${SEARCH_SESSIONS_CLEANUP_TASK_TYPE} Error while deleting session ${session.id}: ${e.message}`
	);
	}
	// Send a delete request for each async search to ES
	Object.keys(session.attributes.idMapping).map(async (searchKey: string) => {
	const searchInfo = session.attributes.idMapping[searchKey];
	if (searchInfo.strategy === ENHANCED_ES_SEARCH_STRATEGY) {
	try {
	await client.asyncSearch.delete({ id: searchInfo.id });
	} catch (e) {
	if (e.message !== 'resource_not_found_exception') {
	logger.error(
	`${SEARCH_SESSIONS_CLEANUP_TASK_TYPE} Error while deleting async_search ${searchInfo.id}: ${e.message}`
	);
	}
	}
	}
	});
	}
	})

but we can do it earlier from the client.

Also, existing cleanup from the client is not centralized and causes an out-of-sync state between searches in elasticsearch and kibana's search sessions saved objects #104309, #98698

Possible existing issue 1

On the client we abort unfinished searches if they don't belong to a saved session:

kibana/src/plugins/data/public/search/search_interceptor/search_interceptor.ts

Lines 230 to 232 in 1c5d548

	const cancel = once(() => {
	if (id && !isSavedToBackground) this.deps.http.delete(`/internal/search/${strategy}/${id}`);
	});

This is causing a desync issue, as we don't delete search from non-persisted sessions here

Possible existing issue 2

We can delete a search session

kibana/src/plugins/data/public/search/session/session_service.ts

Lines 305 to 307 in 1c5d548

	if (isStoredSession) {
	await this.sessionsClient.delete(this.state.get().sessionId!);
	}

But since this is happening separately, searchers might get out of sync and still be executing

Possible existing issue 3

Does server side search strategy handles request cancelation? #113423

---

There was a pr that should have addressed this: #99658 but new use-case come up where search session is reused between Dashboard<->Lens and it needs a different approach now.

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[lizozom]

@Dosant It feels like there a lot of duplucates of this issue (for example #106395).
I'd like to leave one issue open, close the others and add it to the performance project board. Do you think that's possible?

[Dosant]

@lizozom, thanks! let's close this one in favor of #106395