Troubleshooting

Mapping Troubleshooting

Many score set mapping states are stuck in processing or queued

On occasions of high volume, the worker job controlling mapping can sputter and fail to complete mapping routines. This causes additional jobs to build up in the queue, stalling the whole process. Unfortunately once this occurs, it becomes a rather manual process to sort out. To fix this, we need to clear the queue of mapping jobs, find all mapping jobs that should be retried, and then retry them (ideally, one at a time).

1. Log onto the staging worker and then connect to our redis instance

ssh mavedb-prod-worker
cd redis-7.2.5/
src/redis-cli -h master.mavedb2-prod-redis.ssidtu.usw2.cache.amazonaws.com --tls -p 6379

2. Clear the mapping queue and current mapping job

ltrim vrs_mapping_queue -1 0
set vrs_mapping_current_job_id ""

3. Wait for the worker processes to clear

It's likely that many worker processes are running because no mapping jobs ever successfully completed. You can check whether worker processes are still running with get arq:queue:health-check. If any jobs are queued or ongoing, you should wait until they complete. This can take a long time if there are many jobs to await.

Once no jobs are queued, it is safe to continue to the next step.

4. Open a new connection to the MaveDB worker

ssh mavedb-prod-worker
sudo docker container ls
sudo docker container exec -it <worker_container_id> bash

5. Begin an interactive Python session on the worker and create a function that adds jobs to the worker

python3

from mavedb.worker.settings import RedisWorkerSettings
from arq import create_pool
async def add_to_worker(user):
     redis = await create_pool(RedisWorkerSettings)
     await redis.enqueue_job("variant_mapper_manager", "manual_1", user)

6. Find the score sets you need to remap

SELECT
	id,
	urn,
	created_by_id,
	modified_by_id,
	mapping_state,
	modification_date
FROM
	scoresets
WHERE
	scoresets.mapping_state = 'processing'
	OR scoresets.mapping_state = 'queued'
ORDER BY
	modification_date DESC

7. Retry mapping jobs by adding score sets to the queue and mapping them

lpush vrs_mapping_queue <score_set_id>

asyncio.run(add_to_worker(<modified_by_id>))

8. Repeat the above for all score sets you need to remap