Content
View differences
Updated by Marcello Rocha Pereira about 2 years ago
:warning: I'll post my findings as I go through them
After restoring the background job database dump that Markus provided me, I've run a couple of simple counts over all enqueued jobs. We can see the majority of them are `CreateDateAlertNotificationJob`
Only the top 5 are show above
<figure class="table op-uc-figure_align-center op-uc-figure"><table class="op-uc-table"><thead class="op-uc-table--head"><tr class="op-uc-table--row"><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">count</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">job_class</p></th></tr></thead><tbody><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">2631528</p></td><td class="op-uc-table--cell"><p class="op-uc-p">Notifications::CreateDateAlertsNotificationsJob</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">124</p></td><td class="op-uc-table--cell"><p class="op-uc-p">Notifications::ScheduleReminderMailsJob</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">124</p></td><td class="op-uc-table--cell"><p class="op-uc-p">Notifications::ScheduleDateAlertsNotificationsJob</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">57</p></td><td class="op-uc-table--cell"><p class="op-uc-p">Mails::MailerJob</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">14</p></td><td class="op-uc-table--cell"><p class="op-uc-p">Notifications::WorkflowJob</p></td></tr></tbody></table></figure>
So I've split the queuing of those by day and we see a growing amount starting on the 7th and finally an explosion on the 8th.
<figure class="table op-uc-figure_align-center op-uc-figure"><table class="op-uc-table"><thead class="op-uc-table--head"><tr class="op-uc-table--row"><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">on_queue</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">executed</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">truncated_enqueued_at</p></th></tr></thead><tbody><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">0</p></td><td class="op-uc-table--cell"><p class="op-uc-p">1924</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-01 00:00:00.000000 +00:00</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">0</p></td><td class="op-uc-table--cell"><p class="op-uc-p">93379</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 00:00:00.000000 +00:00</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">0</p></td><td class="op-uc-table--cell"><p class="op-uc-p">84411</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-03 00:00:00.000000 +00:00</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">0</p></td><td class="op-uc-table--cell"><p class="op-uc-p">84432</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-04 00:00:00.000000 +00:00</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">0</p></td><td class="op-uc-table--cell"><p class="op-uc-p">85434</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 00:00:00.000000 +00:00</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">0</p></td><td class="op-uc-table--cell"><p class="op-uc-p">125242</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 00:00:00.000000 +00:00</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">14049</p></td><td class="op-uc-table--cell"><p class="op-uc-p">89830</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 00:00:00.000000 +00:00</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">2617603</p></td><td class="op-uc-table--cell"><p class="op-uc-p">0</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 00:00:00.000000 +00:00</p></td></tr></tbody></table></figure>
Following Jens explanation and looking at the source of the `CreateDateAlertsNotificationsJob`, I looked at the enqueue of the `ScheduleDateAlertsNotificationsJob`
<figure class="table op-uc-figure_align-center op-uc-figure"><table class="op-uc-table"><thead class="op-uc-table--head"><tr class="op-uc-table--row"><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">active_job_id</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">scheduled_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">finished_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">started_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">delay</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">runtime</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">schema</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">error</p></th></tr></thead><tbody><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">862b3ba7-5ea4-48f6-a377-227f80771ce6</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:00:00.070331 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:00:00.172972 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:00:00.121424 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.051</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.051</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">17814761-b30c-48f9-bf82-90807abc3f98</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:00:00.094852 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:02:48.664494 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:00:00.159832 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.064</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:02:48.504</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-02 01:00:00 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">17814761-b30c-48f9-bf82-90807abc3f98</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:00:00.094852 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:32:08.213442 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:02:48.664494 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:02:48.569</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:29:19.548</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">a47880b2-2517-4885-a14c-c8518f3caaae</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:15:00.115414 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:35:06.729832 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:35:06.696190 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:20:06.580</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.033</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">a40e98aa-aca7-4c26-a17b-e3e583da9dae</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:15:00.140329 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:35:06.923172 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:35:06.885191 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:20:06.744</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.037</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr></tbody></table></figure>
This shows the 1am job for 02.05, there's a couple of interesting behaviours there:
1. The job is taking around 30 mins to run
2. There was a try that was interrupted, a common cause of this is OOM errors
3. The long run makes all enqueued jobs to be delayed and this normalizes only around 3:15
The same patterns continues and finally on 05.05 this happens
<figure class="table op-uc-figure_align-center op-uc-figure"><table class="op-uc-table"><thead class="op-uc-table--head"><tr class="op-uc-table--row"><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">active_job_id</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">scheduled_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">finished_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">started_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">delay</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">runtime</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">schema</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">error</p></th></tr></thead><tbody><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">b31a4753-2b68-4e93-889b-20add4456010</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:00:00.802293 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:00:01.350520 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:00:01.315983 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.513</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.034</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">d30ef7fd-afb1-4c64-92f0-04ed3b8263cc</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:00:01.607027 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:01:33.970951 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:00:01.920690 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.313</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:01:32.050</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-05 01:00:01 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">d30ef7fd-afb1-4c64-92f0-04ed3b8263cc</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:00:01.607027 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:03:33.326353 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:01:33.970951 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:01:32.363</p></td><td class="op-uc-table--cell"><p class="op-uc-p">01 00:01:59.355</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">46ce8d24-3012-4ac5-ad76-c809083196cf</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:15:07.237609 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:15:08.005316 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:15:07.952629 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.715</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.052</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">ff4e3375-b326-4c97-9344-b219767a04e2</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:15:08.846730 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:15:09.479305 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:15:09.452708 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.605</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.026</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr></tbody></table></figure>
The job gets interrupted after 1.5 mins and retries taking 25:02 hours to finish, this strangely don't seem to the delay the other jobs.
Same pattern on the 06.05 with the 1am job taking 23:50 hours to finish with 2 interruptions: at 2.5 hours and then at almost 11 hours.
<figure class="table op-uc-figure_align-center op-uc-figure"><table class="op-uc-table"><thead class="op-uc-table--head"><tr class="op-uc-table--row"><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">active_job_id</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">scheduled_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">finished_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">started_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">delay</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">runtime</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">schema</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">error</p></th></tr></thead><tbody><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">ffb9b829-3ba4-4ed6-9e21-ce42f9a8377d</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:00:03.430621 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:00:03.845342 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:00:03.813307 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.382</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.032</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">39860a68-2737-4abb-af82-d277062677b2</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:00:04.132475 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:02:25.641638 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:00:04.401091 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.268</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:02:21.240</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-06 01:00:04 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">39860a68-2737-4abb-af82-d277062677b2</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:00:04.132475 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:13:14.973803 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:02:25.641638 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:02:21.509</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:10:49.332</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-06 01:02:25 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">39860a68-2737-4abb-af82-d277062677b2</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:00:04.132475 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 01:03:57.041996 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:13:14.973803 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:13:10.841</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 23:50:42.068</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr></tbody></table></figure>
Then the pattern below happens on 08.05 until the time of the backup I'm looking at.
<figure class="table op-uc-figure_align-center op-uc-figure"><table class="op-uc-table"><thead class="op-uc-table--head"><tr class="op-uc-table--row"><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">active_job_id</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">scheduled_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">finished_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">started_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">delay</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">runtime</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">schema</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">error</p></th></tr></thead><tbody><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">db053027-3eeb-4ed2-b288-af33fb4eb8ed</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 17:45:07.934343 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:10:45.139565 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:01:44.879429 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 11:16:36.945</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:09:00.260</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-08 05:01:44 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">db053027-3eeb-4ed2-b288-af33fb4eb8ed</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 17:45:07.934343 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:37:39.683227 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:10:45.139565 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 11:25:37.205</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:26:54.543</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">a49bb080-d911-4377-a265-2f5f6b8a048e</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:00:03.705552 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:28:24.638506 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:28:23.722365 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 11:28:20.016</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.916</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">e7be387f-b562-4122-bd13-6f5285080b84</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:00:05.114364 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:55:23.001638 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:28:26.310240 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 11:28:21.195</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:26:56.691</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">100e3d7b-3968-4cac-b008-102a7885880d</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:15:05.443624 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:56:15.159538 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:56:14.280344 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 11:41:08.836</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.879</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">37666428-398e-4938-abad-b1f77bc3ff1e</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:15:06.561415 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 06:22:49.906694 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:56:18.012932 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 11:41:11.451</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:26:31.893</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">5f5ef0e3-d6a9-4249-a015-e56e455a0744</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:30:03.738354 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 06:25:13.555871 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 06:25:12.601089 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 11:55:08.862</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.954</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">90199d16-bc21-4a63-bd82-812561b73451</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:30:04.739588 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 06:33:11.754345 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 06:25:18.473215 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 11:55:13.733</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:07:53.281</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-08 06:25:18 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">90199d16-bc21-4a63-bd82-812561b73451</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:30:04.739588 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 07:00:28.407080 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 06:33:11.754345 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 12:03:07.014</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:27:16.652</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">5fadf25d-020b-4df2-b258-9e41c4d1ed45</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:45:04.274557 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 06:56:08.494854 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 06:56:07.587229 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 12:11:03.312</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.907</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">b75f5a69-2e56-4030-8f66-383a8f4d9c80</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:45:05.481160 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 07:02:36.923730 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 06:56:12.634076 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 12:11:07.152</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:06:24.289</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-08 06:56:12 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">b75f5a69-2e56-4030-8f66-383a8f4d9c80</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:45:05.481160 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 07:31:12.962194 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 07:02:36.923730 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 12:17:31.442</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:28:36.038</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">62f91452-6665-4593-8644-244a1f43f55f</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:00:05.478634 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 07:30:40.052707 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 07:30:38.857178 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 12:30:33.378</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:01.195</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">2c600466-906c-478b-a444-33860a008d25</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:00:06.880589 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 07:59:20.840159 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 07:30:44.075242 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 12:30:37.194</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:28:36.764</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-08 07:30:44 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">2c600466-906c-478b-a444-33860a008d25</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:00:06.880589 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:09:01.492008 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 07:59:20.840159 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 12:59:13.959</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:09:40.651</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-08 07:59:20 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">2c600466-906c-478b-a444-33860a008d25</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:00:06.880589 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:36:48.911702 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:09:01.492008 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 13:08:54.611</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:27:47.419</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">8512aece-892e-4dd3-9835-6f57ecc6926c</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:15:01.796063 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:05:54.583811 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:05:52.783637 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 12:50:50.987</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:01.800</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">d9bcbca5-b2a7-4134-844d-5cecee3876f9</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:15:02.253981 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:33:27.918643 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:05:59.723830 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 12:50:57.469</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:27:28.194</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">8e22fcb5-0086-415e-97c1-5163e0bb6d44</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:30:06.386107 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:47:13.957244 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:47:13.015826 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 13:17:06.629</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.941</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">5c718ede-2ee2-49ad-a6e7-ccd0b764a1ce</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:30:07.287625 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:51:50.322844 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:47:18.517378 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 13:17:11.229</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:04:31.805</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-08 08:47:18 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">5c718ede-2ee2-49ad-a6e7-ccd0b764a1ce</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:30:07.287625 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 09:24:44.512229 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:51:50.322844 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 13:21:43.035</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:32:54.189</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">3c6b2e6f-ff43-453f-8493-ff317e03a8d8</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:45:03.635417 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 09:26:05.906154 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 09:26:05.118873 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 13:41:01.483</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.787</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">41e17a41-8790-4939-a87a-001a798698c2</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:45:04.535777 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 09:53:48.219635 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 09:26:10.593886 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 13:41:06.058</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:27:37.625</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-08 09:26:10 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">41e17a41-8790-4939-a87a-001a798698c2</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:45:04.535777 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 10:04:39.564416 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 09:53:48.219635 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 14:08:43.683</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:10:51.344</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-08 09:53:48 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">41e17a41-8790-4939-a87a-001a798698c2</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:45:04.535777 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 10:35:36.089022 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 10:04:39.564416 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 14:19:35.028</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:30:56.524</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">2f7efa66-e891-46a2-8d30-93c61eac6f98</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 20:00:01.676876 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 10:19:07.903468 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 10:19:06.874122 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 14:19:05.197</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:01.029</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">22996e3e-be6f-4db4-8748-e599e88ad9e5</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 20:00:02.187321 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 10:49:35.411589 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 10:19:12.963022 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 14:19:10.775</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:30:22.448</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">e69fb8de-13e0-4039-b17d-5d4e2356acc5</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 20:15:05.697095 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 11:25:57.650256 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 11:25:56.743363 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 15:10:51.046</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.906</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">99d933a1-0565-42f5-96a1-fa55c0ef0952</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 20:15:06.600098 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 11:58:50.760613 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 11:26:02.662943 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 15:10:56.062</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:32:48.097</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr></tbody></table></figure>
After restoring the background job database dump that Markus provided me, I've run a couple of simple counts over all enqueued jobs. We can see the majority of them are `CreateDateAlertNotificationJob`
Only the top 5 are show above
<figure class="table op-uc-figure_align-center op-uc-figure"><table class="op-uc-table"><thead class="op-uc-table--head"><tr class="op-uc-table--row"><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">count</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">job_class</p></th></tr></thead><tbody><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">2631528</p></td><td class="op-uc-table--cell"><p class="op-uc-p">Notifications::CreateDateAlertsNotificationsJob</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">124</p></td><td class="op-uc-table--cell"><p class="op-uc-p">Notifications::ScheduleReminderMailsJob</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">124</p></td><td class="op-uc-table--cell"><p class="op-uc-p">Notifications::ScheduleDateAlertsNotificationsJob</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">57</p></td><td class="op-uc-table--cell"><p class="op-uc-p">Mails::MailerJob</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">14</p></td><td class="op-uc-table--cell"><p class="op-uc-p">Notifications::WorkflowJob</p></td></tr></tbody></table></figure>
So I've split the queuing of those by day and we see a growing amount starting on the 7th and finally an explosion on the 8th.
<figure class="table op-uc-figure_align-center op-uc-figure"><table class="op-uc-table"><thead class="op-uc-table--head"><tr class="op-uc-table--row"><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">on_queue</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">executed</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">truncated_enqueued_at</p></th></tr></thead><tbody><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">0</p></td><td class="op-uc-table--cell"><p class="op-uc-p">1924</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-01 00:00:00.000000 +00:00</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">0</p></td><td class="op-uc-table--cell"><p class="op-uc-p">93379</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 00:00:00.000000 +00:00</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">0</p></td><td class="op-uc-table--cell"><p class="op-uc-p">84411</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-03 00:00:00.000000 +00:00</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">0</p></td><td class="op-uc-table--cell"><p class="op-uc-p">84432</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-04 00:00:00.000000 +00:00</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">0</p></td><td class="op-uc-table--cell"><p class="op-uc-p">85434</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 00:00:00.000000 +00:00</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">0</p></td><td class="op-uc-table--cell"><p class="op-uc-p">125242</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 00:00:00.000000 +00:00</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">14049</p></td><td class="op-uc-table--cell"><p class="op-uc-p">89830</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 00:00:00.000000 +00:00</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">2617603</p></td><td class="op-uc-table--cell"><p class="op-uc-p">0</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 00:00:00.000000 +00:00</p></td></tr></tbody></table></figure>
Following Jens explanation and looking at the source of the `CreateDateAlertsNotificationsJob`, I looked at the enqueue of the `ScheduleDateAlertsNotificationsJob`
<figure class="table op-uc-figure_align-center op-uc-figure"><table class="op-uc-table"><thead class="op-uc-table--head"><tr class="op-uc-table--row"><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">active_job_id</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">scheduled_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">finished_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">started_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">delay</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">runtime</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">schema</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">error</p></th></tr></thead><tbody><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">862b3ba7-5ea4-48f6-a377-227f80771ce6</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:00:00.070331 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:00:00.172972 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:00:00.121424 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.051</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.051</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">17814761-b30c-48f9-bf82-90807abc3f98</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:00:00.094852 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:02:48.664494 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:00:00.159832 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.064</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:02:48.504</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-02 01:00:00 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">17814761-b30c-48f9-bf82-90807abc3f98</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:00:00.094852 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:32:08.213442 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:02:48.664494 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:02:48.569</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:29:19.548</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">a47880b2-2517-4885-a14c-c8518f3caaae</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:15:00.115414 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:35:06.729832 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:35:06.696190 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:20:06.580</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.033</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">a40e98aa-aca7-4c26-a17b-e3e583da9dae</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:15:00.140329 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:35:06.923172 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-02 01:35:06.885191 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:20:06.744</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.037</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr></tbody></table></figure>
This shows the 1am job for 02.05, there's a couple of interesting behaviours there:
1. The job is taking around 30 mins to run
2. There was a try that was interrupted, a common cause of this is OOM errors
3. The long run makes all enqueued jobs to be delayed and this normalizes only around 3:15
The same patterns continues and finally on 05.05 this happens
<figure class="table op-uc-figure_align-center op-uc-figure"><table class="op-uc-table"><thead class="op-uc-table--head"><tr class="op-uc-table--row"><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">active_job_id</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">scheduled_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">finished_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">started_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">delay</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">runtime</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">schema</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">error</p></th></tr></thead><tbody><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">b31a4753-2b68-4e93-889b-20add4456010</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:00:00.802293 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:00:01.350520 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:00:01.315983 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.513</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.034</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">d30ef7fd-afb1-4c64-92f0-04ed3b8263cc</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:00:01.607027 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:01:33.970951 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:00:01.920690 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.313</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:01:32.050</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-05 01:00:01 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">d30ef7fd-afb1-4c64-92f0-04ed3b8263cc</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:00:01.607027 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:03:33.326353 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:01:33.970951 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:01:32.363</p></td><td class="op-uc-table--cell"><p class="op-uc-p">01 00:01:59.355</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">46ce8d24-3012-4ac5-ad76-c809083196cf</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:15:07.237609 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:15:08.005316 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:15:07.952629 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.715</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.052</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">ff4e3375-b326-4c97-9344-b219767a04e2</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:15:08.846730 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:15:09.479305 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-05 01:15:09.452708 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.605</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.026</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr></tbody></table></figure>
The job gets interrupted after 1.5 mins and retries taking 25:02 hours to finish, this strangely don't seem to the delay the other jobs.
Same pattern on the 06.05 with the 1am job taking 23:50 hours to finish with 2 interruptions: at 2.5 hours and then at almost 11 hours.
<figure class="table op-uc-figure_align-center op-uc-figure"><table class="op-uc-table"><thead class="op-uc-table--head"><tr class="op-uc-table--row"><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">active_job_id</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">scheduled_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">finished_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">started_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">delay</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">runtime</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">schema</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">error</p></th></tr></thead><tbody><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">ffb9b829-3ba4-4ed6-9e21-ce42f9a8377d</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:00:03.430621 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:00:03.845342 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:00:03.813307 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.382</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.032</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">39860a68-2737-4abb-af82-d277062677b2</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:00:04.132475 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:02:25.641638 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:00:04.401091 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.268</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:02:21.240</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-06 01:00:04 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">39860a68-2737-4abb-af82-d277062677b2</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:00:04.132475 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:13:14.973803 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:02:25.641638 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:02:21.509</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:10:49.332</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-06 01:02:25 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">39860a68-2737-4abb-af82-d277062677b2</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:00:04.132475 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 01:03:57.041996 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-06 01:13:14.973803 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:13:10.841</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 23:50:42.068</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr></tbody></table></figure>
Then the pattern below happens on 08.05 until the time of the backup I'm looking at.
<figure class="table op-uc-figure_align-center op-uc-figure"><table class="op-uc-table"><thead class="op-uc-table--head"><tr class="op-uc-table--row"><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">active_job_id</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">scheduled_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">finished_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">started_at</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">delay</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">runtime</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">schema</p></th><th class="op-uc-table--cell op-uc-table--cell_head"><p class="op-uc-p">error</p></th></tr></thead><tbody><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">db053027-3eeb-4ed2-b288-af33fb4eb8ed</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 17:45:07.934343 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:10:45.139565 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:01:44.879429 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 11:16:36.945</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:09:00.260</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-08 05:01:44 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">db053027-3eeb-4ed2-b288-af33fb4eb8ed</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 17:45:07.934343 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:37:39.683227 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:10:45.139565 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 11:25:37.205</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:26:54.543</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">a49bb080-d911-4377-a265-2f5f6b8a048e</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:00:03.705552 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:28:24.638506 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:28:23.722365 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 11:28:20.016</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.916</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">e7be387f-b562-4122-bd13-6f5285080b84</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:00:05.114364 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:55:23.001638 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:28:26.310240 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 11:28:21.195</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:26:56.691</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">100e3d7b-3968-4cac-b008-102a7885880d</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:15:05.443624 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:56:15.159538 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:56:14.280344 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 11:41:08.836</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.879</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">37666428-398e-4938-abad-b1f77bc3ff1e</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:15:06.561415 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 06:22:49.906694 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 05:56:18.012932 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 11:41:11.451</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:26:31.893</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">5f5ef0e3-d6a9-4249-a015-e56e455a0744</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:30:03.738354 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 06:25:13.555871 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 06:25:12.601089 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 11:55:08.862</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.954</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">90199d16-bc21-4a63-bd82-812561b73451</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:30:04.739588 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 06:33:11.754345 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 06:25:18.473215 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 11:55:13.733</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:07:53.281</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-08 06:25:18 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">90199d16-bc21-4a63-bd82-812561b73451</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:30:04.739588 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 07:00:28.407080 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 06:33:11.754345 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 12:03:07.014</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:27:16.652</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">5fadf25d-020b-4df2-b258-9e41c4d1ed45</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:45:04.274557 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 06:56:08.494854 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 06:56:07.587229 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 12:11:03.312</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.907</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">b75f5a69-2e56-4030-8f66-383a8f4d9c80</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:45:05.481160 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 07:02:36.923730 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 06:56:12.634076 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 12:11:07.152</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:06:24.289</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-08 06:56:12 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">b75f5a69-2e56-4030-8f66-383a8f4d9c80</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 18:45:05.481160 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 07:31:12.962194 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 07:02:36.923730 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 12:17:31.442</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:28:36.038</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">62f91452-6665-4593-8644-244a1f43f55f</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:00:05.478634 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 07:30:40.052707 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 07:30:38.857178 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 12:30:33.378</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:01.195</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">2c600466-906c-478b-a444-33860a008d25</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:00:06.880589 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 07:59:20.840159 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 07:30:44.075242 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 12:30:37.194</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:28:36.764</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-08 07:30:44 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">2c600466-906c-478b-a444-33860a008d25</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:00:06.880589 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:09:01.492008 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 07:59:20.840159 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 12:59:13.959</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:09:40.651</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-08 07:59:20 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">2c600466-906c-478b-a444-33860a008d25</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:00:06.880589 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:36:48.911702 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:09:01.492008 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 13:08:54.611</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:27:47.419</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">8512aece-892e-4dd3-9835-6f57ecc6926c</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:15:01.796063 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:05:54.583811 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:05:52.783637 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 12:50:50.987</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:01.800</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">d9bcbca5-b2a7-4134-844d-5cecee3876f9</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:15:02.253981 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:33:27.918643 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:05:59.723830 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 12:50:57.469</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:27:28.194</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">8e22fcb5-0086-415e-97c1-5163e0bb6d44</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:30:06.386107 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:47:13.957244 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:47:13.015826 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 13:17:06.629</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.941</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">5c718ede-2ee2-49ad-a6e7-ccd0b764a1ce</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:30:07.287625 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:51:50.322844 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:47:18.517378 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 13:17:11.229</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:04:31.805</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-08 08:47:18 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">5c718ede-2ee2-49ad-a6e7-ccd0b764a1ce</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:30:07.287625 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 09:24:44.512229 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 08:51:50.322844 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 13:21:43.035</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:32:54.189</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">3c6b2e6f-ff43-453f-8493-ff317e03a8d8</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:45:03.635417 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 09:26:05.906154 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 09:26:05.118873 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 13:41:01.483</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.787</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">41e17a41-8790-4939-a87a-001a798698c2</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:45:04.535777 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 09:53:48.219635 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 09:26:10.593886 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 13:41:06.058</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:27:37.625</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-08 09:26:10 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">41e17a41-8790-4939-a87a-001a798698c2</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:45:04.535777 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 10:04:39.564416 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 09:53:48.219635 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 14:08:43.683</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:10:51.344</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">GoodJob::InterruptError: Interrupted after starting perform at '2024-05-08 09:53:48 UTC'</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">41e17a41-8790-4939-a87a-001a798698c2</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 19:45:04.535777 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 10:35:36.089022 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 10:04:39.564416 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 14:19:35.028</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:30:56.524</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">2f7efa66-e891-46a2-8d30-93c61eac6f98</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 20:00:01.676876 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 10:19:07.903468 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 10:19:06.874122 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 14:19:05.197</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:01.029</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">22996e3e-be6f-4db4-8748-e599e88ad9e5</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 20:00:02.187321 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 10:49:35.411589 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 10:19:12.963022 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 14:19:10.775</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:30:22.448</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">e69fb8de-13e0-4039-b17d-5d4e2356acc5</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 20:15:05.697095 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 11:25:57.650256 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 11:25:56.743363 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 15:10:51.046</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:00:00.906</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_1524136372_610391_7891</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell"><p class="op-uc-p">99d933a1-0565-42f5-96a1-fa55c0ef0952</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-07 20:15:06.600098 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 11:58:50.760613 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">2024-05-08 11:26:02.662943 +00:00</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 15:10:56.062</p></td><td class="op-uc-table--cell"><p class="op-uc-p">00 00:32:48.097</p></td><td class="op-uc-table--cell"><p class="op-uc-p">instance_community</p></td><td class="op-uc-table--cell"><p class="op-uc-p">null</p></td></tr></tbody></table></figure>