kelora 2.0.0

<27>Mar 14 08:00:01 worker01 checkout[5331]: Connection timeout to database host db-replica-1.prod.internal after 10404ms
<27>Mar 14 08:00:03 api01 worker[5571]: Payment gateway adyen.gateway.internal rejected transaction c39f2e8b-2392-4450-939c-a51d53906989: insufficient_funds
<27>Mar 14 08:00:05 api01 checkout[5735]: Connection timeout to database host db-replica-1.prod.internal after 22354ms
<27>Mar 14 08:00:06 edge01 checkout[8861]: Upstream search.svc.internal returned 504 for request 267f0b10-b1df-40dd-ba9e-8bcdf85452c6
<27>Mar 14 08:00:09 api02 checkout[2996]: Connection timeout to database host db-replica-2.prod.internal after 26137ms
<27>Mar 14 08:00:11 worker02 checkout[3399]: Connection timeout to database host db-shard-3.prod.internal after 7504ms
<27>Mar 14 08:00:13 web01 checkout[2294]: Connection timeout to database host db-shard-3.prod.internal after 12361ms
<27>Mar 14 08:00:15 edge01 checkout[8849]: Connection timeout to database host db-replica-1.prod.internal after 15399ms
<27>Mar 14 08:00:16 api01 checkout[5450]: Connection timeout to database host db-replica-2.prod.internal after 914ms
<27>Mar 14 08:00:18 web02 worker[4972]: Connection timeout to database host db-replica-1.prod.internal after 5072ms
<27>Mar 14 08:00:20 api01 worker[1860]: Connection timeout to database host db-primary.prod.internal after 20126ms
<27>Mar 14 08:00:22 worker02 checkout[8739]: Failed to acquire lock on resource /var/lib/app/cache.lock after 14300ms
<27>Mar 14 08:00:24 web03 api[3587]: Connection timeout to database host db-shard-3.prod.internal after 16129ms
<27>Mar 14 08:00:27 edge01 scheduler[4267]: Connection timeout to database host db-shard-3.prod.internal after 27239ms
<27>Mar 14 08:00:29 worker01 checkout[1969]: Upstream auth.svc.internal returned 502 for request 17157763-10d7-441b-8c4f-f4820148a90e
<27>Mar 14 08:00:30 api01 api[3868]: Failed to acquire lock on resource /var/lib/app/session.lock after 9196ms
<27>Mar 14 08:00:33 web03 api[7504]: Upstream notifications.svc.internal returned 503 for request 98fbb537-9224-4e4d-87b6-c66c75a511f0
<27>Mar 14 08:00:35 web03 checkout[1613]: Connection timeout to database host db-replica-1.prod.internal after 12918ms
<27>Mar 14 08:00:36 api02 api[6489]: Connection timeout to database host db-primary.prod.internal after 17428ms
<27>Mar 14 08:00:39 edge01 scheduler[3198]: Connection timeout to database host db-shard-3.prod.internal after 24362ms
<27>Mar 14 08:00:41 api02 worker[2845]: Upstream notifications.svc.internal returned 504 for request 24b22dcc-b0c1-4d44-8f3f-11b9fba31423
<27>Mar 14 08:00:43 web03 scheduler[5215]: Connection timeout to database host db-replica-1.prod.internal after 6732ms
<27>Mar 14 08:00:44 web01 checkout[5833]: Upstream notifications.svc.internal returned 502 for request 34114609-03c7-4f83-9bb8-09d3a7cba8e4
<27>Mar 14 08:00:46 web03 scheduler[8007]: Connection timeout to database host db-replica-1.prod.internal after 9625ms
<27>Mar 14 08:00:49 web02 scheduler[2562]: Connection timeout to database host db-replica-2.prod.internal after 14855ms
<27>Mar 14 08:00:50 web03 scheduler[6301]: Connection timeout to database host db-shard-3.prod.internal after 15262ms
<27>Mar 14 08:00:52 web01 checkout[7284]: Connection timeout to database host db-replica-1.prod.internal after 27721ms
<27>Mar 14 08:00:54 edge01 checkout[6400]: Failed to acquire lock on resource /tmp/deploy.lock after 10394ms
<27>Mar 14 08:00:57 web01 checkout[6890]: Upstream search.svc.internal returned 503 for request 2154b57d-484a-48d9-9922-386d68905925
<27>Mar 14 08:00:58 web01 api[8625]: Upstream notifications.svc.internal returned 504 for request 174d25d8-a0d0-4462-864d-e963784969e2
<27>Mar 14 08:01:01 web03 scheduler[9233]: Upstream billing.svc.internal returned 504 for request 4d86cabe-e4cd-42e8-b08e-40ad50235f24
<27>Mar 14 08:01:03 web02 checkout[7021]: Upstream notifications.svc.internal returned 503 for request 211721b3-43b2-4e4f-b443-0c06e95ca619
<27>Mar 14 08:01:04 api01 worker[8834]: Connection timeout to database host db-shard-3.prod.internal after 21037ms
<27>Mar 14 08:01:06 edge01 checkout[7813]: Connection timeout to database host db-shard-3.prod.internal after 15185ms
<27>Mar 14 08:01:09 api01 worker[2341]: Connection timeout to database host db-primary.prod.internal after 29311ms
<27>Mar 14 08:01:11 worker01 api[2838]: Connection timeout to database host db-primary.prod.internal after 22674ms
<27>Mar 14 08:01:12 api01 worker[3695]: Connection timeout to database host db-replica-1.prod.internal after 23376ms
<27>Mar 14 08:01:15 edge01 api[4365]: Connection timeout to database host db-replica-1.prod.internal after 24911ms
<27>Mar 14 08:01:17 web02 scheduler[1911]: Upstream billing.svc.internal returned 504 for request 7e5a52b9-71f2-4352-8e77-7c1abda8c9f1
<27>Mar 14 08:01:19 web03 scheduler[8484]: Connection timeout to database host db-replica-1.prod.internal after 24095ms
<27>Mar 14 08:01:20 edge01 checkout[1356]: Upstream search.svc.internal returned 500 for request bd04d654-2ec9-401b-b621-17229312d779
<27>Mar 14 08:01:23 api02 api[4570]: Connection timeout to database host db-primary.prod.internal after 24842ms
<27>Mar 14 08:01:24 web01 scheduler[6644]: Upstream search.svc.internal returned 504 for request 1c5c3711-cbcd-4bb0-8c69-263fb26c500c
<27>Mar 14 08:01:26 web01 api[8734]: Connection timeout to database host db-primary.prod.internal after 26590ms
<27>Mar 14 08:01:28 api02 scheduler[3948]: Upstream search.svc.internal returned 503 for request 32aedf3b-d938-4ccf-abb1-e0fe7a32b7a7
<27>Mar 14 08:01:30 worker02 checkout[7266]: Connection timeout to database host db-replica-1.prod.internal after 16498ms
<27>Mar 14 08:01:33 worker02 scheduler[2314]: Payment gateway stripe.gateway.internal rejected transaction 6c90b8cc-3fd3-4b60-8b90-6e7d3e1391bd: insufficient_funds
<27>Mar 14 08:01:34 worker01 api[3924]: Connection timeout to database host db-primary.prod.internal after 18589ms
<27>Mar 14 08:01:37 web03 worker[1056]: Upstream notifications.svc.internal returned 504 for request a4716325-eac2-44bc-a22d-8c8522830374
<27>Mar 14 08:01:38 api02 scheduler[9920]: Connection timeout to database host db-primary.prod.internal after 22625ms
<27>Mar 14 08:01:41 worker02 worker[5055]: Connection timeout to database host db-replica-1.prod.internal after 24545ms
<27>Mar 14 08:01:43 worker01 worker[5516]: Connection timeout to database host db-primary.prod.internal after 13293ms
<27>Mar 14 08:01:44 web02 api[7868]: Upstream auth.svc.internal returned 500 for request 342ac636-234b-4fc6-a5ec-b7eaf8f1b679
<27>Mar 14 08:01:46 api01 worker[2102]: Connection timeout to database host db-primary.prod.internal after 29274ms
<27>Mar 14 08:01:48 web01 scheduler[1797]: Connection timeout to database host db-primary.prod.internal after 2001ms
<27>Mar 14 08:01:50 web01 scheduler[8191]: Connection timeout to database host db-replica-2.prod.internal after 22417ms
<27>Mar 14 08:01:52 api01 api[3295]: Connection timeout to database host db-replica-1.prod.internal after 19782ms
<27>Mar 14 08:01:55 api01 checkout[6253]: Connection timeout to database host db-replica-1.prod.internal after 22933ms
<27>Mar 14 08:01:56 api02 worker[9540]: Connection timeout to database host db-replica-2.prod.internal after 3560ms
<27>Mar 14 08:01:58 worker02 checkout[5508]: Failed to acquire lock on resource /var/lib/app/cache.lock after 6105ms
<27>Mar 14 08:02:00 web03 scheduler[9120]: Payment gateway braintree.gateway.internal rejected transaction b360c3c7-0ea1-4d50-bfad-90b7c4143fc0: insufficient_funds
<27>Mar 14 08:02:02 worker01 scheduler[9592]: Connection timeout to database host db-replica-2.prod.internal after 7383ms
<27>Mar 14 08:02:05 worker02 scheduler[3444]: Connection timeout to database host db-primary.prod.internal after 20048ms
<27>Mar 14 08:02:07 worker02 worker[9813]: Connection timeout to database host db-replica-1.prod.internal after 12608ms
<27>Mar 14 08:02:09 api01 worker[5927]: Connection timeout to database host db-replica-2.prod.internal after 2759ms
<27>Mar 14 08:02:10 edge01 api[7761]: Connection timeout to database host db-shard-3.prod.internal after 16699ms
<27>Mar 14 08:02:13 web03 scheduler[4978]: Connection timeout to database host db-shard-3.prod.internal after 1312ms
<27>Mar 14 08:02:15 api01 checkout[2295]: Connection timeout to database host db-shard-3.prod.internal after 8494ms
<27>Mar 14 08:02:17 worker01 api[9775]: Failed to acquire lock on resource /tmp/deploy.lock after 2140ms
<27>Mar 14 08:02:18 web01 checkout[7179]: Connection timeout to database host db-shard-3.prod.internal after 2145ms
<27>Mar 14 08:02:20 web02 worker[4555]: Connection timeout to database host db-shard-3.prod.internal after 22375ms
<27>Mar 14 08:02:23 api01 checkout[5967]: Upstream notifications.svc.internal returned 500 for request c533c21d-1ce7-4a47-845d-549aec3930f8
<27>Mar 14 08:02:24 api01 worker[2922]: Connection timeout to database host db-replica-1.prod.internal after 28041ms
<27>Mar 14 08:02:27 api01 worker[7498]: Connection timeout to database host db-shard-3.prod.internal after 14988ms
<27>Mar 14 08:02:28 worker01 checkout[7236]: Failed to acquire lock on resource /var/lib/app/session.lock after 8046ms
<27>Mar 14 08:02:31 worker01 checkout[5285]: Upstream auth.svc.internal returned 504 for request de2e2988-4fd1-400a-84a9-1fc0e8050404
<27>Mar 14 08:02:33 edge01 scheduler[2615]: Connection timeout to database host db-shard-3.prod.internal after 3958ms
<27>Mar 14 08:02:35 worker01 worker[7073]: Connection timeout to database host db-primary.prod.internal after 19849ms
<27>Mar 14 08:02:37 worker02 api[4625]: Failed to acquire lock on resource /var/lib/app/queue.lock after 7730ms
<27>Mar 14 08:02:38 web01 checkout[7841]: Failed to acquire lock on resource /var/lib/app/queue.lock after 1125ms
<27>Mar 14 08:02:41 web03 worker[6404]: Payment gateway stripe.gateway.internal rejected transaction 5e371a5e-5016-4c26-9f15-18dac669e10e: insufficient_funds
<27>Mar 14 08:02:42 worker02 worker[9180]: Connection timeout to database host db-shard-3.prod.internal after 7520ms
<27>Mar 14 08:02:45 api02 scheduler[9037]: Connection timeout to database host db-primary.prod.internal after 5471ms
<27>Mar 14 08:02:47 web03 checkout[3773]: Connection timeout to database host db-replica-2.prod.internal after 5794ms
<27>Mar 14 08:02:48 edge01 worker[9882]: Connection timeout to database host db-primary.prod.internal after 12903ms
<27>Mar 14 08:02:50 web01 api[1794]: Failed to acquire lock on resource /tmp/deploy.lock after 6120ms
<27>Mar 14 08:02:52 worker02 worker[4791]: Connection timeout to database host db-shard-3.prod.internal after 23346ms
<27>Mar 14 08:02:54 web03 api[4583]: Connection timeout to database host db-replica-1.prod.internal after 14770ms
<27>Mar 14 08:02:57 worker01 api[8916]: Connection timeout to database host db-shard-3.prod.internal after 9560ms
<27>Mar 14 08:02:59 web01 api[9719]: Connection timeout to database host db-primary.prod.internal after 8005ms
<27>Mar 14 08:03:01 web01 api[9673]: Connection timeout to database host db-replica-1.prod.internal after 10692ms
<27>Mar 14 08:03:03 api02 api[9228]: Failed to acquire lock on resource /var/lib/app/cache.lock after 7240ms
<27>Mar 14 08:03:05 web03 api[2578]: Upstream billing.svc.internal returned 504 for request 6def65d3-f061-4c9a-8f8b-e78bf3dc1c81
<27>Mar 14 08:03:06 api01 worker[9625]: Connection timeout to database host db-replica-1.prod.internal after 23129ms
<27>Mar 14 08:03:09 worker01 checkout[7502]: Upstream search.svc.internal returned 504 for request 4cce37c8-c081-4ef7-8bbc-bdb702c52408
<27>Mar 14 08:03:10 worker01 scheduler[8520]: Connection timeout to database host db-replica-2.prod.internal after 29762ms
<27>Mar 14 08:03:12 api01 checkout[2320]: Upstream search.svc.internal returned 500 for request 8d01da4e-88d2-4f72-8d55-03f91e0de418
<27>Mar 14 08:03:14 web02 scheduler[7213]: Connection timeout to database host db-replica-2.prod.internal after 15357ms
<27>Mar 14 08:03:17 edge01 api[1154]: Upstream search.svc.internal returned 500 for request 408154c4-34af-409a-863d-7864be5eac4e
<27>Mar 14 08:03:18 web02 scheduler[8109]: Connection timeout to database host db-primary.prod.internal after 15423ms
<27>Mar 14 08:03:21 web02 api[4764]: Connection timeout to database host db-shard-3.prod.internal after 11188ms
<27>Mar 14 08:03:22 edge01 checkout[1762]: Connection timeout to database host db-primary.prod.internal after 24084ms
<27>Mar 14 08:03:24 api02 api[3938]: Connection timeout to database host db-replica-2.prod.internal after 13645ms
<27>Mar 14 08:03:27 web01 worker[3362]: Connection timeout to database host db-replica-1.prod.internal after 23962ms
<27>Mar 14 08:03:29 web02 checkout[3661]: Upstream notifications.svc.internal returned 502 for request 6e90fc1a-be8f-4b85-9f01-b135eae8d982
<27>Mar 14 08:03:30 worker02 checkout[7288]: Connection timeout to database host db-shard-3.prod.internal after 11827ms
<27>Mar 14 08:03:32 web02 checkout[1879]: Failed to acquire lock on resource /var/lib/app/queue.lock after 5458ms
<27>Mar 14 08:03:34 worker02 worker[2245]: Connection timeout to database host db-primary.prod.internal after 29238ms
<27>Mar 14 08:03:37 worker02 checkout[1485]: Upstream search.svc.internal returned 504 for request 54583be6-3bbf-4245-ad2f-eec6847d493d
<27>Mar 14 08:03:39 edge01 scheduler[4737]: Connection timeout to database host db-replica-1.prod.internal after 2604ms
<27>Mar 14 08:03:41 worker02 scheduler[4043]: Connection timeout to database host db-primary.prod.internal after 3570ms
<27>Mar 14 08:03:43 web02 checkout[5037]: Upstream search.svc.internal returned 503 for request 612e2243-2cdb-4524-8d35-b322eac3fe33
<27>Mar 14 08:03:44 web02 checkout[3526]: Connection timeout to database host db-replica-1.prod.internal after 24702ms
<27>Mar 14 08:03:47 web03 scheduler[6188]: Upstream auth.svc.internal returned 500 for request 3ae8a471-9b21-4792-b6ee-a1a7ade7e763
<27>Mar 14 08:03:49 web02 api[1085]: Connection timeout to database host db-replica-1.prod.internal after 19085ms
<27>Mar 14 08:03:51 edge01 checkout[5419]: Failed to acquire lock on resource /tmp/deploy.lock after 6553ms
<27>Mar 14 08:03:52 web03 api[4070]: Connection timeout to database host db-replica-2.prod.internal after 28118ms
<27>Mar 14 08:03:55 edge01 scheduler[2692]: Upstream notifications.svc.internal returned 503 for request 9c977067-2ad6-4ac8-8236-48125aa5868c
<27>Mar 14 08:03:56 web02 checkout[2524]: Upstream notifications.svc.internal returned 500 for request a9e7054b-f979-451e-8c13-260499a508bb
<27>Mar 14 08:03:59 worker02 api[5781]: Upstream auth.svc.internal returned 502 for request 6dd8ed07-7346-47e4-8ab1-6b32af2bdc4d
<27>Mar 14 08:04:01 worker02 api[4977]: Failed to acquire lock on resource /tmp/deploy.lock after 8957ms
<27>Mar 14 08:04:02 worker02 worker[3274]: Connection timeout to database host db-replica-2.prod.internal after 25189ms
<27>Mar 14 08:04:05 api02 checkout[9079]: Connection timeout to database host db-replica-1.prod.internal after 26760ms
<27>Mar 14 08:04:06 web02 worker[8121]: Connection timeout to database host db-primary.prod.internal after 1529ms
<27>Mar 14 08:04:09 worker02 checkout[8933]: Connection timeout to database host db-replica-2.prod.internal after 8524ms
<27>Mar 14 08:04:10 worker01 checkout[5040]: Payment gateway stripe.gateway.internal rejected transaction ffef564f-c1c2-41c7-88eb-8473b10577be: insufficient_funds
<27>Mar 14 08:04:13 api01 scheduler[2752]: Connection timeout to database host db-replica-1.prod.internal after 14583ms
<27>Mar 14 08:04:14 api02 api[7483]: Payment gateway adyen.gateway.internal rejected transaction 4f191fc5-8af3-4e02-aaab-95879e3755a8: insufficient_funds
<27>Mar 14 08:04:17 worker02 checkout[8212]: Connection timeout to database host db-shard-3.prod.internal after 5093ms
<27>Mar 14 08:04:19 worker02 worker[5922]: Upstream search.svc.internal returned 504 for request 7b4c3f31-0a48-461c-99f0-92249cdd409a
<27>Mar 14 08:04:21 api01 scheduler[6147]: Connection timeout to database host db-replica-2.prod.internal after 17117ms
<27>Mar 14 08:04:22 worker02 checkout[5775]: Connection timeout to database host db-primary.prod.internal after 16697ms
<27>Mar 14 08:04:25 api01 checkout[7161]: Connection timeout to database host db-primary.prod.internal after 3306ms
<27>Mar 14 08:04:27 worker02 checkout[2865]: Connection timeout to database host db-replica-2.prod.internal after 5075ms
<27>Mar 14 08:04:28 web03 api[8563]: Connection timeout to database host db-shard-3.prod.internal after 18577ms
<27>Mar 14 08:04:30 edge01 checkout[2133]: Connection timeout to database host db-shard-3.prod.internal after 12109ms
<27>Mar 14 08:04:33 edge01 worker[5961]: Connection timeout to database host db-replica-2.prod.internal after 6541ms
<27>Mar 14 08:04:34 api02 worker[7932]: Connection timeout to database host db-shard-3.prod.internal after 4465ms
<27>Mar 14 08:04:37 web02 scheduler[8851]: Connection timeout to database host db-replica-1.prod.internal after 22546ms
<27>Mar 14 08:04:39 worker02 api[6890]: Upstream search.svc.internal returned 500 for request 169f7529-4cd0-47cb-b1f7-edec862fd906
<27>Mar 14 08:04:40 web02 worker[6816]: Failed to acquire lock on resource /tmp/deploy.lock after 5743ms
<27>Mar 14 08:04:42 web01 scheduler[6432]: Upstream auth.svc.internal returned 503 for request 9bef1081-1b46-481c-b22e-d167b915dae6
<27>Mar 14 08:04:45 web02 api[2615]: Connection timeout to database host db-primary.prod.internal after 25009ms
<27>Mar 14 08:04:47 api01 worker[8303]: Connection timeout to database host db-replica-2.prod.internal after 19587ms
<27>Mar 14 08:04:49 worker02 scheduler[9199]: Upstream billing.svc.internal returned 503 for request e7b7d95b-9874-4132-8474-ff24d9e5da28
<27>Mar 14 08:04:50 worker01 api[8909]: Connection timeout to database host db-replica-1.prod.internal after 9443ms
<27>Mar 14 08:04:52 api02 scheduler[2406]: Connection timeout to database host db-replica-1.prod.internal after 13556ms
<27>Mar 14 08:04:54 web03 checkout[6110]: Upstream auth.svc.internal returned 504 for request 7d46f1c7-f9ea-4a24-b436-50ff0fb05f40
<27>Mar 14 08:04:57 edge01 worker[9537]: Connection timeout to database host db-shard-3.prod.internal after 27820ms
<27>Mar 14 08:04:59 worker01 scheduler[2599]: Connection timeout to database host db-shard-3.prod.internal after 20837ms
<27>Mar 14 08:05:01 edge01 checkout[2100]: Upstream notifications.svc.internal returned 503 for request 21b95085-59d7-4947-97d0-b8849edfe41b
<27>Mar 14 08:05:03 web01 api[1370]: Connection timeout to database host db-replica-2.prod.internal after 20319ms
<27>Mar 14 08:05:05 web02 worker[4996]: Upstream auth.svc.internal returned 503 for request 4c0691a0-d2fd-4cf3-b3fd-23cd8871063e
<27>Mar 14 08:05:06 web03 checkout[7985]: Connection timeout to database host db-primary.prod.internal after 8001ms
<27>Mar 14 08:05:09 api01 scheduler[4022]: Connection timeout to database host db-replica-2.prod.internal after 13574ms
<27>Mar 14 08:05:10 worker02 scheduler[1478]: Connection timeout to database host db-shard-3.prod.internal after 18879ms
<27>Mar 14 08:05:12 edge01 scheduler[7370]: Connection timeout to database host db-replica-2.prod.internal after 27023ms
<27>Mar 14 08:05:14 api01 worker[5560]: Connection timeout to database host db-replica-1.prod.internal after 9488ms
<27>Mar 14 08:05:16 web02 worker[6993]: Connection timeout to database host db-primary.prod.internal after 23700ms
<27>Mar 14 08:05:19 api01 scheduler[2867]: Upstream search.svc.internal returned 504 for request f92cc326-0ec5-4570-afe9-f5e2d207c87d
<27>Mar 14 08:05:21 edge01 checkout[7367]: Connection timeout to database host db-shard-3.prod.internal after 6739ms
<27>Mar 14 08:05:23 web02 checkout[6766]: Upstream search.svc.internal returned 504 for request 9fbdc76c-9ea7-4438-9e0e-b595f71b37db
<27>Mar 14 08:05:25 web03 checkout[2388]: Connection timeout to database host db-replica-1.prod.internal after 25831ms
<27>Mar 14 08:05:26 worker01 api[4908]: Connection timeout to database host db-shard-3.prod.internal after 28806ms
<27>Mar 14 08:05:29 web02 worker[7116]: Connection timeout to database host db-shard-3.prod.internal after 3478ms
<27>Mar 14 08:05:30 worker02 scheduler[3258]: Upstream notifications.svc.internal returned 500 for request 41a17616-1627-4c09-b172-08918906bc21
<27>Mar 14 08:05:33 worker02 worker[8973]: Upstream auth.svc.internal returned 500 for request 080a3849-43fa-4c5b-916f-8399da94f71e
<27>Mar 14 08:05:34 web03 scheduler[5732]: Connection timeout to database host db-replica-1.prod.internal after 10222ms
<27>Mar 14 08:05:36 web03 checkout[8389]: Connection timeout to database host db-primary.prod.internal after 2157ms
<27>Mar 14 08:05:38 worker01 api[8939]: Connection timeout to database host db-shard-3.prod.internal after 18515ms
<27>Mar 14 08:05:40 api01 scheduler[9281]: Connection timeout to database host db-shard-3.prod.internal after 4238ms
<27>Mar 14 08:05:43 worker02 scheduler[7809]: Failed to acquire lock on resource /var/lib/app/session.lock after 7163ms
<27>Mar 14 08:05:45 edge01 worker[2354]: Connection timeout to database host db-replica-2.prod.internal after 22352ms
<27>Mar 14 08:05:46 api01 checkout[1535]: Connection timeout to database host db-shard-3.prod.internal after 13497ms
<27>Mar 14 08:05:49 api01 checkout[3755]: Upstream search.svc.internal returned 504 for request 2f547489-cf8d-47b3-a96a-5a3b63259788
<27>Mar 14 08:05:51 edge01 api[2871]: Connection timeout to database host db-replica-2.prod.internal after 5829ms
<27>Mar 14 08:05:53 worker01 api[8231]: Upstream billing.svc.internal returned 504 for request 09a9f488-c14d-46e8-905d-ce2459bbcecb
<27>Mar 14 08:05:54 worker02 api[5055]: Connection timeout to database host db-replica-2.prod.internal after 29293ms
<27>Mar 14 08:05:57 web01 scheduler[5305]: Connection timeout to database host db-replica-2.prod.internal after 29346ms
<27>Mar 14 08:05:59 web02 scheduler[2923]: Connection timeout to database host db-shard-3.prod.internal after 13970ms
<27>Mar 14 08:06:00 api01 checkout[8015]: Failed to acquire lock on resource /var/lib/app/cache.lock after 4531ms
<27>Mar 14 08:06:02 web03 worker[4390]: Payment gateway stripe.gateway.internal rejected transaction 59c3587b-abbf-45d6-97d7-6ad9e36b7a3b: insufficient_funds
<27>Mar 14 08:06:04 worker01 checkout[9827]: Connection timeout to database host db-replica-1.prod.internal after 24202ms
<27>Mar 14 08:06:06 worker01 checkout[5740]: Upstream search.svc.internal returned 503 for request 4f1417ac-a72f-4a37-adc1-5be289261c62
<27>Mar 14 08:06:09 web02 worker[7743]: Connection timeout to database host db-primary.prod.internal after 2286ms
<27>Mar 14 08:06:10 api02 worker[3174]: Connection timeout to database host db-shard-3.prod.internal after 16413ms
<27>Mar 14 08:06:12 web03 checkout[5055]: Upstream auth.svc.internal returned 504 for request ba500143-7ed9-43dd-a5b4-1ca9de0920cb
<27>Mar 14 08:06:15 edge01 worker[5402]: Connection timeout to database host db-primary.prod.internal after 3676ms
<27>Mar 14 08:06:17 worker02 scheduler[2237]: Connection timeout to database host db-primary.prod.internal after 24161ms
<27>Mar 14 08:06:18 worker02 checkout[7046]: Upstream billing.svc.internal returned 503 for request 6fdef03c-aff0-48ac-b936-a0262861eedf
<27>Mar 14 08:06:21 edge01 checkout[1040]: Connection timeout to database host db-replica-2.prod.internal after 20543ms
<27>Mar 14 08:06:22 edge01 checkout[2037]: Failed to acquire lock on resource /var/lib/app/queue.lock after 14627ms
<27>Mar 14 08:06:25 api01 scheduler[4470]: Upstream notifications.svc.internal returned 500 for request 5609eab2-842c-494f-8582-3fb2014a8d36
<27>Mar 14 08:06:27 api02 checkout[5646]: Connection timeout to database host db-replica-2.prod.internal after 6351ms
<27>Mar 14 08:06:29 web03 scheduler[6631]: Connection timeout to database host db-shard-3.prod.internal after 29536ms
<27>Mar 14 08:06:30 web01 api[8529]: Connection timeout to database host db-replica-2.prod.internal after 14746ms
<27>Mar 14 08:06:32 web02 worker[8230]: Connection timeout to database host db-replica-1.prod.internal after 12325ms
<27>Mar 14 08:06:35 web01 scheduler[4314]: Connection timeout to database host db-primary.prod.internal after 25493ms
<27>Mar 14 08:06:36 api02 worker[5489]: Upstream auth.svc.internal returned 504 for request c066cca3-fcbe-43b1-8bda-b5fe674ee7da
<27>Mar 14 08:06:38 worker01 checkout[2350]: Upstream notifications.svc.internal returned 503 for request f3532846-8c3f-4e7f-a8d7-3a90d74d1137
<27>Mar 14 08:06:41 web03 api[7493]: Failed to acquire lock on resource /var/lib/app/session.lock after 3535ms
<27>Mar 14 08:06:43 api01 scheduler[2494]: Connection timeout to database host db-shard-3.prod.internal after 3038ms
<27>Mar 14 08:06:44 web01 worker[8812]: Connection timeout to database host db-primary.prod.internal after 23694ms
<27>Mar 14 08:06:46 web03 worker[9510]: Connection timeout to database host db-primary.prod.internal after 28573ms
<27>Mar 14 08:06:49 web01 api[6629]: Upstream notifications.svc.internal returned 504 for request 9314f38f-d4a5-4c9f-a591-d7a65bb41482
<27>Mar 14 08:06:50 worker02 worker[8826]: Upstream search.svc.internal returned 500 for request 0002991e-bf7a-47a0-8d75-f564fc454577
<27>Mar 14 08:06:52 api01 scheduler[2463]: Upstream search.svc.internal returned 502 for request d3bfb230-2ef3-4fee-97ab-09eed7d479e9
<27>Mar 14 08:06:54 web02 checkout[7022]: Failed to acquire lock on resource /tmp/deploy.lock after 12813ms
<27>Mar 14 08:06:57 web03 scheduler[3181]: Connection timeout to database host db-primary.prod.internal after 19642ms
<27>Mar 14 08:06:58 web02 api[3688]: Upstream search.svc.internal returned 500 for request 5132afb0-686f-4e89-9c8c-af8a9816e82a
<27>Mar 14 08:07:01 worker01 worker[3480]: Upstream billing.svc.internal returned 500 for request c7d96d2d-caac-4a25-8b11-c5438ac9c9af
<27>Mar 14 08:07:03 edge01 worker[2420]: Connection timeout to database host db-shard-3.prod.internal after 19809ms
<27>Mar 14 08:07:04 web03 api[7291]: Upstream billing.svc.internal returned 504 for request 1dbb542f-37e4-4df8-b1ab-d789fccac2f1
<27>Mar 14 08:07:07 worker02 checkout[3290]: Connection timeout to database host db-replica-1.prod.internal after 8123ms
<27>Mar 14 08:07:08 api02 api[5065]: Connection timeout to database host db-replica-1.prod.internal after 29382ms
<27>Mar 14 08:07:11 web01 checkout[4366]: Upstream auth.svc.internal returned 502 for request 82896076-d3de-4081-9e0f-dcd20064749e
<27>Mar 14 08:07:12 worker02 checkout[1894]: Failed to acquire lock on resource /var/lib/app/cache.lock after 7132ms
<27>Mar 14 08:07:14 edge01 scheduler[2847]: Connection timeout to database host db-shard-3.prod.internal after 26787ms
<27>Mar 14 08:07:16 edge01 api[9655]: Failed to acquire lock on resource /tmp/deploy.lock after 4743ms
<27>Mar 14 08:07:18 worker01 checkout[3162]: Failed to acquire lock on resource /var/lib/app/queue.lock after 12042ms
<27>Mar 14 08:07:21 worker02 checkout[9951]: Connection timeout to database host db-primary.prod.internal after 15457ms
<27>Mar 14 08:07:22 edge01 api[1273]: Connection timeout to database host db-replica-1.prod.internal after 22445ms
<27>Mar 14 08:07:24 web01 checkout[4534]: Connection timeout to database host db-replica-1.prod.internal after 17076ms
<27>Mar 14 08:07:26 web01 scheduler[9300]: Connection timeout to database host db-shard-3.prod.internal after 23239ms
<27>Mar 14 08:07:29 worker02 scheduler[7590]: Connection timeout to database host db-replica-2.prod.internal after 11559ms
<27>Mar 14 08:07:30 web03 checkout[2369]: Upstream billing.svc.internal returned 503 for request 24fb86f5-9c78-46f3-a57b-d56a2cbfea48
<27>Mar 14 08:07:33 web02 worker[3545]: Connection timeout to database host db-replica-1.prod.internal after 25917ms
<27>Mar 14 08:07:35 worker02 api[6081]: Upstream search.svc.internal returned 504 for request 9789eae0-f02c-4c37-929d-662f1ab9e74d
<27>Mar 14 08:07:37 api01 api[5005]: Connection timeout to database host db-replica-2.prod.internal after 15881ms
<27>Mar 14 08:07:38 worker02 api[8428]: Connection timeout to database host db-replica-2.prod.internal after 15659ms
<27>Mar 14 08:07:40 api02 worker[2956]: Connection timeout to database host db-primary.prod.internal after 24411ms
<27>Mar 14 08:07:42 web03 api[3684]: Failed to acquire lock on resource /var/lib/app/queue.lock after 8147ms
<27>Mar 14 08:07:45 worker01 checkout[3735]: Upstream billing.svc.internal returned 502 for request 8a479fc9-adfa-4eba-9f0a-2dbad99f5962
<27>Mar 14 08:07:47 web03 checkout[2934]: Payment gateway adyen.gateway.internal rejected transaction 97c6d26b-8dd8-402b-970a-4db663a6713f: insufficient_funds
<27>Mar 14 08:07:48 worker01 scheduler[5456]: Connection timeout to database host db-shard-3.prod.internal after 27446ms
<27>Mar 14 08:07:50 api02 api[9128]: Upstream auth.svc.internal returned 502 for request a476a5d4-39f1-464c-9a5c-36345ec205b8
<27>Mar 14 08:07:53 worker01 api[9185]: Failed to acquire lock on resource /var/lib/app/queue.lock after 6736ms
<27>Mar 14 08:07:54 worker01 worker[5179]: Upstream auth.svc.internal returned 504 for request 930716ff-cca1-4dec-a5a4-3346e497fecc
<27>Mar 14 08:07:56 web02 worker[9336]: Connection timeout to database host db-replica-1.prod.internal after 24038ms
<27>Mar 14 08:07:58 web01 api[5006]: Connection timeout to database host db-replica-2.prod.internal after 2668ms
<27>Mar 14 08:08:00 edge01 checkout[7274]: Connection timeout to database host db-primary.prod.internal after 2014ms
<27>Mar 14 08:08:02 web03 api[7870]: Connection timeout to database host db-primary.prod.internal after 12263ms
<27>Mar 14 08:08:04 worker01 worker[7819]: Connection timeout to database host db-replica-1.prod.internal after 9242ms
<27>Mar 14 08:08:07 api02 api[7108]: Connection timeout to database host db-replica-2.prod.internal after 25685ms
<27>Mar 14 08:08:08 web01 worker[4656]: Upstream auth.svc.internal returned 504 for request c13a95b1-7e82-44ae-90e1-03f3cbbbf0ff
<27>Mar 14 08:08:10 worker02 api[8620]: Failed to acquire lock on resource /tmp/deploy.lock after 7532ms
<27>Mar 14 08:08:13 api02 checkout[2415]: Upstream notifications.svc.internal returned 500 for request bab780b4-2db7-41f3-a747-f663ec76b2fd
<27>Mar 14 08:08:15 web01 checkout[4287]: Connection timeout to database host db-primary.prod.internal after 8773ms
<27>Mar 14 08:08:17 api02 worker[8600]: Connection timeout to database host db-replica-1.prod.internal after 23647ms
<27>Mar 14 08:08:19 edge01 worker[9954]: Failed to acquire lock on resource /var/lib/app/queue.lock after 7636ms
<27>Mar 14 08:08:20 web03 api[7718]: Payment gateway adyen.gateway.internal rejected transaction b9b0fac3-ede0-46aa-8c68-f6e39333ba24: insufficient_funds
<27>Mar 14 08:08:22 api01 checkout[1157]: Connection timeout to database host db-replica-2.prod.internal after 8183ms
<27>Mar 14 08:08:25 web01 scheduler[6038]: Connection timeout to database host db-replica-2.prod.internal after 25485ms
<27>Mar 14 08:08:26 api01 worker[7878]: Connection timeout to database host db-replica-1.prod.internal after 18002ms
<27>Mar 14 08:08:29 web01 worker[5584]: Connection timeout to database host db-replica-2.prod.internal after 7761ms
<27>Mar 14 08:08:30 web01 worker[9515]: Connection timeout to database host db-replica-1.prod.internal after 19470ms
<27>Mar 14 08:08:33 worker01 scheduler[2391]: Failed to acquire lock on resource /var/lib/app/session.lock after 14225ms
<27>Mar 14 08:08:34 web03 scheduler[2226]: Connection timeout to database host db-replica-1.prod.internal after 10207ms
<27>Mar 14 08:08:36 worker02 worker[4879]: Upstream search.svc.internal returned 502 for request df09fb3c-dc21-4185-9822-e60ee072a930
<27>Mar 14 08:08:39 api02 checkout[7438]: Connection timeout to database host db-primary.prod.internal after 10457ms
<27>Mar 14 08:08:41 worker01 scheduler[5455]: Connection timeout to database host db-replica-1.prod.internal after 27257ms
<27>Mar 14 08:08:42 web02 worker[3230]: Connection timeout to database host db-shard-3.prod.internal after 7670ms
<27>Mar 14 08:08:44 web03 api[3693]: Upstream auth.svc.internal returned 504 for request cc8b3cf6-13f5-4f86-9334-11b73fe0b7c0
<27>Mar 14 08:08:47 edge01 checkout[7660]: Connection timeout to database host db-shard-3.prod.internal after 20043ms
<27>Mar 14 08:08:48 worker01 worker[8404]: Upstream auth.svc.internal returned 504 for request 15c0fa38-2b04-467a-b5a3-0b4c1c5ae4b9
<27>Mar 14 08:08:51 edge01 checkout[2964]: Connection timeout to database host db-replica-1.prod.internal after 22089ms
<27>Mar 14 08:08:52 web03 checkout[1660]: Connection timeout to database host db-primary.prod.internal after 13122ms
<27>Mar 14 08:08:54 worker01 worker[2494]: Failed to acquire lock on resource /var/lib/app/cache.lock after 11900ms
<27>Mar 14 08:08:56 worker01 scheduler[9412]: Connection timeout to database host db-replica-1.prod.internal after 8316ms
<27>Mar 14 08:08:58 api02 checkout[3848]: Failed to acquire lock on resource /var/lib/app/queue.lock after 10977ms
<27>Mar 14 08:09:00 worker02 scheduler[5093]: Connection timeout to database host db-replica-1.prod.internal after 15738ms
<27>Mar 14 08:09:02 edge01 api[3290]: Connection timeout to database host db-shard-3.prod.internal after 13856ms
<27>Mar 14 08:09:04 web01 api[4641]: Connection timeout to database host db-replica-1.prod.internal after 8257ms
<27>Mar 14 08:09:06 api01 scheduler[7098]: Connection timeout to database host db-replica-1.prod.internal after 26710ms
<27>Mar 14 08:09:08 api02 api[1357]: Upstream auth.svc.internal returned 502 for request 07b94ddf-8d3f-4d8b-8a12-5a7f5453ac84
<27>Mar 14 08:09:10 api01 scheduler[3110]: Upstream search.svc.internal returned 500 for request 06d4104c-acaa-4d01-8a7c-fb65314e2b2d
<27>Mar 14 08:09:12 worker01 api[1699]: Connection timeout to database host db-shard-3.prod.internal after 4946ms
<27>Mar 14 08:09:15 web01 worker[8096]: Connection timeout to database host db-replica-1.prod.internal after 27079ms
<27>Mar 14 08:09:17 edge01 api[7270]: Connection timeout to database host db-replica-2.prod.internal after 11505ms
<27>Mar 14 08:09:19 web03 checkout[4517]: Connection timeout to database host db-replica-1.prod.internal after 8077ms
<27>Mar 14 08:09:20 api02 checkout[8307]: Connection timeout to database host db-primary.prod.internal after 13108ms
<27>Mar 14 08:09:22 web01 worker[5835]: Upstream notifications.svc.internal returned 503 for request 54872d49-091b-4109-8cb7-04be27e3f1b8
<27>Mar 14 08:09:25 worker02 scheduler[1999]: Connection timeout to database host db-replica-1.prod.internal after 20996ms
<27>Mar 14 08:09:26 api02 scheduler[3248]: Upstream notifications.svc.internal returned 502 for request f404a979-18ab-4c41-8602-aa532bcb6fd1
<27>Mar 14 08:09:29 api01 worker[9907]: Upstream auth.svc.internal returned 502 for request bd7d6b97-45fa-4372-b7ec-30cf453a22c8
<27>Mar 14 08:09:30 worker02 checkout[8859]: Upstream notifications.svc.internal returned 504 for request 7eda99dd-894d-415e-a842-b49e0d731372
<27>Mar 14 08:09:33 worker01 checkout[7363]: Connection timeout to database host db-replica-1.prod.internal after 6786ms
<27>Mar 14 08:09:35 web03 api[6224]: Connection timeout to database host db-primary.prod.internal after 2486ms
<27>Mar 14 08:09:36 web01 checkout[1959]: Connection timeout to database host db-primary.prod.internal after 22838ms
<27>Mar 14 08:09:39 worker01 worker[3615]: Failed to acquire lock on resource /tmp/deploy.lock after 14631ms
<27>Mar 14 08:09:40 api01 worker[5366]: Connection timeout to database host db-replica-1.prod.internal after 21803ms
<27>Mar 14 08:09:42 api01 scheduler[6813]: Connection timeout to database host db-shard-3.prod.internal after 26960ms
<27>Mar 14 08:09:44 web03 worker[6210]: Connection timeout to database host db-replica-2.prod.internal after 15736ms
<27>Mar 14 08:09:47 web01 checkout[3328]: Connection timeout to database host db-shard-3.prod.internal after 22176ms
<27>Mar 14 08:09:48 edge01 checkout[3876]: Failed to acquire lock on resource /tmp/deploy.lock after 9584ms
<27>Mar 14 08:09:51 web01 api[1920]: Connection timeout to database host db-replica-2.prod.internal after 28079ms
<27>Mar 14 08:09:52 api01 api[9625]: Upstream auth.svc.internal returned 503 for request f2e9fa92-de08-4d5c-9b28-bd7dd23e69c7
<27>Mar 14 08:09:55 web01 checkout[4300]: Connection timeout to database host db-shard-3.prod.internal after 16977ms
<27>Mar 14 08:09:56 worker01 worker[6301]: Connection timeout to database host db-replica-1.prod.internal after 21997ms
<27>Mar 14 08:09:58 web01 worker[3320]: Failed to acquire lock on resource /var/lib/app/session.lock after 14128ms
<27>Mar 14 08:10:00 worker01 api[7120]: Connection timeout to database host db-replica-1.prod.internal after 28892ms
<27>Mar 14 08:10:03 web02 checkout[5944]: Payment gateway stripe.gateway.internal rejected transaction d79457e0-ba2e-4a3e-838c-04e7c84ef877: insufficient_funds
<27>Mar 14 08:10:05 web03 worker[8157]: Connection timeout to database host db-replica-2.prod.internal after 15482ms
<27>Mar 14 08:10:07 worker01 worker[6846]: Upstream search.svc.internal returned 502 for request 955a7735-f999-42aa-a92e-777b97138da6
<27>Mar 14 08:10:08 web03 scheduler[6026]: Connection timeout to database host db-shard-3.prod.internal after 9617ms
<27>Mar 14 08:10:11 web03 worker[1029]: Connection timeout to database host db-replica-2.prod.internal after 8346ms
<27>Mar 14 08:10:13 web01 worker[6213]: Payment gateway stripe.gateway.internal rejected transaction 4cbf1a24-b578-4bf5-8008-208e12cceb15: insufficient_funds
<27>Mar 14 08:10:14 web02 scheduler[3304]: Connection timeout to database host db-replica-2.prod.internal after 12141ms
<27>Mar 14 08:10:17 web02 worker[6682]: Upstream billing.svc.internal returned 502 for request c89406f2-a172-4acd-961e-bbbe3e8f104f
<27>Mar 14 08:10:19 web03 api[6500]: Connection timeout to database host db-primary.prod.internal after 28716ms
<27>Mar 14 08:10:21 web01 checkout[4435]: Connection timeout to database host db-shard-3.prod.internal after 20367ms
<27>Mar 14 08:10:22 web02 checkout[5205]: Connection timeout to database host db-replica-2.prod.internal after 18666ms
<27>Mar 14 08:10:24 web01 api[7323]: Failed to acquire lock on resource /tmp/deploy.lock after 14836ms
<27>Mar 14 08:10:27 worker02 worker[7783]: Connection timeout to database host db-replica-2.prod.internal after 2821ms
<27>Mar 14 08:10:29 worker02 checkout[9946]: Failed to acquire lock on resource /tmp/deploy.lock after 12671ms
<27>Mar 14 08:10:31 web02 scheduler[4695]: Connection timeout to database host db-replica-1.prod.internal after 5949ms
<27>Mar 14 08:10:32 edge01 api[1547]: Failed to acquire lock on resource /var/lib/app/queue.lock after 6321ms
<27>Mar 14 08:10:35 web01 checkout[5278]: Connection timeout to database host db-replica-1.prod.internal after 5032ms
<27>Mar 14 08:10:36 web02 checkout[3787]: Connection timeout to database host db-replica-2.prod.internal after 25248ms
<27>Mar 14 08:10:39 web03 api[9962]: Upstream search.svc.internal returned 503 for request f0f4ceab-592a-4114-9805-6982bba2514f
<27>Mar 14 08:10:40 worker01 scheduler[1507]: Failed to acquire lock on resource /var/lib/app/queue.lock after 9149ms
<27>Mar 14 08:10:43 web03 scheduler[3573]: Connection timeout to database host db-replica-1.prod.internal after 20669ms
<27>Mar 14 08:10:44 web02 checkout[4367]: Failed to acquire lock on resource /tmp/deploy.lock after 12953ms
<27>Mar 14 08:10:46 worker02 checkout[5612]: Connection timeout to database host db-primary.prod.internal after 25364ms
<27>Mar 14 08:10:49 api01 api[1813]: Failed to acquire lock on resource /var/lib/app/cache.lock after 2180ms
<27>Mar 14 08:10:51 edge01 worker[1893]: Upstream billing.svc.internal returned 503 for request b6cf653e-c149-4afa-a3ed-623668925988
<27>Mar 14 08:10:53 web01 scheduler[2435]: Upstream billing.svc.internal returned 502 for request 00ed421d-3c55-405d-a8d1-e94ace6f94b3
<27>Mar 14 08:10:55 edge01 worker[2589]: Connection timeout to database host db-replica-2.prod.internal after 8321ms
<27>Mar 14 08:10:56 api01 worker[5781]: Failed to acquire lock on resource /var/lib/app/cache.lock after 1100ms
<27>Mar 14 08:10:58 edge01 api[6052]: Upstream search.svc.internal returned 502 for request f2e74613-bba1-43b8-ae52-81b3a3b0048e
<27>Mar 14 08:11:01 edge01 scheduler[5341]: Connection timeout to database host db-replica-1.prod.internal after 16069ms
<27>Mar 14 08:11:02 worker02 worker[7078]: Connection timeout to database host db-primary.prod.internal after 22791ms
<27>Mar 14 08:11:05 worker01 worker[8303]: Failed to acquire lock on resource /var/lib/app/session.lock after 5477ms
<27>Mar 14 08:11:06 api02 scheduler[8619]: Failed to acquire lock on resource /var/lib/app/queue.lock after 6655ms
<27>Mar 14 08:11:09 api01 checkout[2009]: Failed to acquire lock on resource /var/lib/app/session.lock after 5938ms
<27>Mar 14 08:11:10 web02 api[6632]: Connection timeout to database host db-replica-2.prod.internal after 1944ms
<27>Mar 14 08:11:13 api01 api[7342]: Connection timeout to database host db-shard-3.prod.internal after 27676ms
<27>Mar 14 08:11:15 edge01 worker[8803]: Connection timeout to database host db-shard-3.prod.internal after 22766ms
<27>Mar 14 08:11:17 web02 scheduler[1568]: Failed to acquire lock on resource /tmp/deploy.lock after 3878ms
<27>Mar 14 08:11:18 web02 scheduler[3832]: Connection timeout to database host db-shard-3.prod.internal after 14124ms
<27>Mar 14 08:11:20 api01 worker[3067]: Upstream notifications.svc.internal returned 500 for request aa0d1954-2b12-4a9c-b856-61a7a0c6565b
<27>Mar 14 08:11:22 worker02 api[8133]: Failed to acquire lock on resource /var/lib/app/session.lock after 10115ms
<27>Mar 14 08:11:24 web03 checkout[6238]: Failed to acquire lock on resource /var/lib/app/cache.lock after 6761ms
<27>Mar 14 08:11:26 web02 scheduler[7510]: Upstream auth.svc.internal returned 500 for request b5ef61aa-a940-406b-aabf-c71fce501fd0
<27>Mar 14 08:11:29 api02 scheduler[1202]: Connection timeout to database host db-shard-3.prod.internal after 8728ms
<27>Mar 14 08:11:31 web02 scheduler[3451]: Payment gateway braintree.gateway.internal rejected transaction 21911543-4ce7-405b-9abf-f224726972b9: insufficient_funds
<27>Mar 14 08:11:32 web02 api[2654]: Connection timeout to database host db-primary.prod.internal after 22564ms
<27>Mar 14 08:11:35 worker01 scheduler[5327]: Upstream notifications.svc.internal returned 503 for request bc6f1aa4-79b0-4e91-a834-b8a1e417e74e
<27>Mar 14 08:11:37 edge01 scheduler[1613]: Upstream billing.svc.internal returned 504 for request f84dbae5-0380-42e0-8dde-6a4349905afc
<27>Mar 14 08:11:38 api02 scheduler[3356]: Connection timeout to database host db-replica-1.prod.internal after 25518ms
<27>Mar 14 08:11:41 web01 scheduler[6431]: Connection timeout to database host db-replica-1.prod.internal after 7804ms
<27>Mar 14 08:11:42 web01 scheduler[5400]: Upstream billing.svc.internal returned 503 for request c4ac3f08-f0ae-4b9b-bb8f-d08073aac635
<27>Mar 14 08:11:45 web01 checkout[5965]: Connection timeout to database host db-shard-3.prod.internal after 13262ms
<27>Mar 14 08:11:47 api02 scheduler[2609]: Payment gateway braintree.gateway.internal rejected transaction 64f38ed7-9e15-4e31-8dda-814acc8be92f: insufficient_funds
<27>Mar 14 08:11:48 web03 checkout[8259]: Connection timeout to database host db-shard-3.prod.internal after 11649ms
<27>Mar 14 08:11:51 api02 scheduler[2462]: Upstream auth.svc.internal returned 500 for request 80992d02-878c-48e5-98f9-401a3af99533
<27>Mar 14 08:11:52 edge01 worker[7671]: Connection timeout to database host db-primary.prod.internal after 18980ms
<27>Mar 14 08:11:55 worker01 api[9442]: Upstream auth.svc.internal returned 504 for request 8d7626b0-3a41-4808-83d1-91fef06ac8de
<27>Mar 14 08:11:56 web02 checkout[9303]: Connection timeout to database host db-replica-2.prod.internal after 17061ms
<27>Mar 14 08:11:59 web03 api[3846]: Connection timeout to database host db-primary.prod.internal after 12089ms
<27>Mar 14 08:12:00 api01 api[5087]: Upstream auth.svc.internal returned 504 for request e8d63c57-89f6-4fd9-bf49-b1ece6ae5aa0
<27>Mar 14 08:12:02 edge01 api[6904]: Failed to acquire lock on resource /var/lib/app/session.lock after 13520ms
<27>Mar 14 08:12:04 api02 checkout[8605]: Upstream auth.svc.internal returned 503 for request 306dfd11-ceee-4a0b-8974-a1288f23c3b8
<27>Mar 14 08:12:07 web02 api[4963]: Connection timeout to database host db-replica-1.prod.internal after 23587ms
<27>Mar 14 08:12:09 edge01 scheduler[3909]: Connection timeout to database host db-primary.prod.internal after 3250ms
<27>Mar 14 08:12:11 worker01 checkout[3679]: Upstream billing.svc.internal returned 504 for request 08fbbf30-a5a8-4e1a-a6f5-242aaa374b30
<27>Mar 14 08:12:13 web02 checkout[1449]: Connection timeout to database host db-shard-3.prod.internal after 11957ms
<27>Mar 14 08:12:15 web01 worker[4511]: Upstream search.svc.internal returned 503 for request 94a4960f-7a2b-4473-a5f8-3d61c4de6e54
<27>Mar 14 08:12:16 api01 worker[5408]: Upstream auth.svc.internal returned 504 for request 9bf5e99f-2b6c-49f7-8867-53d9b3299808
<27>Mar 14 08:12:19 web01 api[8767]: Failed to acquire lock on resource /tmp/deploy.lock after 8432ms
<27>Mar 14 08:12:20 web03 api[4666]: Connection timeout to database host db-shard-3.prod.internal after 11523ms
<27>Mar 14 08:12:23 api02 checkout[7943]: Connection timeout to database host db-replica-1.prod.internal after 9166ms
<27>Mar 14 08:12:25 worker01 scheduler[5128]: Upstream billing.svc.internal returned 502 for request 743037fd-72f8-4fe8-b1f8-4f76d2be179b
<27>Mar 14 08:12:26 edge01 checkout[9530]: Connection timeout to database host db-primary.prod.internal after 19462ms
<27>Mar 14 08:12:29 web02 api[9380]: Upstream billing.svc.internal returned 502 for request 45397e82-3bcc-4efa-bfec-229b317c42ba
<27>Mar 14 08:12:31 api02 checkout[2771]: Connection timeout to database host db-primary.prod.internal after 13035ms
<27>Mar 14 08:12:33 web02 checkout[5529]: Upstream search.svc.internal returned 502 for request a51c9320-c6c9-4581-a376-9772a28e831e
<27>Mar 14 08:12:35 api02 checkout[5895]: Connection timeout to database host db-shard-3.prod.internal after 16509ms
<27>Mar 14 08:12:36 web03 worker[1930]: Connection timeout to database host db-primary.prod.internal after 25840ms
<27>Mar 14 08:12:39 worker01 worker[1417]: Payment gateway braintree.gateway.internal rejected transaction 270fa0ce-a41b-414d-bc43-9f6f2261213f: insufficient_funds
<27>Mar 14 08:12:41 api02 checkout[7857]: Connection timeout to database host db-replica-1.prod.internal after 2745ms
<27>Mar 14 08:12:43 worker02 api[4111]: Connection timeout to database host db-primary.prod.internal after 11967ms
<27>Mar 14 08:12:45 api01 scheduler[6873]: Connection timeout to database host db-primary.prod.internal after 2900ms
<27>Mar 14 08:12:47 web03 worker[5813]: Upstream auth.svc.internal returned 504 for request c90a19e7-0540-4157-bfe9-bee943e68676
<27>Mar 14 08:12:48 edge01 worker[5237]: Connection timeout to database host db-replica-2.prod.internal after 7467ms
<27>Mar 14 08:12:50 worker01 api[4641]: Upstream notifications.svc.internal returned 502 for request a94c7697-4366-4e15-839a-fe53beaf50be
<27>Mar 14 08:12:53 api01 checkout[7889]: Connection timeout to database host db-shard-3.prod.internal after 10506ms
<27>Mar 14 08:12:55 worker01 checkout[5817]: Failed to acquire lock on resource /var/lib/app/session.lock after 7475ms
<27>Mar 14 08:12:57 web02 scheduler[1879]: Payment gateway stripe.gateway.internal rejected transaction 08c06227-0a05-46e5-9b9c-4aff668931a6: insufficient_funds
<27>Mar 14 08:12:58 edge01 worker[9199]: Failed to acquire lock on resource /tmp/deploy.lock after 2204ms
<27>Mar 14 08:13:00 worker02 checkout[7768]: Upstream notifications.svc.internal returned 500 for request 09ae958d-97e2-4d38-bac6-4832b942f4ad
<27>Mar 14 08:13:02 web03 worker[4281]: Payment gateway braintree.gateway.internal rejected transaction 638b570b-020d-4ff9-af58-25d1c520537c: insufficient_funds
<27>Mar 14 08:13:05 worker02 scheduler[3346]: Connection timeout to database host db-replica-1.prod.internal after 26300ms
<27>Mar 14 08:13:07 web03 api[8988]: Failed to acquire lock on resource /tmp/deploy.lock after 14847ms
<27>Mar 14 08:13:09 web03 checkout[1950]: Failed to acquire lock on resource /tmp/deploy.lock after 9633ms
<27>Mar 14 08:13:10 edge01 checkout[4430]: Connection timeout to database host db-shard-3.prod.internal after 2421ms
<27>Mar 14 08:13:12 web03 scheduler[2442]: Connection timeout to database host db-primary.prod.internal after 9799ms
<27>Mar 14 08:13:14 worker02 scheduler[7609]: Connection timeout to database host db-replica-2.prod.internal after 3794ms
<27>Mar 14 08:13:17 edge01 scheduler[1076]: Connection timeout to database host db-shard-3.prod.internal after 9377ms
<27>Mar 14 08:13:18 api01 checkout[1230]: Connection timeout to database host db-primary.prod.internal after 10722ms
<27>Mar 14 08:13:21 api01 api[1067]: Connection timeout to database host db-shard-3.prod.internal after 22694ms
<27>Mar 14 08:13:22 api01 checkout[6113]: Upstream billing.svc.internal returned 504 for request 39fe7514-2d67-4b39-aac4-32016e9ea7b8
<27>Mar 14 08:13:24 web02 scheduler[9246]: Upstream billing.svc.internal returned 503 for request 1bc14aa6-b9f2-44bf-a213-7823e960e0ce
<27>Mar 14 08:13:27 web03 api[1749]: Connection timeout to database host db-primary.prod.internal after 29437ms
<27>Mar 14 08:13:29 api02 scheduler[9524]: Connection timeout to database host db-primary.prod.internal after 22127ms
<27>Mar 14 08:13:31 worker02 worker[6615]: Upstream billing.svc.internal returned 502 for request 42d2a76a-c91c-4e54-9f80-230131fe69db
<27>Mar 14 08:13:33 worker01 worker[3701]: Upstream search.svc.internal returned 503 for request abda2bb0-14a4-47d4-b9ef-9bb07b04ecc1
<27>Mar 14 08:13:35 web01 worker[4675]: Connection timeout to database host db-replica-2.prod.internal after 27387ms
<27>Mar 14 08:13:36 api01 api[9267]: Connection timeout to database host db-primary.prod.internal after 3524ms
<27>Mar 14 08:13:38 web02 checkout[1612]: Failed to acquire lock on resource /var/lib/app/session.lock after 3579ms
<27>Mar 14 08:13:41 api02 api[1785]: Connection timeout to database host db-replica-2.prod.internal after 26192ms
<27>Mar 14 08:13:42 web01 api[2897]: Upstream auth.svc.internal returned 502 for request ff83aa15-ff47-4f69-9f00-75b698d20cac
<27>Mar 14 08:13:45 edge01 scheduler[3024]: Connection timeout to database host db-shard-3.prod.internal after 8766ms
<27>Mar 14 08:13:47 edge01 worker[4366]: Connection timeout to database host db-shard-3.prod.internal after 18071ms
<27>Mar 14 08:13:48 api02 scheduler[2654]: Connection timeout to database host db-replica-2.prod.internal after 12028ms
<27>Mar 14 08:13:51 web03 scheduler[2718]: Upstream billing.svc.internal returned 503 for request eed7da80-0fd9-4266-a127-f12f28a27d6d
<27>Mar 14 08:13:52 api02 api[2741]: Connection timeout to database host db-replica-2.prod.internal after 2373ms
<27>Mar 14 08:13:55 api01 worker[5308]: Connection timeout to database host db-replica-1.prod.internal after 23861ms
<27>Mar 14 08:13:56 worker01 scheduler[5839]: Connection timeout to database host db-shard-3.prod.internal after 11226ms
<27>Mar 14 08:13:58 web01 scheduler[4402]: Upstream auth.svc.internal returned 502 for request 9f2863ef-7baa-4871-9281-6347a090b908
<27>Mar 14 08:14:01 worker01 worker[1044]: Connection timeout to database host db-primary.prod.internal after 20476ms
<27>Mar 14 08:14:02 web02 api[9095]: Upstream auth.svc.internal returned 504 for request 19ae6020-526a-47e9-9b77-97057907d3eb
<27>Mar 14 08:14:04 web02 api[2659]: Connection timeout to database host db-replica-1.prod.internal after 12434ms
<27>Mar 14 08:14:07 api01 scheduler[5850]: Failed to acquire lock on resource /tmp/deploy.lock after 11845ms
<27>Mar 14 08:14:09 edge01 api[2635]: Connection timeout to database host db-shard-3.prod.internal after 3861ms
<27>Mar 14 08:14:10 web01 checkout[6901]: Connection timeout to database host db-replica-2.prod.internal after 12448ms
<27>Mar 14 08:14:13 worker02 api[2556]: Upstream auth.svc.internal returned 503 for request 6db4af89-4480-4e00-b62e-fecb05fe95b8
<27>Mar 14 08:14:14 web01 worker[3793]: Upstream billing.svc.internal returned 503 for request a5059c9c-9ea8-45c1-adb8-f42ac7441ab7
<27>Mar 14 08:14:17 worker01 scheduler[5395]: Connection timeout to database host db-replica-2.prod.internal after 13469ms
<27>Mar 14 08:14:18 worker01 checkout[3996]: Connection timeout to database host db-replica-2.prod.internal after 20282ms
<27>Mar 14 08:14:20 worker02 scheduler[8479]: Connection timeout to database host db-primary.prod.internal after 11578ms
<27>Mar 14 08:14:23 worker02 scheduler[7909]: Upstream search.svc.internal returned 503 for request e536ef08-4f5f-4e10-8e05-747ebee26c68
<27>Mar 14 08:14:24 web02 worker[1960]: Upstream search.svc.internal returned 504 for request 232072e3-dea2-4a9e-8100-872a27fd8f93
<27>Mar 14 08:14:26 web02 scheduler[8908]: Upstream billing.svc.internal returned 500 for request 92f2e60b-929f-4806-b9e3-881a16e80ee1
<27>Mar 14 08:14:29 web03 scheduler[6875]: Upstream search.svc.internal returned 504 for request f2bdba20-0c5b-4b9f-a9e9-4d1cae9cd22b
<27>Mar 14 08:14:30 web03 scheduler[8997]: Failed to acquire lock on resource /var/lib/app/cache.lock after 1474ms
<27>Mar 14 08:14:32 worker02 scheduler[1753]: Connection timeout to database host db-replica-2.prod.internal after 1753ms
<27>Mar 14 08:14:34 api02 checkout[1202]: Connection timeout to database host db-replica-2.prod.internal after 16007ms
<27>Mar 14 08:14:36 worker01 api[8091]: Connection timeout to database host db-shard-3.prod.internal after 12368ms
<27>Mar 14 08:14:38 edge01 checkout[7305]: Connection timeout to database host db-replica-2.prod.internal after 3114ms
<27>Mar 14 08:14:40 web01 api[3571]: Failed to acquire lock on resource /tmp/deploy.lock after 10648ms
<27>Mar 14 08:14:43 edge01 worker[2226]: Connection timeout to database host db-replica-2.prod.internal after 23352ms
<27>Mar 14 08:14:45 worker01 worker[6421]: Upstream search.svc.internal returned 502 for request 25e33436-a003-4101-a20d-e245741450cb
<27>Mar 14 08:14:47 web01 worker[7203]: Upstream notifications.svc.internal returned 504 for request 9688b99f-c672-4d86-9b2d-d94e9f05922e
<27>Mar 14 08:14:49 api01 api[8521]: Connection timeout to database host db-replica-2.prod.internal after 22237ms
<27>Mar 14 08:14:51 edge01 api[9166]: Connection timeout to database host db-replica-1.prod.internal after 2990ms
<27>Mar 14 08:14:53 worker01 api[9148]: Connection timeout to database host db-replica-2.prod.internal after 14799ms
<27>Mar 14 08:14:54 web02 scheduler[7755]: Connection timeout to database host db-replica-2.prod.internal after 5998ms
<27>Mar 14 08:14:56 web01 api[5990]: Connection timeout to database host db-replica-2.prod.internal after 14442ms
<27>Mar 14 08:14:58 edge01 checkout[6953]: Connection timeout to database host db-shard-3.prod.internal after 2927ms
<27>Mar 14 08:15:00 api01 scheduler[6439]: Connection timeout to database host db-primary.prod.internal after 8392ms
<27>Mar 14 08:15:03 worker01 scheduler[2868]: Failed to acquire lock on resource /tmp/deploy.lock after 7785ms
<27>Mar 14 08:15:04 api01 api[6592]: Connection timeout to database host db-replica-1.prod.internal after 21401ms
<27>Mar 14 08:15:07 api02 checkout[2741]: Connection timeout to database host db-replica-2.prod.internal after 5669ms
<27>Mar 14 08:15:08 web03 scheduler[3301]: Upstream search.svc.internal returned 503 for request 2d519397-dde4-45d6-b9b5-6d26618edc80
<27>Mar 14 08:15:11 web02 checkout[4209]: Upstream search.svc.internal returned 500 for request 6986a976-0807-4031-b900-2ff658ff1d46
<27>Mar 14 08:15:12 web02 scheduler[4532]: Upstream auth.svc.internal returned 504 for request a267398f-a11d-4f50-9d79-f4e38919fc0c
<27>Mar 14 08:15:15 api01 checkout[2765]: Connection timeout to database host db-shard-3.prod.internal after 17838ms
<27>Mar 14 08:15:17 web01 api[8381]: Connection timeout to database host db-replica-2.prod.internal after 29364ms
<27>Mar 14 08:15:19 web03 scheduler[1071]: Connection timeout to database host db-replica-1.prod.internal after 3305ms
<27>Mar 14 08:15:20 worker02 scheduler[9328]: Failed to acquire lock on resource /tmp/deploy.lock after 2816ms
<27>Mar 14 08:15:22 web03 worker[2696]: Connection timeout to database host db-replica-1.prod.internal after 9498ms
<27>Mar 14 08:15:25 worker01 scheduler[7859]: Upstream auth.svc.internal returned 504 for request d72a32ef-ee63-4a59-8b3a-8c825cb7052a
<27>Mar 14 08:15:26 api02 scheduler[6617]: Failed to acquire lock on resource /var/lib/app/queue.lock after 14972ms
<27>Mar 14 08:15:29 edge01 worker[3263]: Connection timeout to database host db-primary.prod.internal after 13242ms
<27>Mar 14 08:15:31 api02 checkout[4560]: Failed to acquire lock on resource /var/lib/app/cache.lock after 8433ms
<27>Mar 14 08:15:32 api01 checkout[6657]: Connection timeout to database host db-primary.prod.internal after 9639ms
<27>Mar 14 08:15:34 web03 checkout[2890]: Upstream notifications.svc.internal returned 503 for request d158365d-0de1-4f68-a986-aec05c4691b6
<27>Mar 14 08:15:37 edge01 scheduler[9747]: Failed to acquire lock on resource /var/lib/app/queue.lock after 2503ms
<27>Mar 14 08:15:39 worker02 api[9564]: Upstream auth.svc.internal returned 504 for request d687d670-2ee6-496c-b6e1-d845f883177b
<27>Mar 14 08:15:40 web01 scheduler[3381]: Connection timeout to database host db-replica-1.prod.internal after 10839ms
<27>Mar 14 08:15:42 web01 worker[5374]: Connection timeout to database host db-replica-1.prod.internal after 9424ms
<27>Mar 14 08:15:44 api02 checkout[3399]: Connection timeout to database host db-primary.prod.internal after 18890ms
<27>Mar 14 08:15:46 api02 worker[9866]: Upstream notifications.svc.internal returned 500 for request cafd302d-0fe6-48fc-a56e-1da0abdb3c6e
<27>Mar 14 08:15:48 worker01 scheduler[2679]: Connection timeout to database host db-replica-1.prod.internal after 29155ms
<27>Mar 14 08:15:50 worker02 worker[9370]: Connection timeout to database host db-primary.prod.internal after 11429ms
<27>Mar 14 08:15:53 web01 scheduler[7349]: Upstream billing.svc.internal returned 502 for request b094bf0f-7e15-4610-8355-fd18705a269e
<27>Mar 14 08:15:55 worker01 api[1212]: Connection timeout to database host db-shard-3.prod.internal after 4682ms
<27>Mar 14 08:15:57 api01 scheduler[4695]: Connection timeout to database host db-replica-2.prod.internal after 26947ms
<27>Mar 14 08:15:59 api01 worker[5819]: Connection timeout to database host db-primary.prod.internal after 23725ms
<27>Mar 14 08:16:01 web03 api[4063]: Upstream auth.svc.internal returned 500 for request 2ac45cc2-c200-449d-9b4a-7840389934a1
<27>Mar 14 08:16:02 web02 scheduler[1665]: Connection timeout to database host db-primary.prod.internal after 27848ms
<27>Mar 14 08:16:05 api02 checkout[3131]: Connection timeout to database host db-shard-3.prod.internal after 7723ms
<27>Mar 14 08:16:06 api02 api[6415]: Connection timeout to database host db-replica-1.prod.internal after 28298ms
<27>Mar 14 08:16:09 web03 api[3518]: Upstream auth.svc.internal returned 502 for request fc2b9c5d-7cd2-4f1e-97e6-3470ddd837dd
<27>Mar 14 08:16:11 web01 checkout[9037]: Connection timeout to database host db-replica-1.prod.internal after 4453ms
<27>Mar 14 08:16:13 web02 checkout[9608]: Connection timeout to database host db-shard-3.prod.internal after 26247ms
<27>Mar 14 08:16:14 web03 api[9276]: Connection timeout to database host db-primary.prod.internal after 29750ms
<27>Mar 14 08:16:16 worker02 scheduler[2506]: Connection timeout to database host db-replica-1.prod.internal after 5328ms
<27>Mar 14 08:16:19 api01 worker[6437]: Upstream auth.svc.internal returned 504 for request 9683e291-dfea-4360-b679-db210e2236aa
<27>Mar 14 08:16:21 worker01 checkout[4579]: Connection timeout to database host db-primary.prod.internal after 12955ms
<27>Mar 14 08:16:23 web01 api[6668]: Upstream search.svc.internal returned 504 for request a721b68b-2844-4a21-a989-cccf90611ca7
<27>Mar 14 08:16:25 api01 worker[1653]: Connection timeout to database host db-replica-2.prod.internal after 7861ms
<27>Mar 14 08:16:27 worker02 api[7322]: Connection timeout to database host db-shard-3.prod.internal after 12933ms
<27>Mar 14 08:16:29 api02 api[9468]: Failed to acquire lock on resource /var/lib/app/queue.lock after 5712ms
<27>Mar 14 08:16:31 edge01 scheduler[1718]: Connection timeout to database host db-primary.prod.internal after 26099ms
<27>Mar 14 08:16:33 api01 checkout[9419]: Failed to acquire lock on resource /var/lib/app/session.lock after 2076ms
<27>Mar 14 08:16:34 web02 worker[9814]: Connection timeout to database host db-replica-1.prod.internal after 24635ms
<27>Mar 14 08:16:36 web02 worker[4492]: Connection timeout to database host db-shard-3.prod.internal after 15247ms
<27>Mar 14 08:16:39 web02 worker[9810]: Upstream billing.svc.internal returned 504 for request 33fa4e24-2485-48eb-a955-0759c03b4bc3
<27>Mar 14 08:16:41 web02 worker[1389]: Connection timeout to database host db-primary.prod.internal after 29654ms
<27>Mar 14 08:16:43 worker01 api[2433]: Connection timeout to database host db-primary.prod.internal after 12026ms
<27>Mar 14 08:16:44 web01 worker[5127]: Connection timeout to database host db-primary.prod.internal after 1319ms
<27>Mar 14 08:16:46 api01 checkout[7678]: Upstream billing.svc.internal returned 500 for request 5a448e2d-fc31-45f0-a27c-4c79b805670e
<27>Mar 14 08:16:49 edge01 scheduler[8094]: Upstream billing.svc.internal returned 502 for request 6bf0f56a-751d-4f16-aee1-8aed62c070ac
<27>Mar 14 08:16:51 web01 scheduler[2938]: Connection timeout to database host db-replica-2.prod.internal after 4155ms
<27>Mar 14 08:16:52 web02 api[2189]: Connection timeout to database host db-shard-3.prod.internal after 23677ms
<27>Mar 14 08:16:55 web02 checkout[5937]: Connection timeout to database host db-replica-2.prod.internal after 26733ms
<27>Mar 14 08:16:56 api02 scheduler[7249]: Connection timeout to database host db-replica-1.prod.internal after 28020ms
<27>Mar 14 08:16:59 web01 scheduler[3561]: Connection timeout to database host db-shard-3.prod.internal after 26509ms
<27>Mar 14 08:17:00 web03 scheduler[9941]: Payment gateway adyen.gateway.internal rejected transaction 8cebb2c6-7356-4451-a528-2e89d881bf4d: insufficient_funds
<27>Mar 14 08:17:03 web03 checkout[7113]: Connection timeout to database host db-replica-2.prod.internal after 5594ms
<27>Mar 14 08:17:04 web03 api[1684]: Upstream notifications.svc.internal returned 502 for request d0457569-8096-4464-bb60-a88161b61d5e
<27>Mar 14 08:17:06 worker02 api[5992]: Upstream auth.svc.internal returned 500 for request b91bfd79-622d-4851-9a7a-b7617569a9e3
<27>Mar 14 08:17:08 worker01 worker[7760]: Connection timeout to database host db-replica-1.prod.internal after 26454ms
<27>Mar 14 08:17:10 web03 worker[4617]: Upstream notifications.svc.internal returned 503 for request 22ccd402-3083-44c7-9660-0112c500700e
<27>Mar 14 08:17:13 api01 api[3971]: Connection timeout to database host db-replica-1.prod.internal after 4381ms
<27>Mar 14 08:17:14 edge01 api[5762]: Upstream billing.svc.internal returned 504 for request ebcaa579-00f1-4ccf-9f64-07993c3dd4f1
<27>Mar 14 08:17:16 api02 api[4234]: Connection timeout to database host db-primary.prod.internal after 26879ms
<27>Mar 14 08:17:19 worker01 checkout[3100]: Connection timeout to database host db-primary.prod.internal after 8647ms
<27>Mar 14 08:17:20 web02 checkout[2143]: Connection timeout to database host db-replica-2.prod.internal after 18280ms
<27>Mar 14 08:17:22 worker01 scheduler[8383]: Upstream search.svc.internal returned 500 for request fa4e872a-9ce8-4e8f-bdb4-745490f5cc6d
<27>Mar 14 08:17:25 api02 worker[6971]: Connection timeout to database host db-replica-2.prod.internal after 25594ms
<27>Mar 14 08:17:27 worker02 worker[1079]: Connection timeout to database host db-shard-3.prod.internal after 8462ms
<27>Mar 14 08:17:29 api01 worker[8145]: Failed to acquire lock on resource /var/lib/app/queue.lock after 6277ms
<27>Mar 14 08:17:31 worker01 worker[6432]: Failed to acquire lock on resource /var/lib/app/queue.lock after 2527ms
<27>Mar 14 08:17:33 edge01 scheduler[6877]: Connection timeout to database host db-shard-3.prod.internal after 7788ms
<27>Mar 14 08:17:35 edge01 api[3879]: Payment gateway stripe.gateway.internal rejected transaction 627cb8f0-64e5-4851-a8ad-6f915a155d47: insufficient_funds
<27>Mar 14 08:17:36 api02 worker[9824]: Connection timeout to database host db-primary.prod.internal after 12949ms
<27>Mar 14 08:17:38 api02 api[2235]: Connection timeout to database host db-replica-1.prod.internal after 5508ms
<27>Mar 14 08:17:40 web01 scheduler[4485]: Connection timeout to database host db-shard-3.prod.internal after 1541ms
<27>Mar 14 08:17:42 api02 scheduler[3435]: Upstream auth.svc.internal returned 503 for request 3887d508-7f92-4439-84de-a76c31991226
<27>Mar 14 08:17:45 worker02 worker[5767]: Upstream notifications.svc.internal returned 504 for request c930dd0f-40c0-4b64-ab0f-68e11a80db14
<27>Mar 14 08:17:46 edge01 api[7500]: Connection timeout to database host db-shard-3.prod.internal after 5572ms
<27>Mar 14 08:17:49 web01 worker[3717]: Upstream search.svc.internal returned 502 for request a0113f82-125d-4ea6-a8d6-a727266b3012
<27>Mar 14 08:17:50 edge01 worker[3935]: Failed to acquire lock on resource /var/lib/app/session.lock after 7229ms
<27>Mar 14 08:17:52 edge01 checkout[1662]: Connection timeout to database host db-replica-2.prod.internal after 23497ms
<27>Mar 14 08:17:54 edge01 worker[2029]: Upstream auth.svc.internal returned 502 for request 6d3d5b53-11fe-4fa7-a77e-f7ec0aee1607
<27>Mar 14 08:17:57 worker01 worker[1701]: Connection timeout to database host db-replica-1.prod.internal after 2363ms
<27>Mar 14 08:17:58 worker01 scheduler[8695]: Connection timeout to database host db-replica-1.prod.internal after 27999ms
<27>Mar 14 08:18:01 web01 api[6309]: Connection timeout to database host db-replica-1.prod.internal after 7862ms
<27>Mar 14 08:18:02 api02 checkout[9918]: Connection timeout to database host db-primary.prod.internal after 13411ms
<27>Mar 14 08:18:04 worker02 scheduler[6985]: Connection timeout to database host db-replica-2.prod.internal after 19418ms
<27>Mar 14 08:18:06 api02 scheduler[4651]: Connection timeout to database host db-replica-2.prod.internal after 12006ms
<27>Mar 14 08:18:09 worker01 checkout[8976]: Connection timeout to database host db-replica-2.prod.internal after 11652ms
<27>Mar 14 08:18:11 web01 worker[8207]: Payment gateway stripe.gateway.internal rejected transaction e507ca7b-0b74-4586-976e-7b2bcf718e7f: insufficient_funds
<27>Mar 14 08:18:12 api01 api[9430]: Failed to acquire lock on resource /var/lib/app/queue.lock after 10608ms
<27>Mar 14 08:18:15 api01 worker[4062]: Connection timeout to database host db-shard-3.prod.internal after 555ms
<27>Mar 14 08:18:16 worker01 api[4773]: Connection timeout to database host db-replica-2.prod.internal after 26643ms
<27>Mar 14 08:18:19 worker01 api[8889]: Upstream search.svc.internal returned 502 for request 0a00c4f9-6708-427f-8a6a-f21e11f28bbd
<27>Mar 14 08:18:21 edge01 worker[3641]: Connection timeout to database host db-replica-1.prod.internal after 6776ms
<27>Mar 14 08:18:23 web01 worker[1706]: Connection timeout to database host db-shard-3.prod.internal after 5181ms
<27>Mar 14 08:18:24 web03 worker[9399]: Connection timeout to database host db-primary.prod.internal after 13630ms
<27>Mar 14 08:18:26 web03 api[3725]: Connection timeout to database host db-replica-2.prod.internal after 27677ms
<27>Mar 14 08:18:29 web03 scheduler[1305]: Connection timeout to database host db-replica-1.prod.internal after 3839ms
<27>Mar 14 08:18:31 web02 worker[8359]: Failed to acquire lock on resource /var/lib/app/queue.lock after 1861ms
<27>Mar 14 08:18:33 edge01 worker[4594]: Connection timeout to database host db-replica-2.prod.internal after 9629ms
<27>Mar 14 08:18:35 web03 worker[9369]: Connection timeout to database host db-replica-1.prod.internal after 4707ms
<27>Mar 14 08:18:37 web03 checkout[8429]: Upstream auth.svc.internal returned 502 for request 0d956e6e-7228-40f1-9cef-8cba839d03b5
<27>Mar 14 08:18:38 api01 scheduler[7315]: Connection timeout to database host db-primary.prod.internal after 28396ms
<27>Mar 14 08:18:40 web02 api[4315]: Upstream billing.svc.internal returned 500 for request df5d6ceb-4761-4fce-8dcb-91b8669dbd0e
<27>Mar 14 08:18:43 web03 api[3906]: Failed to acquire lock on resource /var/lib/app/cache.lock after 12111ms
<27>Mar 14 08:18:45 web01 api[2526]: Connection timeout to database host db-primary.prod.internal after 26007ms
<27>Mar 14 08:18:47 web03 api[8069]: Connection timeout to database host db-replica-2.prod.internal after 25137ms
<27>Mar 14 08:18:49 worker01 api[9680]: Connection timeout to database host db-replica-2.prod.internal after 16510ms
<27>Mar 14 08:18:51 api01 scheduler[7246]: Payment gateway braintree.gateway.internal rejected transaction 13cb3f65-7e5a-4a54-9c09-68919a61c045: insufficient_funds
<27>Mar 14 08:18:53 web02 scheduler[3193]: Connection timeout to database host db-shard-3.prod.internal after 6512ms
<27>Mar 14 08:18:55 edge01 worker[2373]: Upstream billing.svc.internal returned 503 for request c9ceb9de-09af-4a54-8649-0b42a67cef85
<27>Mar 14 08:18:57 worker02 checkout[4096]: Failed to acquire lock on resource /var/lib/app/cache.lock after 13727ms
<27>Mar 14 08:18:58 web01 api[9515]: Failed to acquire lock on resource /var/lib/app/cache.lock after 1744ms
<27>Mar 14 08:19:00 worker02 worker[6175]: Connection timeout to database host db-replica-2.prod.internal after 6458ms
<27>Mar 14 08:19:03 worker01 api[3535]: Upstream notifications.svc.internal returned 503 for request 1ea2e029-0e54-477f-aa98-e4597503f552
<27>Mar 14 08:19:05 api02 checkout[9783]: Failed to acquire lock on resource /var/lib/app/queue.lock after 10468ms
<27>Mar 14 08:19:07 worker01 checkout[7511]: Connection timeout to database host db-replica-2.prod.internal after 13806ms
<27>Mar 14 08:19:08 worker02 scheduler[3928]: Connection timeout to database host db-shard-3.prod.internal after 23325ms
<27>Mar 14 08:19:11 edge01 checkout[6024]: Connection timeout to database host db-primary.prod.internal after 29955ms
<27>Mar 14 08:19:13 worker02 worker[2951]: Connection timeout to database host db-primary.prod.internal after 805ms
<27>Mar 14 08:19:14 edge01 worker[2642]: Connection timeout to database host db-replica-2.prod.internal after 3291ms
<27>Mar 14 08:19:16 worker02 checkout[1852]: Upstream notifications.svc.internal returned 503 for request f85842b9-2c26-4123-acc2-aa31e5fa3cbc
<27>Mar 14 08:19:18 worker02 api[2492]: Upstream auth.svc.internal returned 504 for request 5349b5e1-9dc6-45fd-8ad0-c88846a0b72f
<27>Mar 14 08:19:20 edge01 scheduler[2444]: Connection timeout to database host db-shard-3.prod.internal after 21664ms
<27>Mar 14 08:19:22 api02 worker[7075]: Connection timeout to database host db-shard-3.prod.internal after 20575ms
<27>Mar 14 08:19:24 web01 scheduler[7564]: Failed to acquire lock on resource /var/lib/app/queue.lock after 8532ms
<27>Mar 14 08:19:27 web01 worker[9099]: Upstream search.svc.internal returned 502 for request 7dc85c75-cb75-41c2-9be2-ce587762cf9a
<27>Mar 14 08:19:29 web01 worker[9489]: Upstream search.svc.internal returned 504 for request bc05ded6-7a53-450d-a83b-2212e6c89aa5
<27>Mar 14 08:19:31 api01 api[9364]: Connection timeout to database host db-shard-3.prod.internal after 8490ms
<27>Mar 14 08:19:32 web03 api[6893]: Connection timeout to database host db-replica-1.prod.internal after 18377ms
<27>Mar 14 08:19:34 web01 worker[2422]: Failed to acquire lock on resource /var/lib/app/queue.lock after 4986ms
<27>Mar 14 08:19:37 worker02 checkout[7474]: Upstream auth.svc.internal returned 500 for request 51a2eaae-95f8-4aaf-97d6-91ff0fb3fe61
<27>Mar 14 08:19:39 api01 api[2124]: Connection timeout to database host db-replica-2.prod.internal after 20919ms
<27>Mar 14 08:19:40 api02 checkout[6693]: Upstream notifications.svc.internal returned 503 for request 9d976ba1-8190-4e3d-8f46-cdfcc4638dfa
<27>Mar 14 08:19:43 api01 worker[8450]: Connection timeout to database host db-replica-1.prod.internal after 8846ms
<27>Mar 14 08:19:45 web01 checkout[3867]: Payment gateway braintree.gateway.internal rejected transaction 7ca08558-d806-4490-b802-43ce3a351696: insufficient_funds
<27>Mar 14 08:19:47 web02 api[9243]: Connection timeout to database host db-replica-1.prod.internal after 24868ms
<27>Mar 14 08:19:49 worker01 api[9409]: Upstream search.svc.internal returned 504 for request 860ef571-e01f-465f-9722-9c79c1cf93a8
<27>Mar 14 08:19:51 api02 scheduler[7178]: Connection timeout to database host db-shard-3.prod.internal after 2792ms
<27>Mar 14 08:19:53 api01 checkout[8517]: Connection timeout to database host db-replica-2.prod.internal after 11061ms
<27>Mar 14 08:19:55 web01 scheduler[2904]: Connection timeout to database host db-shard-3.prod.internal after 14933ms
<27>Mar 14 08:19:57 web02 scheduler[4610]: Connection timeout to database host db-replica-2.prod.internal after 21471ms
<27>Mar 14 08:19:58 worker01 scheduler[6785]: Failed to acquire lock on resource /var/lib/app/session.lock after 3347ms
<27>Mar 14 08:20:00 api02 worker[5706]: Connection timeout to database host db-replica-2.prod.internal after 4840ms
<27>Mar 14 08:20:02 worker02 checkout[2245]: Connection timeout to database host db-replica-2.prod.internal after 7171ms
<27>Mar 14 08:20:04 edge01 checkout[6792]: Connection timeout to database host db-shard-3.prod.internal after 20854ms
<27>Mar 14 08:20:06 api02 worker[4115]: Connection timeout to database host db-primary.prod.internal after 14237ms
<27>Mar 14 08:20:09 api01 worker[4965]: Failed to acquire lock on resource /tmp/deploy.lock after 2624ms
<27>Mar 14 08:20:11 api01 worker[4322]: Connection timeout to database host db-shard-3.prod.internal after 16413ms
<27>Mar 14 08:20:12 worker01 worker[2747]: Connection timeout to database host db-primary.prod.internal after 20859ms
<27>Mar 14 08:20:14 web02 scheduler[1108]: Failed to acquire lock on resource /tmp/deploy.lock after 13157ms
<27>Mar 14 08:20:16 edge01 checkout[6167]: Failed to acquire lock on resource /var/lib/app/cache.lock after 5742ms
<27>Mar 14 08:20:19 api02 scheduler[3996]: Upstream notifications.svc.internal returned 503 for request ff69a0e6-4ab6-4b1d-8add-0d9e3f1b4f1e
<27>Mar 14 08:20:21 worker02 worker[3101]: Failed to acquire lock on resource /var/lib/app/session.lock after 13572ms
<27>Mar 14 08:20:22 worker02 scheduler[9859]: Connection timeout to database host db-primary.prod.internal after 19777ms
<27>Mar 14 08:20:24 worker01 scheduler[3863]: Connection timeout to database host db-primary.prod.internal after 5034ms
<27>Mar 14 08:20:27 web02 scheduler[6171]: Connection timeout to database host db-primary.prod.internal after 29025ms
<27>Mar 14 08:20:28 web01 worker[7216]: Connection timeout to database host db-replica-2.prod.internal after 9044ms
<27>Mar 14 08:20:30 worker01 checkout[5254]: Connection timeout to database host db-shard-3.prod.internal after 9847ms
<27>Mar 14 08:20:32 api02 worker[5413]: Connection timeout to database host db-replica-2.prod.internal after 1404ms
<27>Mar 14 08:20:35 worker01 checkout[4013]: Connection timeout to database host db-primary.prod.internal after 4165ms
<27>Mar 14 08:20:37 worker02 worker[7866]: Upstream billing.svc.internal returned 502 for request 284686c0-22bc-4106-a654-5901db465b89
<27>Mar 14 08:20:39 worker01 scheduler[5581]: Connection timeout to database host db-replica-2.prod.internal after 25629ms
<27>Mar 14 08:20:41 web03 worker[4744]: Connection timeout to database host db-primary.prod.internal after 28634ms
<27>Mar 14 08:20:42 api02 worker[8081]: Connection timeout to database host db-replica-1.prod.internal after 24014ms
<27>Mar 14 08:20:45 api01 worker[3450]: Connection timeout to database host db-shard-3.prod.internal after 28465ms
<27>Mar 14 08:20:47 web01 worker[7906]: Connection timeout to database host db-shard-3.prod.internal after 15732ms
<27>Mar 14 08:20:49 web03 checkout[8972]: Connection timeout to database host db-replica-2.prod.internal after 3849ms
<27>Mar 14 08:20:51 api02 api[7527]: Connection timeout to database host db-replica-2.prod.internal after 20682ms
<27>Mar 14 08:20:52 web03 api[4530]: Upstream notifications.svc.internal returned 500 for request 9d56454c-738d-4839-b808-23e90af9ac9b
<27>Mar 14 08:20:55 edge01 api[8370]: Connection timeout to database host db-shard-3.prod.internal after 10243ms
<27>Mar 14 08:20:57 api02 checkout[6903]: Connection timeout to database host db-replica-1.prod.internal after 9185ms
<27>Mar 14 08:20:59 worker01 api[7746]: Upstream notifications.svc.internal returned 503 for request 3b43ee20-2c6a-433f-9263-b54768860f42
<27>Mar 14 08:21:00 worker01 worker[9821]: Connection timeout to database host db-primary.prod.internal after 15020ms
<27>Mar 14 08:21:02 api02 checkout[6326]: Connection timeout to database host db-primary.prod.internal after 15911ms
<27>Mar 14 08:21:05 edge01 api[7243]: Connection timeout to database host db-replica-2.prod.internal after 18712ms
<27>Mar 14 08:21:07 edge01 worker[3745]: Upstream auth.svc.internal returned 504 for request 7caedd3c-0473-45c2-9590-22399978e6c2
<27>Mar 14 08:21:09 worker02 scheduler[2168]: Payment gateway adyen.gateway.internal rejected transaction 01d5bb5d-a244-48c0-aeb0-63e8c4c56574: insufficient_funds
<27>Mar 14 08:21:10 web03 checkout[4739]: Failed to acquire lock on resource /var/lib/app/cache.lock after 3578ms
<27>Mar 14 08:21:13 worker01 checkout[5382]: Upstream search.svc.internal returned 502 for request 0821dde1-20c9-4db1-abcf-dd4ad3a5c16c
<27>Mar 14 08:21:15 api02 scheduler[6950]: Connection timeout to database host db-primary.prod.internal after 3556ms
<27>Mar 14 08:21:17 worker02 worker[4012]: Connection timeout to database host db-replica-2.prod.internal after 8737ms
<27>Mar 14 08:21:18 web03 worker[5122]: Failed to acquire lock on resource /var/lib/app/queue.lock after 6129ms
<27>Mar 14 08:21:20 api01 worker[7505]: Connection timeout to database host db-replica-2.prod.internal after 28362ms
<27>Mar 14 08:21:22 web02 api[8752]: Connection timeout to database host db-replica-1.prod.internal after 2416ms
<27>Mar 14 08:21:24 web01 scheduler[9584]: Upstream search.svc.internal returned 503 for request daed1fd4-dfa5-4148-b4f5-646b169b9e88
<27>Mar 14 08:21:26 web02 checkout[3544]: Connection timeout to database host db-replica-1.prod.internal after 18174ms
<27>Mar 14 08:21:28 worker02 checkout[4757]: Connection timeout to database host db-primary.prod.internal after 25214ms
<27>Mar 14 08:21:30 api02 scheduler[3071]: Connection timeout to database host db-replica-1.prod.internal after 13730ms
<27>Mar 14 08:21:32 web03 api[1575]: Upstream auth.svc.internal returned 504 for request 6d71de48-7bc0-41ad-9370-0d5fd8107a8f
<27>Mar 14 08:21:35 worker01 scheduler[8654]: Connection timeout to database host db-replica-2.prod.internal after 7387ms
<27>Mar 14 08:21:37 edge01 worker[9835]: Connection timeout to database host db-shard-3.prod.internal after 29899ms
<27>Mar 14 08:21:38 edge01 scheduler[9576]: Connection timeout to database host db-shard-3.prod.internal after 23168ms
<27>Mar 14 08:21:41 worker01 scheduler[9841]: Upstream notifications.svc.internal returned 503 for request 09b9b28b-d87c-4448-b55c-1be30938e493
<27>Mar 14 08:21:42 web02 scheduler[6044]: Connection timeout to database host db-shard-3.prod.internal after 7440ms
<27>Mar 14 08:21:45 worker02 checkout[3551]: Connection timeout to database host db-primary.prod.internal after 25044ms
<27>Mar 14 08:21:47 web01 api[9574]: Connection timeout to database host db-shard-3.prod.internal after 20026ms
<27>Mar 14 08:21:48 worker01 checkout[8687]: Connection timeout to database host db-primary.prod.internal after 8602ms
<27>Mar 14 08:21:50 edge01 scheduler[5908]: Failed to acquire lock on resource /var/lib/app/cache.lock after 2443ms
<27>Mar 14 08:21:52 worker02 checkout[4788]: Connection timeout to database host db-shard-3.prod.internal after 13464ms
<27>Mar 14 08:21:54 worker01 worker[7745]: Upstream search.svc.internal returned 502 for request 74a4fd03-b321-4191-85f0-0d72affd813f
<27>Mar 14 08:21:57 worker02 worker[3675]: Connection timeout to database host db-replica-1.prod.internal after 16017ms
<27>Mar 14 08:21:59 worker02 worker[5308]: Connection timeout to database host db-primary.prod.internal after 19310ms
<27>Mar 14 08:22:00 worker02 worker[9464]: Connection timeout to database host db-replica-1.prod.internal after 10533ms
<27>Mar 14 08:22:03 api02 checkout[7765]: Connection timeout to database host db-shard-3.prod.internal after 27716ms
<27>Mar 14 08:22:04 web02 worker[3238]: Connection timeout to database host db-shard-3.prod.internal after 7450ms
<27>Mar 14 08:22:07 web02 scheduler[8418]: Connection timeout to database host db-shard-3.prod.internal after 27557ms
<27>Mar 14 08:22:09 edge01 worker[6253]: Connection timeout to database host db-replica-1.prod.internal after 22509ms
<27>Mar 14 08:22:10 web03 api[9764]: Upstream notifications.svc.internal returned 500 for request 47042130-fe2f-4a01-a1f2-437562fa3f48
<27>Mar 14 08:22:12 web03 api[5622]: Upstream billing.svc.internal returned 502 for request 24a69354-6688-4a76-8d57-ce8640ffd2db
<27>Mar 14 08:22:15 web03 scheduler[1134]: Upstream auth.svc.internal returned 504 for request 8c7f941d-81e5-416f-977f-3a6251c87546
<27>Mar 14 08:22:16 web02 worker[1241]: Upstream notifications.svc.internal returned 500 for request 66750349-556f-41b2-ad8c-b14585102c08
<27>Mar 14 08:22:19 api01 checkout[7885]: Upstream search.svc.internal returned 502 for request ea2af290-2731-4701-86e2-b729c1acedfa
<27>Mar 14 08:22:21 edge01 checkout[8597]: Connection timeout to database host db-shard-3.prod.internal after 22288ms
<27>Mar 14 08:22:22 worker01 api[3393]: Connection timeout to database host db-primary.prod.internal after 20460ms
<27>Mar 14 08:22:24 api02 scheduler[2169]: Connection timeout to database host db-primary.prod.internal after 22384ms
<27>Mar 14 08:22:27 web03 worker[5097]: Failed to acquire lock on resource /var/lib/app/cache.lock after 10406ms
<27>Mar 14 08:22:29 web01 api[5078]: Upstream auth.svc.internal returned 502 for request 2fa9cb15-0db7-42b6-b316-2a912c2f9642
<27>Mar 14 08:22:31 edge01 worker[1549]: Connection timeout to database host db-primary.prod.internal after 4165ms
<27>Mar 14 08:22:32 edge01 api[9538]: Connection timeout to database host db-primary.prod.internal after 25986ms
<27>Mar 14 08:22:34 edge01 scheduler[6113]: Connection timeout to database host db-replica-1.prod.internal after 735ms
<27>Mar 14 08:22:37 web03 api[9534]: Upstream auth.svc.internal returned 500 for request ff8a906f-5a94-4213-8b22-f95bd9ba25c8
<27>Mar 14 08:22:38 api02 scheduler[5895]: Connection timeout to database host db-primary.prod.internal after 5508ms
<27>Mar 14 08:22:40 worker01 api[4568]: Payment gateway stripe.gateway.internal rejected transaction bb6699be-8540-47af-a891-da0523676d1b: insufficient_funds
<27>Mar 14 08:22:42 api01 scheduler[6316]: Upstream billing.svc.internal returned 503 for request dd629000-66fc-4757-a6cd-4a2da95b86ad
<27>Mar 14 08:22:44 web03 checkout[2960]: Upstream billing.svc.internal returned 503 for request b24e11db-76ad-4c0d-8b64-067763e01e84
<27>Mar 14 08:22:47 api01 api[1688]: Upstream auth.svc.internal returned 504 for request a94f2971-c795-4223-b37d-511a5901785b
<27>Mar 14 08:22:49 web03 checkout[2466]: Connection timeout to database host db-shard-3.prod.internal after 6454ms
<27>Mar 14 08:22:50 web03 scheduler[2497]: Connection timeout to database host db-replica-1.prod.internal after 21376ms
<27>Mar 14 08:22:53 web01 api[6824]: Failed to acquire lock on resource /var/lib/app/cache.lock after 6567ms
<27>Mar 14 08:22:54 web02 worker[8992]: Upstream auth.svc.internal returned 500 for request 11d31eca-e40a-4338-9883-637cddd96d8b
<27>Mar 14 08:22:56 worker01 checkout[8187]: Upstream search.svc.internal returned 503 for request 7632e06d-b3ee-43b9-b9b2-93b439fa2063
<27>Mar 14 08:22:58 web01 scheduler[6868]: Connection timeout to database host db-replica-1.prod.internal after 5854ms
<27>Mar 14 08:23:00 api02 api[8951]: Upstream notifications.svc.internal returned 503 for request 80f3dbea-8310-4e04-84c1-76429fcb4bcf
<27>Mar 14 08:23:03 api02 scheduler[4109]: Upstream notifications.svc.internal returned 502 for request 5b25949c-398b-4815-baa3-fcbb48eaa002
<27>Mar 14 08:23:04 worker01 checkout[8674]: Upstream search.svc.internal returned 504 for request b9c33ca1-fef7-4116-88e2-4a76bf34ae4b
<27>Mar 14 08:23:07 web03 api[9702]: Connection timeout to database host db-primary.prod.internal after 3549ms
<27>Mar 14 08:23:08 edge01 checkout[5314]: Failed to acquire lock on resource /tmp/deploy.lock after 12624ms
<27>Mar 14 08:23:11 edge01 api[9289]: Upstream billing.svc.internal returned 502 for request 368ba36c-9625-49ca-afbb-543004df2caa
<27>Mar 14 08:23:13 web03 api[6965]: Failed to acquire lock on resource /var/lib/app/session.lock after 5505ms
<27>Mar 14 08:23:14 worker01 api[9830]: Connection timeout to database host db-primary.prod.internal after 25423ms
<27>Mar 14 08:23:17 api01 scheduler[3525]: Failed to acquire lock on resource /var/lib/app/cache.lock after 2882ms
<27>Mar 14 08:23:19 worker01 scheduler[6276]: Connection timeout to database host db-replica-1.prod.internal after 17135ms
<27>Mar 14 08:23:20 web02 api[1717]: Connection timeout to database host db-replica-2.prod.internal after 25134ms
<27>Mar 14 08:23:22 web03 api[4310]: Connection timeout to database host db-shard-3.prod.internal after 19172ms
<27>Mar 14 08:23:24 edge01 worker[2617]: Connection timeout to database host db-shard-3.prod.internal after 29919ms
<27>Mar 14 08:23:26 worker02 scheduler[9615]: Connection timeout to database host db-replica-1.prod.internal after 22460ms
<27>Mar 14 08:23:29 web01 checkout[4616]: Upstream billing.svc.internal returned 500 for request 9c6faef5-5473-4a2f-a0cb-7c24d1bc8755
<27>Mar 14 08:23:30 web01 worker[9051]: Upstream notifications.svc.internal returned 504 for request c01550d2-ae9b-484a-a79e-eb2a336df4cf
<27>Mar 14 08:23:32 edge01 api[6913]: Connection timeout to database host db-shard-3.prod.internal after 18136ms
<27>Mar 14 08:23:35 worker01 api[5355]: Connection timeout to database host db-replica-1.prod.internal after 15367ms
<27>Mar 14 08:23:36 web02 checkout[2276]: Upstream search.svc.internal returned 503 for request fa2c7d9d-fc45-4627-919b-d27b14d43e3d
<27>Mar 14 08:23:38 web02 api[4784]: Connection timeout to database host db-primary.prod.internal after 11877ms
<27>Mar 14 08:23:41 api01 worker[9621]: Connection timeout to database host db-replica-2.prod.internal after 24978ms
<27>Mar 14 08:23:42 web03 scheduler[2728]: Upstream billing.svc.internal returned 502 for request 15a5250f-8a2e-48b1-87cc-e1038c7fd4de
<27>Mar 14 08:23:45 edge01 scheduler[2705]: Connection timeout to database host db-replica-1.prod.internal after 26849ms
<27>Mar 14 08:23:46 web03 scheduler[9268]: Connection timeout to database host db-primary.prod.internal after 7277ms
<27>Mar 14 08:23:49 web03 scheduler[6772]: Failed to acquire lock on resource /tmp/deploy.lock after 2582ms
<27>Mar 14 08:23:51 worker01 checkout[7755]: Upstream auth.svc.internal returned 500 for request 42ef84e4-45c3-4198-919c-26c839fd90c2
<27>Mar 14 08:23:53 worker01 worker[9939]: Upstream billing.svc.internal returned 503 for request 435920b8-9b64-40ee-a76d-edcb7e09b7ee
<27>Mar 14 08:23:54 worker02 checkout[6898]: Connection timeout to database host db-replica-2.prod.internal after 18628ms
<27>Mar 14 08:23:57 edge01 scheduler[1003]: Upstream billing.svc.internal returned 502 for request 458eaa80-e82e-42fc-b44b-a68c3662edde
<27>Mar 14 08:23:58 api02 api[9480]: Connection timeout to database host db-primary.prod.internal after 29672ms
<27>Mar 14 08:24:00 api02 api[5920]: Connection timeout to database host db-shard-3.prod.internal after 24229ms
<27>Mar 14 08:24:02 api01 checkout[1355]: Failed to acquire lock on resource /var/lib/app/session.lock after 12252ms
<27>Mar 14 08:24:05 web03 api[1618]: Connection timeout to database host db-primary.prod.internal after 9024ms
<27>Mar 14 08:24:07 worker01 worker[3267]: Failed to acquire lock on resource /var/lib/app/queue.lock after 6120ms
<27>Mar 14 08:24:09 worker01 worker[4817]: Failed to acquire lock on resource /var/lib/app/cache.lock after 11977ms
<27>Mar 14 08:24:11 api02 api[7550]: Failed to acquire lock on resource /tmp/deploy.lock after 4674ms
<27>Mar 14 08:24:12 web02 checkout[8905]: Upstream notifications.svc.internal returned 500 for request 2e0bae9e-aef8-4533-b52a-67a27fe883bf
<27>Mar 14 08:24:14 edge01 api[5921]: Failed to acquire lock on resource /var/lib/app/cache.lock after 2965ms
<27>Mar 14 08:24:16 worker02 api[1916]: Upstream auth.svc.internal returned 504 for request d39c55f8-5459-4d64-9b7d-866c1abd52c4
<27>Mar 14 08:24:19 worker01 scheduler[7824]: Upstream billing.svc.internal returned 504 for request 43f6808a-941d-4aa3-9adf-4b19f4e8193a
<27>Mar 14 08:24:20 api01 api[7266]: Upstream auth.svc.internal returned 502 for request 37d9df86-c246-4ed5-9011-fa0652a8f098
<27>Mar 14 08:24:22 api02 api[6231]: Upstream auth.svc.internal returned 503 for request ac9a152e-51ef-485a-9c75-1abeea9febcd
<27>Mar 14 08:24:24 worker01 worker[3509]: Upstream notifications.svc.internal returned 504 for request e2ff4d1c-b149-4cb1-8644-44e0090b2ab9
<27>Mar 14 08:24:26 worker02 scheduler[9269]: Failed to acquire lock on resource /tmp/deploy.lock after 2099ms
<27>Mar 14 08:24:28 api01 scheduler[5747]: Connection timeout to database host db-primary.prod.internal after 24827ms
<27>Mar 14 08:24:31 web03 worker[6507]: Connection timeout to database host db-replica-2.prod.internal after 16903ms
<27>Mar 14 08:24:32 edge01 api[9980]: Failed to acquire lock on resource /var/lib/app/session.lock after 13140ms
<27>Mar 14 08:24:35 api02 api[7529]: Connection timeout to database host db-primary.prod.internal after 20795ms
<27>Mar 14 08:24:37 web03 worker[8879]: Upstream notifications.svc.internal returned 500 for request 3de609ba-eb80-4462-9367-aebd0e545672
<27>Mar 14 08:24:38 edge01 scheduler[4212]: Connection timeout to database host db-replica-2.prod.internal after 22237ms
<27>Mar 14 08:24:40 api01 api[5362]: Connection timeout to database host db-replica-1.prod.internal after 1870ms
<27>Mar 14 08:24:43 worker01 api[2843]: Connection timeout to database host db-replica-1.prod.internal after 17250ms