==> Building on scovillain ==> Checking for remote environment... ==> Syncing package to remote host... sending incremental file list created directory packages/loki ./ .SRCINFO 1,840 100% 0.00kB/s 0:00:00 1,840 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=10/12) PKGBUILD 3,824 100% 3.65MB/s 0:00:00 3,824 100% 3.65MB/s 0:00:00 (xfr#2, to-chk=9/12) go-1.19.patch 4,225 100% 4.03MB/s 0:00:00 4,225 100% 4.03MB/s 0:00:00 (xfr#3, to-chk=8/12) loki-2.9.2-1.log 489 100% 477.54kB/s 0:00:00 489 100% 477.54kB/s 0:00:00 (xfr#4, to-chk=7/12) loki.service 700 100% 683.59kB/s 0:00:00 700 100% 683.59kB/s 0:00:00 (xfr#5, to-chk=6/12) loki.sysusers 35 100% 34.18kB/s 0:00:00 35 100% 34.18kB/s 0:00:00 (xfr#6, to-chk=5/12) loki.tmpfiles 33 100% 32.23kB/s 0:00:00 33 100% 32.23kB/s 0:00:00 (xfr#7, to-chk=4/12) no-moving-gc-bump.patch 7,094 100% 6.77MB/s 0:00:00 7,094 100% 6.77MB/s 0:00:00 (xfr#8, to-chk=3/12) promtail.service 711 100% 694.34kB/s 0:00:00 711 100% 694.34kB/s 0:00:00 (xfr#9, to-chk=2/12) promtail.sysusers 74 100% 72.27kB/s 0:00:00 74 100% 72.27kB/s 0:00:00 (xfr#10, to-chk=1/12) promtail.tmpfiles 45 100% 43.95kB/s 0:00:00 45 100% 43.95kB/s 0:00:00 (xfr#11, to-chk=0/12) sent 8,484 bytes received 264 bytes 5,832.00 bytes/sec total size is 18,778 speedup is 2.15 ==> Patching arch to riscv64... ==> Running extra-riscv64-build -- -d /home/felix/packages/riscv64-pkg-cache:/var/cache/pacman/pkg -l root56 on remote host... [?25l:: Synchronizing package databases... core downloading... extra downloading... :: Starting full system upgrade... there is nothing to do [?25h==> Building in chroot for [extra] (riscv64)... ==> Synchronizing chroot copy [/var/lib/archbuild/extra-riscv64/root] -> [root56]...done ==> Making package: loki 2.9.2-1 (Mon Dec 11 19:32:10 2023) ==> Retrieving sources...  -> Downloading loki-2.9.2.tar.gz... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 100 494k 0 494k 0 0 165k 0 --:--:-- 0:00:02 --:--:-- 365k 100 1862k 0 1862k 0 0 515k 0 --:--:-- 0:00:03 --:--:-- 938k 100 3973k 0 3973k 0 0 744k 0 --:--:-- 0:00:05 --:--:-- 1072k 100 4889k 0 4889k 0 0 863k 0 --:--:-- 0:00:05 --:--:-- 1213k 100 7444k 0 7444k 0 0 1114k 0 --:--:-- 0:00:06 --:--:-- 1473k 100 10.8M 0 10.8M 0 0 1444k 0 --:--:-- 0:00:07 --:--:-- 2256k 100 13.8M 0 13.8M 0 0 1649k 0 --:--:-- 0:00:08 --:--:-- 2469k 100 18.3M 0 18.3M 0 0 1943k 0 --:--:-- 0:00:09 --:--:-- 3425k 100 24.9M 0 24.9M 0 0 2233k 0 --:--:-- 0:00:11 --:--:-- 3579k 100 25.9M 0 25.9M 0 0 2271k 0 --:--:-- 0:00:11 --:--:-- 3815k 100 30.0M 0 30.0M 0 0 2438k 0 --:--:-- 0:00:12 --:--:-- 3980k 100 34.6M 0 34.6M 0 0 2566k 0 --:--:-- 0:00:13 --:--:-- 4079k 100 36.5M 0 36.5M 0 0 2439k 0 --:--:-- 0:00:15 --:--:-- 3280k 100 37.6M 0 37.6M 0 0 2463k 0 --:--:-- 0:00:15 --:--:-- 3083k 100 40.3M 0 40.3M 0 0 2486k 0 --:--:-- 0:00:16 --:--:-- 2996k 100 43.5M 0 43.5M 0 0 2587k 0 --:--:-- 0:00:17 --:--:-- 2997k  -> Found promtail.sysusers  -> Found promtail.service  -> Found promtail.tmpfiles  -> Found loki.sysusers  -> Found loki.service  -> Found loki.tmpfiles ==> Validating source files with sha512sums... loki-2.9.2.tar.gz ... Passed promtail.sysusers ... Passed promtail.service ... Passed promtail.tmpfiles ... Passed loki.sysusers ... Passed loki.service ... Passed loki.tmpfiles ... Passed ==> Making package: loki 2.9.2-1 (Mon Dec 11 19:33:16 2023) ==> Checking runtime dependencies... ==> Checking buildtime dependencies... ==> Installing missing dependencies... [?25lresolving dependencies... looking for conflicting packages... Package (10) New Version Net Change core/argon2 20190702-5 0.08 MiB core/cryptsetup 2.6.1-3 2.50 MiB core/dbus 1.14.10-1 0.80 MiB core/device-mapper 2.03.22-2 0.68 MiB core/hwdata 0.377-1 8.95 MiB core/kbd 2.6.3-1 3.01 MiB core/kmod 31-1 0.25 MiB core/popt 1.19-1 0.55 MiB extra/go 2:1.21.5-1 202.21 MiB core/systemd 255-1 28.31 MiB Total Installed Size: 247.35 MiB :: Proceed with installation? [Y/n] checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... installing go... installing device-mapper... installing popt... installing argon2... installing cryptsetup... installing dbus... installing kbd... installing kmod... installing hwdata... installing systemd... warning: directory permissions differ on /var/log/journal/ filesystem: 755 package: 2755 Creating group 'sys' with GID 3. Creating group 'mem' with GID 8. Creating group 'ftp' with GID 11. Creating group 'mail' with GID 12. Creating group 'log' with GID 19. Creating group 'smmsp' with GID 25. Creating group 'proc' with GID 26. Creating group 'games' with GID 50. Creating group 'lock' with GID 54. Creating group 'network' with GID 90. Creating group 'floppy' with GID 94. Creating group 'scanner' with GID 96. Creating group 'power' with GID 98. Creating group 'nobody' with GID 65534. Creating group 'adm' with GID 999. Creating group 'wheel' with GID 998. Creating group 'utmp' with GID 997. Creating group 'audio' with GID 996. Creating group 'disk' with GID 995. Creating group 'input' with GID 994. Creating group 'kmem' with GID 993. Creating group 'kvm' with GID 992. Creating group 'lp' with GID 991. Creating group 'optical' with GID 990. Creating group 'render' with GID 989. Creating group 'sgx' with GID 988. Creating group 'storage' with GID 987. Creating group 'tty' with GID 5. Creating group 'uucp' with GID 986. Creating group 'video' with GID 985. Creating group 'users' with GID 984. Creating group 'groups' with GID 983. Creating group 'systemd-journal' with GID 982. Creating group 'rfkill' with GID 981. Creating group 'bin' with GID 1. Creating user 'bin' (n/a) with UID 1 and GID 1. Creating group 'daemon' with GID 2. Creating user 'daemon' (n/a) with UID 2 and GID 2. Creating user 'mail' (n/a) with UID 8 and GID 12. Creating user 'ftp' (n/a) with UID 14 and GID 11. Creating group 'http' with GID 33. Creating user 'http' (n/a) with UID 33 and GID 33. Creating user 'nobody' (Kernel Overflow User) with UID 65534 and GID 65534. Creating group 'dbus' with GID 81. Creating user 'dbus' (System Message Bus) with UID 81 and GID 81. Creating group 'systemd-coredump' with GID 980. Creating user 'systemd-coredump' (systemd Core Dumper) with UID 980 and GID 980. Creating group 'systemd-network' with GID 979. Creating user 'systemd-network' (systemd Network Management) with UID 979 and GID 979. Creating group 'systemd-oom' with GID 978. Creating user 'systemd-oom' (systemd Userspace OOM Killer) with UID 978 and GID 978. Creating group 'systemd-journal-remote' with GID 977. Creating user 'systemd-journal-remote' (systemd Journal Remote) with UID 977 and GID 977. Creating group 'systemd-journal-upload' with GID 976. Creating user 'systemd-journal-upload' (systemd Journal Upload) with UID 976 and GID 976. Creating group 'systemd-resolve' with GID 975. Creating user 'systemd-resolve' (systemd Resolver) with UID 975 and GID 975. Creating group 'systemd-timesync' with GID 974. Creating user 'systemd-timesync' (systemd Time Synchronization) with UID 974 and GID 974. Creating group 'tss' with GID 973. Creating user 'tss' (tss user for tpm2) with UID 973 and GID 973. Creating group 'uuidd' with GID 68. Creating user 'uuidd' (n/a) with UID 68 and GID 68. Created symlink /etc/systemd/system/getty.target.wants/getty@tty1.service → /usr/lib/systemd/system/getty@.service. Created symlink /etc/systemd/system/multi-user.target.wants/remote-fs.target → /usr/lib/systemd/system/remote-fs.target. Optional dependencies for systemd libmicrohttpd: systemd-journal-gatewayd and systemd-journal-remote quota-tools: kernel-level quota management systemd-sysvcompat: symlink package to provide sysvinit binaries systemd-ukify: combine kernel and initrd into a signed Unified Kernel Image polkit: allow administration as unprivileged user curl: systemd-journal-upload, machinectl pull-tar and pull-raw [installed] gnutls: systemd-journal-gatewayd and systemd-journal-remote [installed] qrencode: show QR codes iptables: firewall features libbpf: support BPF programs libpwquality: check password quality libfido2: unlocking LUKS2 volumes with FIDO2 token libp11-kit: support PKCS#11 [installed] tpm2-tss: unlocking LUKS2 volumes with TPM2 [installed] :: Running post-transaction hooks... ( 1/10) Creating system user accounts... ( 2/10) Updating journal message catalog... ( 3/10) Reloading system manager configuration... Skipped: Current root is not booted. ( 4/10) Reloading user manager configuration... Skipped: Current root is not booted. ( 5/10) Updating udev hardware database... ( 6/10) Applying kernel sysctl settings... Skipped: Current root is not booted. ( 7/10) Creating temporary files... ( 8/10) Reloading device manager configuration... Skipped: Device manager is not running. ( 9/10) Arming ConditionNeedsUpdate... (10/10) Reloading system bus configuration... Skipped: Current root is not booted. [?25h==> Retrieving sources...  -> Found loki-2.9.2.tar.gz  -> Found promtail.sysusers  -> Found promtail.service  -> Found promtail.tmpfiles  -> Found loki.sysusers  -> Found loki.service  -> Found loki.tmpfiles ==> WARNING: Skipping all source file integrity checks. ==> Extracting sources...  -> Extracting loki-2.9.2.tar.gz with bsdtar ==> Starting prepare()... ==> Starting build()... ==> Starting check()... ? github.com/grafana/loki/clients/cmd/promtail [no test files] ok github.com/grafana/loki/clients/cmd/docker-driver 6.284s ok github.com/grafana/loki/clients/cmd/fluent-bit 3.530s ok github.com/grafana/loki/clients/pkg/logentry/logql 0.447s ok github.com/grafana/loki/clients/pkg/logentry/metric 2.880s ok github.com/grafana/loki/clients/pkg/logentry/stages 12.342s ? github.com/grafana/loki/clients/pkg/promtail/api [no test files] ? github.com/grafana/loki/clients/pkg/promtail/client/fake [no test files] ? github.com/grafana/loki/clients/pkg/promtail/limit [no test files] ? github.com/grafana/loki/clients/pkg/promtail/server [no test files] ? github.com/grafana/loki/clients/pkg/promtail/server/ui [no test files] ? github.com/grafana/loki/clients/pkg/promtail/targets [no test files] ok github.com/grafana/loki/clients/pkg/promtail 30.177s ok github.com/grafana/loki/clients/pkg/promtail/client 10.524s ok github.com/grafana/loki/clients/pkg/promtail/config 7.507s ok github.com/grafana/loki/clients/pkg/promtail/discovery/consulagent 3.557s ok github.com/grafana/loki/clients/pkg/promtail/positions 1.561s ok github.com/grafana/loki/clients/pkg/promtail/scrapeconfig 9.761s ok github.com/grafana/loki/clients/pkg/promtail/targets/azureeventhubs 7.498s ok github.com/grafana/loki/clients/pkg/promtail/targets/cloudflare 6.987s ok github.com/grafana/loki/clients/pkg/promtail/targets/docker 11.598s ok github.com/grafana/loki/clients/pkg/promtail/targets/file 58.134s ok github.com/grafana/loki/clients/pkg/promtail/targets/gcplog 6.820s ok github.com/grafana/loki/clients/pkg/promtail/targets/gelf 5.981s ? github.com/grafana/loki/clients/pkg/promtail/targets/journal [no test files] ? github.com/grafana/loki/clients/pkg/promtail/targets/serverutils [no test files] ? github.com/grafana/loki/clients/pkg/promtail/targets/target [no test files] ? github.com/grafana/loki/clients/pkg/promtail/targets/windows [no test files] ? github.com/grafana/loki/clients/pkg/promtail/targets/windows/win_eventlog [no test files] ? github.com/grafana/loki/clients/pkg/promtail/targets/testutils [no test files] ? github.com/grafana/loki/cmd/logcli [no test files] ? github.com/grafana/loki/cmd/logql-analyzer [no test files] ? github.com/grafana/loki/cmd/loki [no test files] ? github.com/grafana/loki/cmd/loki-canary [no test files] ? github.com/grafana/loki/integration/client [no test files] ? github.com/grafana/loki/integration/cluster [no test files] ? github.com/grafana/loki/integration/util [no test files] ? github.com/grafana/loki/pkg/canary/reader [no test files] ? github.com/grafana/loki/pkg/distributor/clientpool [no test files] ? github.com/grafana/loki/pkg/distributor/shardstreams [no test files] ? github.com/grafana/loki/pkg/ingester/client [no test files] ? github.com/grafana/loki/pkg/logcli/index [no test files] ? github.com/grafana/loki/pkg/logcli/labelquery [no test files] ? github.com/grafana/loki/pkg/logcli/seriesquery [no test files] ? github.com/grafana/loki/pkg/logcli/util [no test files] ? github.com/grafana/loki/pkg/logcli/volume [no test files] ? github.com/grafana/loki/pkg/loghttp/legacy [no test files] ? github.com/grafana/loki/pkg/logql/vector [no test files] ? github.com/grafana/loki/pkg/logqlmodel [no test files] ok github.com/grafana/loki/clients/pkg/promtail/targets/heroku 22.558s ok github.com/grafana/loki/clients/pkg/promtail/targets/kafka 16.685s ok github.com/grafana/loki/clients/pkg/promtail/targets/lokipush 17.140s ok github.com/grafana/loki/clients/pkg/promtail/targets/stdin 12.698s ok github.com/grafana/loki/clients/pkg/promtail/targets/syslog 15.336s ok github.com/grafana/loki/clients/pkg/promtail/targets/syslog/syslogparser 1.121s ok github.com/grafana/loki/clients/pkg/promtail/utils 9.449s ok github.com/grafana/loki/clients/pkg/promtail/wal 26.586s ok github.com/grafana/loki/cmd/migrate 10.289s ok github.com/grafana/loki/cmd/querytee 6.557s ? github.com/grafana/loki/pkg/loki/common [no test files] ? github.com/grafana/loki/pkg/lokifrontend [no test files] ? github.com/grafana/loki/pkg/lokifrontend/frontend [no test files] ? github.com/grafana/loki/pkg/lokifrontend/frontend/v1/frontendv1pb [no test files] ? github.com/grafana/loki/pkg/lokifrontend/frontend/v2/frontendv2pb [no test files] ? github.com/grafana/loki/pkg/querier/astmapper [no test files] ? github.com/grafana/loki/pkg/querier/queryrange/queryrangebase/definitions [no test files] ? github.com/grafana/loki/pkg/querier/series [no test files] ? github.com/grafana/loki/pkg/querier/stats [no test files] ? github.com/grafana/loki/pkg/ruler/config [no test files] ? github.com/grafana/loki/pkg/ruler/rulespb [no test files] ? github.com/grafana/loki/pkg/ruler/storage/util [no test files] ? github.com/grafana/loki/pkg/ruler/util [no test files] ? github.com/grafana/loki/pkg/runtime [no test files] ? github.com/grafana/loki/pkg/scheduler/schedulerpb [no test files] ? github.com/grafana/loki/pkg/server [no test files] ? github.com/grafana/loki/pkg/storage/bucket/filesystem [no test files] ? github.com/grafana/loki/pkg/storage/bucket/gcs [no test files] ? github.com/grafana/loki/pkg/storage/bucket/swift [no test files] ? github.com/grafana/loki/pkg/storage/chunk/client/alibaba [no test files] ? github.com/grafana/loki/pkg/storage/chunk/client/testutils [no test files] ? github.com/grafana/loki/pkg/storage/errors [no test files] ? github.com/grafana/loki/pkg/storage/hack [no test files] ? github.com/grafana/loki/pkg/storage/stores/index [no test files] ? github.com/grafana/loki/pkg/storage/stores/indexshipper/compactor/client [no test files] ? github.com/grafana/loki/pkg/storage/stores/indexshipper/compactor/client/grpc [no test files] ? github.com/grafana/loki/pkg/storage/stores/indexshipper/index [no test files] ? github.com/grafana/loki/pkg/storage/stores/shipper [no test files] ? github.com/grafana/loki/pkg/storage/stores/shipper/index/indexfile [no test files] ? github.com/grafana/loki/pkg/storage/stores/shipper/testutil [no test files] ? github.com/grafana/loki/pkg/storage/stores/tsdb/testutil [no test files] ? github.com/grafana/loki/pkg/tracing [no test files] ? github.com/grafana/loki/pkg/util/build [no test files] ? github.com/grafana/loki/pkg/util/chunkcompat [no test files] ? github.com/grafana/loki/pkg/util/deletion [no test files] ? github.com/grafana/loki/pkg/util/encoding [no test files] ? github.com/grafana/loki/pkg/util/extract [no test files] ? github.com/grafana/loki/pkg/util/fakeauth [no test files] ? github.com/grafana/loki/pkg/util/filter [no test files] ? github.com/grafana/loki/pkg/util/httpgrpc [no test files] ? github.com/grafana/loki/pkg/util/limiter [no test files] ? github.com/grafana/loki/pkg/util/math [no test files] ? github.com/grafana/loki/pkg/util/net [no test files] ? github.com/grafana/loki/pkg/util/spanlogger [no test files] ? github.com/grafana/loki/pkg/util/test [no test files] ? github.com/grafana/loki/pkg/util/wal [no test files] ? github.com/grafana/loki/tools/doc-generator [no test files] ? github.com/grafana/loki/tools/tsdb/index-analyzer [no test files] time="2023-12-11T20:00:30Z" level=info msg="server listening on addresses" grpc="[::]:42015" http="[::]:44899" time="2023-12-11T20:00:31Z" level=info msg="server listening on addresses" grpc="[::]:37257" http="[::]:43945" time="2023-12-11T20:00:32Z" level=info msg="server listening on addresses" grpc="[::]:35483" http="[::]:38445" time="2023-12-11T20:00:33Z" level=info msg="server listening on addresses" grpc="[::]:39899" http="[::]:39125" time="2023-12-11T20:00:34Z" level=info msg="server listening on addresses" grpc="[::]:40375" http="[::]:41499" time="2023-12-11T20:00:38Z" level=info msg="server listening on addresses" grpc="[::]:33025" http="[::]:45073" time="2023-12-11T20:00:48Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:00:50Z" level=info msg="server listening on addresses" grpc="[::]:39649" http="[::]:40721" time="2023-12-11T20:00:50Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:00:52Z" level=info msg="server listening on addresses" grpc="[::]:44283" http="[::]:36511" Delete request URL: http://localhost:44899/loki/api/v1/delete?end=1702324839&query=%7Bdeletion_type%3D%22filter%22%7D+%7C%3D+%22lineB%22&start=1702152039 Delete request URL: http://localhost:44899/loki/api/v1/delete?end=1702324839&query=%7Bdeletion_type%3D%22filter_no_match%22%7D+%7C%3D+%22foo%22&start=1702152039 Delete request URL: http://localhost:44899/loki/api/v1/delete?end=1702324239&query=%7Bdeletion_type%3D%22partially_by_time%22%7D&start=1702152039 Delete request URL: http://localhost:44899/loki/api/v1/delete?end=1702324839&query=%7Bdeletion_type%3D%22whole%22%7D&start=1702152039 Delete request URL: http://localhost:44899/loki/api/v1/delete?end=1702324839&query=%7Bdeletion_type%3D%22with_structured_metadata%22%7D+%7C+line%3D%22A%22&start=1702152039 time="2023-12-11T20:01:07Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:01:07Z" level=info msg="server listening on addresses" grpc="[::]:40773" http="[::]:36547" time="2023-12-11T20:01:25Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:01:26Z" level=info msg="server listening on addresses" grpc="[::]:42049" http="[::]:34403" time="2023-12-11T20:01:40Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:01:40Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:01:40Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:01:40Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:01:40Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:01:40Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:01:40Z" level=warning msg=gRPC duration=13.140899366s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:01:40Z" level=warning msg=gRPC duration=13.140914346s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:01:40Z" level=warning msg=gRPC duration=13.134552338s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:01:40Z" level=warning msg=gRPC duration=13.141181705s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:01:40Z" level=warning msg=gRPC duration=13.141115085s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:01:40Z" level=warning msg=gRPC duration=13.141864864s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:01:40Z" level=warning msg=gRPC duration=13.134454118s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:01:41Z" level=info msg="server listening on addresses" grpc="[::]:40169" http="[::]:46305" time="2023-12-11T20:01:42Z" level=info msg="server listening on addresses" grpc="[::]:45569" http="[::]:41725" time="2023-12-11T20:01:42Z" level=info msg="server listening on addresses" grpc="[::]:40905" http="[::]:39799" time="2023-12-11T20:01:43Z" level=info msg="server listening on addresses" grpc="[::]:41817" http="[::]:36001" time="2023-12-11T20:01:43Z" level=info msg="server listening on addresses" grpc="[::]:41253" http="[::]:33451" time="2023-12-11T20:01:44Z" level=info msg="server listening on addresses" grpc="[::]:33583" http="[::]:39439" time="2023-12-11T20:01:45Z" level=info msg="server listening on addresses" grpc="[::]:34347" http="[::]:33463" time="2023-12-11T20:01:50Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:01:50Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:01:50Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:01:50Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:01:50Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:01:50Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:01:50Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:01:50Z" level=warning msg=gRPC duration=4.979130661s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:01:50Z" level=warning msg=gRPC duration=4.98004572s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:01:50Z" level=warning msg=gRPC duration=4.972118955s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:01:50Z" level=warning msg=gRPC duration=4.979213181s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:01:50Z" level=warning msg=gRPC duration=4.971735855s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:01:50Z" level=warning msg=gRPC duration=4.970964217s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:01:50Z" level=warning msg=gRPC duration=4.980260199s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:01:50Z" level=warning msg=gRPC duration=4.971689516s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:01:50Z" level=warning msg=gRPC duration=4.981112297s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:01:52Z" level=info msg="server listening on addresses" grpc="[::]:35943" http="[::]:39363" time="2023-12-11T20:01:53Z" level=info msg="server listening on addresses" grpc="[::]:45983" http="[::]:36517" time="2023-12-11T20:01:53Z" level=info msg="server listening on addresses" grpc="[::]:40329" http="[::]:45675" time="2023-12-11T20:01:54Z" level=info msg="server listening on addresses" grpc="[::]:44765" http="[::]:36121" time="2023-12-11T20:01:55Z" level=info msg="server listening on addresses" grpc="[::]:37345" http="[::]:39451" time="2023-12-11T20:01:56Z" level=info msg="server listening on addresses" grpc="[::]:42885" http="[::]:35053" time="2023-12-11T20:02:08Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:02:09Z" level=info msg="server listening on addresses" grpc="[::]:38343" http="[::]:42763" time="2023-12-11T20:02:17Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:02:19Z" level=info msg="server listening on addresses" grpc="[::]:45637" http="[::]:34235" time="2023-12-11T20:02:19Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:02:20Z" level=info msg="server listening on addresses" grpc="[::]:43873" http="[::]:37037" time="2023-12-11T20:02:21Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:02:22Z" level=info msg="server listening on addresses" grpc="[::]:41067" http="[::]:43291" time="2023-12-11T20:02:30Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:02:31Z" level=info msg="server listening on addresses" grpc="[::]:41107" http="[::]:46011" time="2023-12-11T20:02:31Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:02:33Z" level=info msg="server listening on addresses" grpc="[::]:34141" http="[::]:37021" time="2023-12-11T20:02:34Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:02:36Z" level=info msg="server listening on addresses" grpc="[::]:38305" http="[::]:36293" time="2023-12-11T20:02:44Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:02:45Z" level=info msg="server listening on addresses" grpc="[::]:40789" http="[::]:42245" time="2023-12-11T20:02:46Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:02:48Z" level=info msg="server listening on addresses" grpc="[::]:34695" http="[::]:45047" time="2023-12-11T20:02:53Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:02:53Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:02:53Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:02:53Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:02:53Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:02:53Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:02:55Z" level=info msg="server listening on addresses" grpc="[::]:43005" http="[::]:39135" time="2023-12-11T20:02:56Z" level=info msg="server listening on addresses" grpc="[::]:39983" http="[::]:40133" time="2023-12-11T20:02:56Z" level=info msg="server listening on addresses" grpc="[::]:38045" http="[::]:34497" time="2023-12-11T20:02:57Z" level=info msg="server listening on addresses" grpc="[::]:36643" http="[::]:41133" time="2023-12-11T20:02:57Z" level=info msg="server listening on addresses" grpc="[::]:44387" http="[::]:39339" time="2023-12-11T20:02:59Z" level=info msg="server listening on addresses" grpc="[::]:36627" http="[::]:36127" time="2023-12-11T20:03:03Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:05Z" level=info msg="server listening on addresses" grpc="[::]:38775" http="[::]:34153" time="2023-12-11T20:03:05Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:07Z" level=info msg="server listening on addresses" grpc="[::]:45613" http="[::]:39593" time="2023-12-11T20:03:12Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:12Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:12Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:12Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:12Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:12Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:14Z" level=info msg="server listening on addresses" grpc="[::]:44465" http="[::]:43411" time="2023-12-11T20:03:14Z" level=info msg="server listening on addresses" grpc="[::]:36599" http="[::]:46011" time="2023-12-11T20:03:15Z" level=info msg="server listening on addresses" grpc="[::]:40267" http="[::]:36675" time="2023-12-11T20:03:16Z" level=info msg="server listening on addresses" grpc="[::]:36241" http="[::]:35351" time="2023-12-11T20:03:17Z" level=info msg="server listening on addresses" grpc="[::]:42697" http="[::]:42183" time="2023-12-11T20:03:17Z" level=info msg="server listening on addresses" grpc="[::]:45293" http="[::]:34037" time="2023-12-11T20:03:22Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:25Z" level=info msg="server listening on addresses" grpc="[::]:46807" http="[::]:44129" time="2023-12-11T20:03:25Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:27Z" level=info msg="server listening on addresses" grpc="[::]:37383" http="[::]:38929" time="2023-12-11T20:03:35Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:35Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:35Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:35Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:35Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:35Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:37Z" level=info msg="server listening on addresses" grpc="[::]:41769" http="[::]:41951" time="2023-12-11T20:03:37Z" level=info msg="server listening on addresses" grpc="[::]:36257" http="[::]:34907" time="2023-12-11T20:03:38Z" level=info msg="server listening on addresses" grpc="[::]:42455" http="[::]:42037" time="2023-12-11T20:03:38Z" level=info msg="server listening on addresses" grpc="[::]:44871" http="[::]:43077" time="2023-12-11T20:03:39Z" level=info msg="server listening on addresses" grpc="[::]:44753" http="[::]:44527" time="2023-12-11T20:03:39Z" level=info msg="server listening on addresses" grpc="[::]:37757" http="[::]:43449" time="2023-12-11T20:03:44Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:44Z" level=info msg="server listening on addresses" grpc="[::]:46653" http="[::]:42245" time="2023-12-11T20:03:45Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:47Z" level=info msg="server listening on addresses" grpc="[::]:41637" http="[::]:34625" time="2023-12-11T20:03:53Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:53Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:53Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:53Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:53Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:53Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:03:53Z" level=warning msg=gRPC duration=5.667613966s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:03:53Z" level=warning msg=gRPC duration=5.673742814s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:03:53Z" level=warning msg=gRPC duration=5.673763774s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:03:53Z" level=warning msg=gRPC duration=5.668075365s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:03:53Z" level=warning msg=gRPC duration=5.674232273s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:03:53Z" level=warning msg=gRPC duration=5.674167693s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:03:53Z" level=warning msg=gRPC duration=5.675122411s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:03:53Z" level=warning msg=gRPC duration=5.669371123s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:03:53Z" level=warning msg=gRPC duration=5.669627062s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:03:53Z" level=warning msg=gRPC duration=5.668571405s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:03:55Z" level=info msg="server listening on addresses" grpc="[::]:45031" http="[::]:46877" time="2023-12-11T20:03:55Z" level=info msg="server listening on addresses" grpc="[::]:34683" http="[::]:34857" time="2023-12-11T20:03:56Z" level=info msg="server listening on addresses" grpc="[::]:41215" http="[::]:34493" time="2023-12-11T20:03:56Z" level=info msg="server listening on addresses" grpc="[::]:37093" http="[::]:36899" time="2023-12-11T20:03:56Z" level=info msg="server listening on addresses" grpc="[::]:40937" http="[::]:46815" time="2023-12-11T20:03:57Z" level=info msg="server listening on addresses" grpc="[::]:38967" http="[::]:40589" time="2023-12-11T20:04:07Z" level=info msg="server listening on addresses" grpc="[::]:45217" http="[::]:42289" time="2023-12-11T20:04:16Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:16Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:16Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:16Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:16Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:16Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:16Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:16Z" level=warning msg=gRPC duration=5.224767529s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:04:16Z" level=warning msg=gRPC duration=5.225046668s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:04:18Z" level=info msg="server listening on addresses" grpc="[::]:39203" http="[::]:44435" time="2023-12-11T20:04:18Z" level=info msg="server listening on addresses" grpc="[::]:41389" http="[::]:41791" time="2023-12-11T20:04:19Z" level=info msg="server listening on addresses" grpc="[::]:44965" http="[::]:43377" time="2023-12-11T20:04:19Z" level=info msg="server listening on addresses" grpc="[::]:40531" http="[::]:33397" time="2023-12-11T20:04:20Z" level=info msg="server listening on addresses" grpc="[::]:37849" http="[::]:37449" time="2023-12-11T20:04:20Z" level=info msg="server listening on addresses" grpc="[::]:44869" http="[::]:39985" time="2023-12-11T20:04:21Z" level=info msg="server listening on addresses" grpc="[::]:46765" http="[::]:33281" time="2023-12-11T20:04:21Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:23Z" level=info msg="server listening on addresses" grpc="[::]:44991" http="[::]:39891" time="2023-12-11T20:04:29Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:29Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:29Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:29Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:29Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:29Z" level=warning msg=gRPC duration=8.367943346s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:04:29Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:29Z" level=warning msg=gRPC duration=8.368388224s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:04:29Z" level=warning msg=gRPC duration=8.369761882s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:04:29Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:29Z" level=warning msg=gRPC duration=8.368785584s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:04:29Z" level=warning msg=gRPC duration=8.369282443s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:04:29Z" level=warning msg=gRPC duration=8.369442963s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:04:29Z" level=warning msg=gRPC duration=8.369084564s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:04:29Z" level=warning msg=gRPC duration=8.369626562s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:04:29Z" level=warning msg=gRPC duration=8.370028821s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop time="2023-12-11T20:04:29Z" level=warning msg=gRPC duration=8.369409523s err="queue is stopped" method=/schedulerpb.SchedulerForQuerier/QuerierLoop --- FAIL: TestQueryTSDB_WithCachedPostings (12.09s) loki_micro_services_test.go:739: Error Trace: /build/loki/src/loki-2.9.2/integration/loki_micro_services_test.go:739 /build/loki/src/loki-2.9.2/integration/loki_micro_services_test.go:686 Error: Not equal: expected: 1 actual : 2 Test: TestQueryTSDB_WithCachedPostings time="2023-12-11T20:04:30Z" level=info msg="server listening on addresses" grpc="[::]:40923" http="[::]:33871" time="2023-12-11T20:04:33Z" level=info msg="server listening on addresses" grpc="[::]:40219" http="[::]:44497" time="2023-12-11T20:04:33Z" level=info msg="server listening on addresses" grpc="[::]:39671" http="[::]:35283" time="2023-12-11T20:04:39Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:39Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:39Z" level=warning msg=gRPC duration=5.917601688s err="queue is stopped" method=/frontend.Frontend/Process time="2023-12-11T20:04:39Z" level=warning msg=gRPC duration=5.920365383s err="queue is stopped" method=/frontend.Frontend/Process time="2023-12-11T20:04:39Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:39Z" level=warning msg=gRPC duration=5.920823722s err="queue is stopped" method=/frontend.Frontend/Process time="2023-12-11T20:04:39Z" level=warning msg=gRPC duration=5.921429621s err="queue is stopped" method=/frontend.Frontend/Process time="2023-12-11T20:04:39Z" level=warning msg=gRPC duration=5.92177472s err="queue is stopped" method=/frontend.Frontend/Process time="2023-12-11T20:04:39Z" level=warning msg=gRPC duration=5.92212822s err="queue is stopped" method=/frontend.Frontend/Process time="2023-12-11T20:04:39Z" level=warning msg=gRPC duration=5.921995039s err="queue is stopped" method=/frontend.Frontend/Process time="2023-12-11T20:04:39Z" level=warning msg=gRPC duration=5.922396799s err="queue is stopped" method=/frontend.Frontend/Process time="2023-12-11T20:04:39Z" level=warning msg=gRPC duration=5.923141678s err="queue is stopped" method=/frontend.Frontend/Process time="2023-12-11T20:04:39Z" level=warning msg=gRPC duration=5.922402959s err="queue is stopped" method=/frontend.Frontend/Process time="2023-12-11T20:04:41Z" level=info msg="server listening on addresses" grpc="[::]:33947" http="[::]:37453" time="2023-12-11T20:04:41Z" level=info msg="server listening on addresses" grpc="[::]:43527" http="[::]:34109" time="2023-12-11T20:04:43Z" level=info msg="server listening on addresses" grpc="[::]:45211" http="[::]:40949" time="2023-12-11T20:04:48Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:48Z" level=info msg="=== Handler.Stop()'d ===" time="2023-12-11T20:04:48Z" level=info msg="=== Handler.Stop()'d ===" panic: test timed out after 10m0s running tests: TestRemoteRuleEval (5m49s) goroutine 189502 [running]: testing.(*M).startAlarm.func1() /usr/lib/go/src/testing/testing.go:2259 +0x304 created by time.goFunc /usr/lib/go/src/time/sleep.go:176 +0x50 goroutine 1 [chan receive, 5 minutes]: testing.(*T).Run(0x3f48202d00, {0x214632f?, 0xa7968?}, 0x2209288) /usr/lib/go/src/testing/testing.go:1649 +0x380 testing.runTests.func1(0x1a0?) /usr/lib/go/src/testing/testing.go:2054 +0x54 testing.tRunner(0x3f48202d00, 0x3f48d9fc68) /usr/lib/go/src/testing/testing.go:1595 +0x104 testing.runTests(0x3f486b8280?, {0x3daf9e0, 0xd, 0xd}, {0x2049c?, 0x3f48ce0b10?, 0x3e5fce0?}) /usr/lib/go/src/testing/testing.go:2052 +0x414 testing.(*M).Run(0x3f486b8280) /usr/lib/go/src/testing/testing.go:1925 +0x564 main.main() _testmain.go:71 +0x1a8 goroutine 21 [select, 10 minutes]: github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1() /build/loki/src/loki-2.9.2/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:375 +0x94 created by github.com/baidubce/bce-sdk-go/util/log.NewLogger in goroutine 1 /build/loki/src/loki-2.9.2/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:368 +0x134 goroutine 152 [select]: go.opencensus.io/stats/view.(*worker).start(0x3f48795e80) /build/loki/src/loki-2.9.2/vendor/go.opencensus.io/stats/view/worker.go:292 +0x94 created by go.opencensus.io/stats/view.init.0 in goroutine 1 /build/loki/src/loki-2.9.2/vendor/go.opencensus.io/stats/view/worker.go:34 +0x98 goroutine 138968 [select, 6 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcd93140, {0x283dd10, 0x3eda5f34a0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 138892 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 185984 [semacquire, 5 minutes]: sync.runtime_Semacquire(0x10?) /usr/lib/go/src/runtime/sema.go:62 +0x38 sync.(*WaitGroup).Wait(0x3edcded128) /usr/lib/go/src/sync/waitgroup.go:116 +0x6c github.com/grafana/loki/integration/cluster.(*Cluster).stop(0x3edcded100, 0x1) /build/loki/src/loki-2.9.2/integration/cluster/cluster.go:239 +0x4e0 github.com/grafana/loki/integration/cluster.(*Cluster).Cleanup(...) /build/loki/src/loki-2.9.2/integration/cluster/cluster.go:212 github.com/grafana/loki/integration.testRuleEval.func1() /build/loki/src/loki-2.9.2/integration/loki_rule_eval_test.go:39 +0x30 testing.(*common).Cleanup.func1() /usr/lib/go/src/testing/testing.go:1169 +0x118 testing.(*common).runCleanup(0x3ebdbfb520, 0x6?) /usr/lib/go/src/testing/testing.go:1347 +0xfc testing.tRunner.func2() /usr/lib/go/src/testing/testing.go:1589 +0x30 testing.tRunner(0x3ebdbfb520, 0x2209288) /usr/lib/go/src/testing/testing.go:1601 +0x130 created by testing.(*T).Run in goroutine 1 /usr/lib/go/src/testing/testing.go:1648 +0x36c goroutine 315 [select]: github.com/grafana/dskit/kv/consul.(*mockKV).loop(0x3f48c89900) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/kv/consul/mock.go:87 +0xb4 created by github.com/grafana/dskit/kv/consul.NewInMemoryClientWithConfig in goroutine 292 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/kv/consul/mock.go:58 +0x248 goroutine 167046 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f48ec8ab8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 167045 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 43025 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3eded0d4a0}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3f4ae940a0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 42783 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 392 [chan receive, 9 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x0?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 391 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 168 [syscall, 9 minutes]: os/signal.signal_recv() /usr/lib/go/src/runtime/sigqueue.go:152 +0x34 os/signal.loop() /usr/lib/go/src/os/signal/signal_unix.go:23 +0x20 created by os/signal.Notify.func1.1 in goroutine 385 /usr/lib/go/src/os/signal/signal.go:151 +0x2c goroutine 186709 [select, 5 minutes]: github.com/grafana/dskit/services.(*BasicService).awaitState(0x3f4bea8120?, {0x283dbf8, 0x3e914e0}, 0x4, 0x3ebd36ea80) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:294 +0xa0 github.com/grafana/dskit/services.(*BasicService).AwaitTerminated(0x8?, {0x283dbf8?, 0x3e914e0?}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:290 +0x48 github.com/grafana/dskit/modules.(*moduleService).waitForModulesToStop(0x3ed8d08550) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/modules/module_service.go:130 +0x228 github.com/grafana/dskit/modules.(*moduleService).stop(0x3ed8d08550, {0x0?, 0x0?}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/modules/module_service.go:102 +0x60 github.com/grafana/dskit/services.(*BasicService).main(0x3eb96cec80) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:207 +0x254 created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 186565 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 187277 [select]: github.com/grafana/loki/pkg/ruler.(*MemStore).run(0x3ebc94ae70) /build/loki/src/loki-2.9.2/pkg/ruler/memstore.go:133 +0x94 created by github.com/grafana/loki/pkg/ruler.(*MemStore).Start in goroutine 187109 /build/loki/src/loki-2.9.2/pkg/ruler/memstore.go:99 +0xb0 goroutine 42911 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f4950e3d8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 42910 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 495 [chan receive, 9 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f48a1b730?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 494 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 52989 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f486474c0, {0x283dd10, 0x3f4a2bdb80}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 52931 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 185695 [chan receive, 5 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3ebb58a330?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 185694 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 187164 [chan receive]: github.com/grafana/loki/pkg/ruler/storage/wal.(*Storage).recordSize(0x3ebb65e0e0) /build/loki/src/loki-2.9.2/pkg/ruler/storage/wal/wal.go:507 +0x90 created by github.com/grafana/loki/pkg/ruler/storage/wal.NewStorage in goroutine 187262 /build/loki/src/loki-2.9.2/pkg/ruler/storage/wal/wal.go:114 +0x620 goroutine 118742 [select, 7 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed842bcc0, {0x283dd10, 0x3f4adab9a0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 118649 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 97883 [select, 7 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed9602de0, {0x283dd10, 0x3f49f75f40}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 97454 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 147652 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f4be2bfa8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 147651 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 186547 [select, 5 minutes]: github.com/grafana/loki/pkg/loki.(*Loki).initServer.NewServerService.func4({0x283dd10, 0x3f48c7d090}) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1462 +0xec github.com/grafana/dskit/services.(*BasicService).main(0x3eb96921e0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 186712 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 185509 [select]: github.com/grafana/loki/pkg/ruler.(*MemStore).run(0x3f48f02540) /build/loki/src/loki-2.9.2/pkg/ruler/memstore.go:133 +0x94 created by github.com/grafana/loki/pkg/ruler.(*MemStore).Start in goroutine 184317 /build/loki/src/loki-2.9.2/pkg/ruler/memstore.go:99 +0xb0 goroutine 187091 [select, 5 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f4bfab8c0, {0x283dd10, 0x3eded0c960}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 186839 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 117798 [select]: github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0x3f4ad41110, 0x3eb9be1440?, 0x3f48e9ca80?) /build/loki/src/loki-2.9.2/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd0 created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache in goroutine 117628 /build/loki/src/loki-2.9.2/pkg/storage/chunk/cache/fifo_cache.go:214 +0xc94 goroutine 86142 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f49da6890?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 86141 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 147913 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f4be2bfa8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 147912 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 43132 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3edda64620, {0x283dd10, 0x3f48868e60}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 43131 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 186616 [chan receive, 5 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:344 +0x60 created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 186565 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:343 +0x1d0 goroutine 51945 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f496f7198?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 51944 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 9978 [chan receive, 9 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f4950e3d8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 9977 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 86959 [select, 7 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed94251e0, {0x283dd10, 0x3f4b041590}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 86958 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 772 [chan receive, 9 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f48785a40?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 771 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 29265 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3edd40c960}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3eddb5cfa0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 29195 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 186223 [chan receive, 5 minutes]: github.com/grafana/loki/pkg/loki.(*Loki).initCacheGenerationLoader.NewIdleService.func2({0x283dd10?, 0x3ebd0e5220?}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:13 +0x48 github.com/grafana/dskit/services.(*BasicService).main(0x3eb9692b40) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 186708 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 107924 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3edb968410}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3f4a1b65a0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 107879 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 118021 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3eba379380?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 118020 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 86960 [select, 7 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed9425240, {0x283dd10, 0x3f4b0415e0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 86958 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 75826 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3ed9ed1218?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 75825 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 504 [chan receive, 9 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x0?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 503 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 76846 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed8defca0, {0x283dd10, 0x3f4946fcc0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 76176 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 166070 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f48ec8ab8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 166069 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 148549 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x18ba458?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 148548 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 129093 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f4948deb8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 129092 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 510 [chan receive, 9 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x0?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 509 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 185373 [select, 5 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed9e3a040, {0x283dd10, 0x3ed9066f00}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 185144 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 186710 [select, 5 minutes]: github.com/grafana/dskit/services.(*BasicService).awaitState(0x3f49f509c0?, {0x283dbf8, 0x3e914e0}, 0x4, 0x3ebd36f9e0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:294 +0xa0 github.com/grafana/dskit/services.(*BasicService).AwaitTerminated(0x8?, {0x283dbf8?, 0x3e914e0?}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:290 +0x48 github.com/grafana/dskit/modules.(*moduleService).waitForModulesToStop(0x3ed8d09bd0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/modules/module_service.go:130 +0x228 github.com/grafana/dskit/modules.(*moduleService).stop(0x3ed8d09bd0, {0x0?, 0x0?}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/modules/module_service.go:102 +0x60 github.com/grafana/dskit/services.(*BasicService).main(0x3edcdee1e0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:207 +0x254 created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 186565 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 118393 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x18ba458?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 118392 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 106921 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3ed950cc98?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 106920 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 187156 [chan receive, 5 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x0?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 187155 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 75728 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3eb882cf50}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3f4a2d15e0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 75566 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 76189 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x0?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 76188 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 1065 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3f4a2f6b90}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3f499fd680) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 1031 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 173518 [select, 5 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f4a2dc5a0, {0x283dd10, 0x3ebd1a6550}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 173517 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 128418 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f4948deb8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 126625 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 97665 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x18ba458?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 97664 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 172531 [select]: github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0x3edb1f7450, 0x3eba0d1788?, 0x3f4bcdcc60?) /build/loki/src/loki-2.9.2/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd0 created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache in goroutine 170417 /build/loki/src/loki-2.9.2/pkg/storage/chunk/cache/fifo_cache.go:214 +0xc94 goroutine 10682 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f48c8ab80, {0x283dd10, 0x3ed9829d60}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 10601 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 19876 [chan receive, 9 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f4950e3d8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 19875 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 42048 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f48b520c0?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 42047 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 148208 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x18ba458?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 148207 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 41937 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3edf3bc300?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 41936 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 186224 [select, 5 minutes]: github.com/grafana/dskit/kv/memberlist.(*KVInitService).running(0x3ed9a06b80, {0x283dd10?, 0x3ebd0e5360?}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/kv/memberlist/kv_init_service.go:70 +0xa8 github.com/grafana/dskit/services.(*BasicService).main(0x3eb96cee60) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 186710 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 128035 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3ed83a3aa0?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 128034 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 86005 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3ed950cc98?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 86004 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 10388 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3f4b0139a0}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3ed9529f40) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 10354 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 138821 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x18ba458?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 138820 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 187124 [select, 5 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc9a2000) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1150 +0x1a0 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 187122 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/transport/http2_server.go:339 +0x14fc goroutine 1020 [chan receive, 9 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x23?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 1019 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 186565 [select, 5 minutes]: github.com/grafana/dskit/services.(*Manager).AwaitStopped(0x3f48e9df80, {0x283dbf8, 0x3e914e0}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/manager.go:145 +0x88 github.com/grafana/loki/pkg/loki.(*Loki).Run(0x3eb8d42000, {0x3ed94502f0?}) /build/loki/src/loki-2.9.2/pkg/loki/loki.go:549 +0xad4 github.com/grafana/loki/integration/cluster.(*Component).run.func2() /build/loki/src/loki-2.9.2/integration/cluster/cluster.go:450 +0xa8 created by github.com/grafana/loki/integration/cluster.(*Component).run in goroutine 185984 /build/loki/src/loki-2.9.2/integration/cluster/cluster.go:447 +0x4ac goroutine 187123 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ed9452050, 0x1) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:418 +0x124 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f48b927e0) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:552 +0x90 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/transport/http2_server.go:336 +0xdc created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 187122 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x14bc goroutine 187080 [IO wait]: internal/poll.runtime_pollWait(0x3f446ef390, 0x72) /usr/lib/go/src/runtime/netpoll.go:343 +0xc4 internal/poll.(*pollDesc).wait(0x3f4a3bc000?, 0x3ebc6b4000?, 0x0) /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x44 internal/poll.(*pollDesc).waitRead(...) /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0x3f4a3bc000, {0x3ebc6b4000, 0x8000, 0x8000}) /usr/lib/go/src/internal/poll/fd_unix.go:164 +0x23c net.(*netFD).Read(0x3f4a3bc000, {0x3ebc6b4000?, 0x47b8520?, 0x2eb28?}) /usr/lib/go/src/net/fd_posix.go:55 +0x4c net.(*conn).Read(0x3ebc69c018, {0x3ebc6b4000?, 0x282cbb8?, 0x8bc010601?}) /usr/lib/go/src/net/net.go:179 +0x58 bufio.(*Reader).Read(0x3ed8a80480, {0x3ebc6aa040, 0x9, 0x18?}) /usr/lib/go/src/bufio/bufio.go:244 +0x1cc io.ReadAtLeast({0x281b858, 0x3ed8a80480}, {0x3ebc6aa040, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd4 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc6aa040, 0x9, 0x3f48d562e8?}, {0x281b858?, 0x3ed8a80480?}) /build/loki/src/loki-2.9.2/vendor/golang.org/x/net/http2/frame.go:237 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc6aa000) /build/loki/src/loki-2.9.2/vendor/golang.org/x/net/http2/frame.go:498 +0x84 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3edd674000, 0x3f48e177a8?) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1595 +0x1f4 created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 187036 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/transport/http2_client.go:397 +0x1788 goroutine 1254 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f4af0dc80, {0x283dd10, 0x3f4ad32910}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1408 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 187081 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3eb9eae730, 0x1) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:418 +0x124 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f48ce8070) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:552 +0x90 google.golang.org/grpc/internal/transport.newHTTP2Client.func6() /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/transport/http2_client.go:451 +0x74 created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 187036 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/transport/http2_client.go:449 +0x1b98 goroutine 173585 [select, 6 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed9350500, {0x283dd10, 0x3ed96046e0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 173583 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 185392 [chan receive, 5 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3eb8aee748?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 185391 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 1255 [select, 9 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f4af0dce0, {0x283dd10, 0x3f4ad32960}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1408 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 10546 [chan receive, 9 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f489418c8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 9857 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 186714 [select, 5 minutes]: github.com/grafana/dskit/services.(*BasicService).awaitState(0x0?, {0x283dbf8, 0x3e914e0}, 0x4, 0x3ebd36f920) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:294 +0xa0 github.com/grafana/dskit/services.(*BasicService).AwaitTerminated(0x3ebd4aee60?, {0x283dbf8?, 0x3e914e0?}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:290 +0x48 github.com/grafana/dskit/services.StopAndAwaitTerminated({0x283dbf8, 0x3e914e0}, {0x284a770, 0x3ebd4aee60}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:122 +0x70 github.com/grafana/dskit/modules.(*moduleService).stop(0x3eb8aeedc0, {0x0?, 0x0?}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/modules/module_service.go:106 +0x188 github.com/grafana/dskit/services.(*BasicService).main(0x3ebd4af7c0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:207 +0x254 created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 186565 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 128344 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x1a2f168?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 128343 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 173152 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x18ba458?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 173151 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 52154 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x0?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 52153 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 157891 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3ed8314e58?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 157890 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 167266 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f4950e3d8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 167233 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 167130 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3f496f6410}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3ebc6dc6e0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 166392 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 186691 [chan receive, 5 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:344 +0x60 created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 186565 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:343 +0x1d0 goroutine 76845 [select, 7 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed8defc40, {0x283dd10, 0x3f4946fc70}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 76176 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 76600 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3f48b776d0}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3f496728c0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 76566 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 96752 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3ed950cc98?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 96751 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 186550 [IO wait, 5 minutes]: internal/poll.runtime_pollWait(0x3f4c0d4530, 0x72) /usr/lib/go/src/runtime/netpoll.go:343 +0xc4 internal/poll.(*pollDesc).wait(0x3f48d05c00?, 0xd0?, 0x0) /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x44 internal/poll.(*pollDesc).waitRead(...) /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0x3f48d05c00) /usr/lib/go/src/internal/poll/fd_unix.go:611 +0x260 net.(*netFD).accept(0x3f48d05c00) /usr/lib/go/src/net/fd_unix.go:172 +0x34 net.(*TCPListener).accept(0x3ed8967340) /usr/lib/go/src/net/tcpsock_posix.go:152 +0x34 net.(*TCPListener).Accept(0x3ed8967340) /usr/lib/go/src/net/tcpsock.go:315 +0x38 github.com/grafana/dskit/middleware.(*countingListener).Accept(0x3ed8967360) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/middleware/counting_listener.go:26 +0x3c net/http.(*Server).Serve(0x3f49036f00, {0x2835dd0, 0x3ed8967360}) /usr/lib/go/src/net/http/server.go:3056 +0x2f0 github.com/grafana/dskit/server.(*Server).Run.func2() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:489 +0x60 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 186548 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:486 +0xdc goroutine 86184 [select]: github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0x3ed906c410, 0x834d4?, 0x3f48753020?) /build/loki/src/loki-2.9.2/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd0 created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache in goroutine 86014 /build/loki/src/loki-2.9.2/pkg/storage/chunk/cache/fifo_cache.go:214 +0xc94 goroutine 187079 [select]: google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0x3edd674000) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1669 +0x140 created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 187036 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/transport/http2_client.go:387 +0x1724 goroutine 174229 [select, 6 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebd8f5400, {0x283dd10, 0x3ed90669b0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 170539 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 52786 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3edd317860?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 52657 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 187035 [select, 5 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcc7ba00, {0x283dd10, 0x3eb81bec80}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 186839 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 118433 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3eb8852eb0}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3eddb5c8c0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 117887 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 186695 [select]: github.com/grafana/dskit/runtimeconfig.(*Manager).loop(0x3f483d2bb0, {0x283dd10, 0x3ed90d44b0}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/runtimeconfig/manager.go:137 +0x1d4 github.com/grafana/dskit/services.(*BasicService).main(0x3eb9693860) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 186709 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 52002 [select]: github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0x3eda72d1e0, 0x3f4b03f620?, 0x3f4b0f79e0?) /build/loki/src/loki-2.9.2/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd0 created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache in goroutine 51961 /build/loki/src/loki-2.9.2/pkg/storage/chunk/cache/fifo_cache.go:214 +0xc94 goroutine 184973 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3eba534f00}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3f48c47680) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 184350 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 9793 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eded002c0, {0x283dd10, 0x3eded0c0a0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 25032 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 76019 [select]: github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0x3ed8da88f0, 0x3edeb0fa40?, 0x3ed8fcbda0?) /build/loki/src/loki-2.9.2/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd0 created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache in goroutine 75955 /build/loki/src/loki-2.9.2/pkg/storage/chunk/cache/fifo_cache.go:214 +0xc94 goroutine 147423 [select, 6 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed8e12ce0, {0x283dd10, 0x3ed8482f50}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 148577 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 42071 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x18ba458?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 42070 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 107107 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3ed950cc98?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 107106 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 62845 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed87bb800, {0x283dd10, 0x3ed87ecf50}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 53562 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 172477 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3ed881f8c8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 172476 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 167833 [select, 6 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f4bdb3a00, {0x283dd10, 0x3ed983cfa0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 167788 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 52314 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f48dba2a0?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 52313 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 128041 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f4948deb8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 128040 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 173059 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3ebdecb6e0?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 173058 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 137969 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x18ba458?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 137968 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 97884 [select, 7 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed9602e40, {0x283dd10, 0x3f49a98000}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 97454 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 52309 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f4950e3d8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 52308 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 43133 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3edda64680, {0x283dd10, 0x3f48868eb0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 43131 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 186549 [chan receive, 5 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x0?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 186548 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 19725 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3edbbc05a0}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3f49500dc0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 19691 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 186619 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3eb8aeee10}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3ebd4af860) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 186565 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 62707 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3ed950cc98?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 62706 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 10683 [select, 9 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f48c8abe0, {0x283dd10, 0x3ed9829db0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 10601 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 173742 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f4ae9a3c0?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 173741 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 186711 [select, 5 minutes]: github.com/grafana/dskit/services.(*BasicService).awaitState(0x3f491fb260?, {0x283dbf8, 0x3e914e0}, 0x4, 0x3ebd36f9e0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:294 +0xa0 github.com/grafana/dskit/services.(*BasicService).AwaitTerminated(0x8?, {0x283dbf8?, 0x3e914e0?}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:290 +0x48 github.com/grafana/dskit/modules.(*moduleService).waitForModulesToStop(0x3ebecb8aa0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/modules/module_service.go:130 +0x228 github.com/grafana/dskit/modules.(*moduleService).stop(0x3ebecb8aa0, {0x0?, 0x0?}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/modules/module_service.go:102 +0x60 github.com/grafana/dskit/services.(*BasicService).main(0x3edcdeef00) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:207 +0x254 created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 186565 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 157692 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3f4a2062d0}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3f4880e500) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 157645 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 29570 [select, 9 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f4969b5c0, {0x283dd10, 0x3f4a0bc4b0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 28976 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 62844 [select, 7 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed87bb7a0, {0x283dd10, 0x3ed87ecf00}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 53562 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 25106 [select, 9 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eded00320, {0x283dd10, 0x3eded0c0f0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 25032 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 42059 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f4950e3d8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 42058 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 158067 [select, 6 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3edbb4e1e0, {0x283dd10, 0x3ed8483680}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 158039 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 186689 [chan receive, 5 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:344 +0x60 created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 186565 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:343 +0x1d0 goroutine 107782 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x18ba458?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 107781 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 87098 [select, 7 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f48807a00, {0x283dd10, 0x3f4a274050}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 87016 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 186712 [select, 5 minutes]: github.com/grafana/dskit/services.(*BasicService).awaitState(0x3f49f50600?, {0x283dbf8, 0x3e914e0}, 0x4, 0x3ebd36f9e0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:294 +0xa0 github.com/grafana/dskit/services.(*BasicService).AwaitTerminated(0x8?, {0x283dbf8?, 0x3e914e0?}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:290 +0x48 github.com/grafana/dskit/modules.(*moduleService).waitForModulesToStop(0x3f4af15220) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/modules/module_service.go:130 +0x228 github.com/grafana/dskit/modules.(*moduleService).stop(0x3f4af15220, {0x0?, 0x0?}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/modules/module_service.go:102 +0x60 github.com/grafana/dskit/services.(*BasicService).main(0x3eb9692aa0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:207 +0x254 created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 186565 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 118743 [select, 7 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed842bd20, {0x283dd10, 0x3f4adab9f0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 118649 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 173422 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f48ec8ab8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 173421 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 52988 [select, 7 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f48647400, {0x283dd10, 0x3f4a2bdb30}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 52931 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 167834 [select, 6 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f4bdb3a60, {0x283dd10, 0x3ed983cff0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 167788 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 51645 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f496c5590?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 51644 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 87097 [select, 7 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f488079a0, {0x283dd10, 0x3edd40df90}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 87016 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 76697 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x0?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 76696 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 128967 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3ed8982be0}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3eb9692640) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 128933 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 85923 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3eb8b7cf00}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3eda950320) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 85665 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 147422 [select, 6 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed8e12c80, {0x283dd10, 0x3ed8482f00}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 148577 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 173210 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3ed848e000?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 173209 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 41905 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f48f35170?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 41904 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 185374 [select, 5 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed9e3a0c0, {0x283dd10, 0x3ed9067040}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 185144 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 62611 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3eda2a5d10}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3f4b022820) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 58801 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 186534 [select, 5 minutes]: github.com/grafana/dskit/services.(*Manager).AwaitStopped(0x3eda2f0c00, {0x283dbf8, 0x3e914e0}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/manager.go:145 +0x88 github.com/grafana/dskit/services.StopManagerAndAwaitStopped({0x283dbf8?, 0x3e914e0?}, 0x3eda2f0c00?) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/manager.go:343 +0x7c github.com/grafana/loki/pkg/lokifrontend/frontend/v1.(*Frontend).stopping(...) /build/loki/src/loki-2.9.2/pkg/lokifrontend/frontend/v1/frontend.go:139 github.com/grafana/dskit/services.(*BasicService).main(0x3f4880e280) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:207 +0x254 created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 186533 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 173668 [select]: github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0x3f4943ca90, 0x3f49835620?, 0x3ed9457140?) /build/loki/src/loki-2.9.2/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd0 created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache in goroutine 173323 /build/loki/src/loki-2.9.2/pkg/storage/chunk/cache/fifo_cache.go:214 +0xc94 goroutine 29247 [chan receive, 9 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x0?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 29246 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 174228 [select, 6 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebd8f5380, {0x283dd10, 0x3ed9066960}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 170539 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 107165 [select]: github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0x3ed9853ad0, 0x3f4a3d0f88?, 0x3eda651260?) /build/loki/src/loki-2.9.2/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd0 created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache in goroutine 107124 /build/loki/src/loki-2.9.2/pkg/storage/chunk/cache/fifo_cache.go:214 +0xc94 goroutine 108075 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3ed950cc98?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 108074 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 29553 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f4969b560, {0x283dd10, 0x3f4a0bc460}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 28976 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 138668 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3ed940ee10}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3ed84eb9a0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 138634 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 85628 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x18ba458?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 85627 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 43325 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f490d5f40, {0x283dd10, 0x3f49f75770}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 43207 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 172382 [select]: github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0x3ed9f85c70, 0x3edb203f88?, 0x3edd7f7440?) /build/loki/src/loki-2.9.2/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd0 created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache in goroutine 172334 /build/loki/src/loki-2.9.2/pkg/storage/chunk/cache/fifo_cache.go:214 +0xc94 goroutine 147249 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x1bca0e0?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 147248 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 43326 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f490d5fa0, {0x283dd10, 0x3f49f757c0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 43207 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 186967 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3f4b11c8c0}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3ebc4d5860) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 186839 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 186535 [chan receive, 5 minutes]: github.com/grafana/dskit/services.(*Manager).AddListener.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/manager.go:244 +0x60 created by github.com/grafana/dskit/services.(*Manager).AddListener in goroutine 186534 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/manager.go:243 +0x1e0 goroutine 148435 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3ed99a2cd0}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3eba110500) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 147921 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 96934 [select]: github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0x3f49a88f70, 0x3eda003380?, 0x3f4ae9b320?) /build/loki/src/loki-2.9.2/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd0 created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache in goroutine 96845 /build/loki/src/loki-2.9.2/pkg/storage/chunk/cache/fifo_cache.go:214 +0xc94 goroutine 108269 [select, 7 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed8029500, {0x283dd10, 0x3f4b01c410}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 107871 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 185402 [select, 5 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed9c94740, {0x283dd10, 0x3f48c7c3c0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 185401 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 107714 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3ed950cc98?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 107681 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 157057 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f48ec8ab8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 157056 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 42064 [chan receive, 8 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3edb8dad30?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 42063 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 185552 [chan receive]: github.com/grafana/loki/pkg/ruler/storage/wal.(*Storage).recordSize(0x3eb93eb340) /build/loki/src/loki-2.9.2/pkg/ruler/storage/wal/wal.go:507 +0x90 created by github.com/grafana/loki/pkg/ruler/storage/wal.NewStorage in goroutine 185494 /build/loki/src/loki-2.9.2/pkg/ruler/storage/wal/wal.go:114 +0x620 goroutine 172245 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f49189b18?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 172329 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 186533 [select, 5 minutes]: github.com/grafana/dskit/services.(*BasicService).awaitState(0x18f4e2c?, {0x283dbf8, 0x3e914e0}, 0x4, 0x3ebd36f260) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:294 +0xa0 github.com/grafana/dskit/services.(*BasicService).AwaitTerminated(0x281b498?, {0x283dbf8?, 0x3e914e0?}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:290 +0x48 github.com/grafana/dskit/services.StopAndAwaitTerminated({0x283dbf8, 0x3e914e0}, {0x3f44798bf8, 0x3eda693dd0}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:122 +0x70 github.com/grafana/loki/pkg/loki.(*Loki).initQueryFrontend.func4({0x3ebd4aee60?, 0x2?}) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:914 +0x6c github.com/grafana/dskit/services.(*BasicService).main(0x3ebd4aee60) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:207 +0x254 created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 186714 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 186575 [select]: github.com/grafana/loki/pkg/storage/stores/indexshipper/compactor/generationnumber.(*GenNumberLoader).loop(0x3f49b010c0) /build/loki/src/loki-2.9.2/pkg/storage/stores/indexshipper/compactor/generationnumber/gennumber_loader.go:53 +0xb8 created by github.com/grafana/loki/pkg/storage/stores/indexshipper/compactor/generationnumber.NewGenNumberLoader in goroutine 186565 /build/loki/src/loki-2.9.2/pkg/storage/stores/indexshipper/compactor/generationnumber/gennumber_loader.go:44 +0x140 goroutine 173253 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3eb835f4f0}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3eb97de3c0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 173271 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 165822 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3eda686060?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 165821 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 129193 [select, 6 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3edda99420, {0x283dd10, 0x3ed8a79720}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 129144 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 187090 [select, 5 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f4bfab840, {0x283dd10, 0x3eded0c910}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 186839 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 97524 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3edb968370}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3eba110140) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 97490 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 52699 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3f4948c7d0}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3f49a8fb80) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 52636 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 86762 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3edbbc0230}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3eb97df5e0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 86728 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 173584 [select, 6 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed9350480, {0x283dd10, 0x3ed9604690}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 173583 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 189499 [select]: google.golang.org/grpc.(*addrConn).resetTransport(0x3ebcd162c0) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/clientconn.go:1352 +0x324 google.golang.org/grpc.(*addrConn).connect(0x3ebcd162c0) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/clientconn.go:978 +0xb0 created by google.golang.org/grpc.(*acBalancerWrapper).Connect in goroutine 185310 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:398 +0x6c goroutine 185310 [select]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed87ba2a0, {0x283dd10, 0x3ebc172820}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 185144 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 185403 [select, 5 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed9c947a0, {0x283dd10, 0x3f48c7c410}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 185401 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 129194 [select, 7 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3edda99480, {0x283dd10, 0x3ed8a79770}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 129144 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 187034 [select, 5 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcc7b9a0, {0x283dd10, 0x3eb81bec30}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 186839 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 185071 [chan receive, 5 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f4bfb1e80?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 185070 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 185173 [select]: github.com/grafana/dskit/ring/client.NewPool.NewTimerService.func1({0x283dd10, 0x3ebd1d5680}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:31 +0xe0 github.com/grafana/dskit/services.(*BasicService).main(0x3ebad7dc20) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 185144 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 189252 [chan send, 5 minutes]: github.com/grafana/loki/pkg/storage.(*AsyncStore).GetChunkRefs.func1() /build/loki/src/loki-2.9.2/pkg/storage/async_store.go:76 +0xe0 created by github.com/grafana/loki/pkg/storage.(*AsyncStore).GetChunkRefs in goroutine 186921 /build/loki/src/loki-2.9.2/pkg/storage/async_store.go:73 +0x224 goroutine 107245 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3ed950cc98?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 107244 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 186685 [chan receive, 5 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:344 +0x60 created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 186565 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:343 +0x1d0 goroutine 166938 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f4be2bfa8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 166937 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 186532 [chan receive, 5 minutes]: github.com/grafana/loki/pkg/loki.(*Loki).initQueryFrontendTripperware.NewIdleService.func1({0x283dd10?, 0x3ed92a2410?}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/services.go:13 +0x48 github.com/grafana/dskit/services.(*BasicService).main(0x3edcdee640) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x19c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 186711 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 173519 [select, 5 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f4a2dc600, {0x283dd10, 0x3ebd1a65a0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 173517 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 186706 [chan receive, 5 minutes]: github.com/grafana/dskit/services.(*Manager).AddListener.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/manager.go:244 +0x60 created by github.com/grafana/dskit/services.(*Manager).AddListener in goroutine 186565 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/manager.go:243 +0x1e0 goroutine 108270 [select, 7 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed8029560, {0x283dd10, 0x3f4b01c4b0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 107871 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 186708 [select, 5 minutes]: github.com/grafana/dskit/services.(*BasicService).awaitState(0x3f49f50720?, {0x283dbf8, 0x3e914e0}, 0x4, 0x3ebd36f9e0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:294 +0xa0 github.com/grafana/dskit/services.(*BasicService).AwaitTerminated(0x8?, {0x283dbf8?, 0x3e914e0?}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:290 +0x48 github.com/grafana/dskit/modules.(*moduleService).waitForModulesToStop(0x3f4af159a0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/modules/module_service.go:130 +0x228 github.com/grafana/dskit/modules.(*moduleService).stop(0x3f4af159a0, {0x0?, 0x0?}) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/modules/module_service.go:102 +0x60 github.com/grafana/dskit/services.(*BasicService).main(0x3eb96937c0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:207 +0x254 created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 186565 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 172454 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x0?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 172453 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 167759 [select, 6 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed95b74c0, {0x283dd10, 0x3edb506190}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 167758 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 138969 [select, 7 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcd931a0, {0x283dd10, 0x3eda5f34f0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 138892 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 185313 [select]: github.com/grafana/loki/pkg/storage/stores/indexshipper/compactor/deletion.(*deleteRequestsClient).updateLoop(0x3f4bf0ef60) /build/loki/src/loki-2.9.2/pkg/storage/stores/indexshipper/compactor/deletion/delete_requests_client.go:99 +0xbc created by github.com/grafana/loki/pkg/storage/stores/indexshipper/compactor/deletion.NewDeleteRequestsClient in goroutine 185144 /build/loki/src/loki-2.9.2/pkg/storage/stores/indexshipper/compactor/deletion/delete_requests_client.go:60 +0x1d0 goroutine 186688 [chan receive, 5 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:344 +0x60 created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 186565 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:343 +0x1d0 goroutine 185311 [select, 5 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed87ba300, {0x283dd10, 0x3ebc172870}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 185144 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 186536 [select, 5 minutes]: github.com/grafana/loki/pkg/scheduler/queue.contextCond.Wait({0x3ed9e8c2c0?, 0x0?}, {0x283dbf8, 0x3e914e0}) /build/loki/src/loki-2.9.2/pkg/scheduler/queue/queue.go:276 +0x188 github.com/grafana/loki/pkg/scheduler/queue.(*RequestQueue).stopping(0x3ebecb96d0, {0x0?, 0x0?}) /build/loki/src/loki-2.9.2/pkg/scheduler/queue/queue.go:200 +0xdc github.com/grafana/dskit/services.(*BasicService).main(0x3f4880e000) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:207 +0x254 created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 186534 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0xfc goroutine 166386 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f4ae0fc20?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 164289 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 127626 [chan receive, 7 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x3f4948deb8?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 127625 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 148249 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x0?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 148248 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 184579 [chan receive, 5 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x2851a00?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 184578 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 158068 [select, 6 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3edbb4e260, {0x283dd10, 0x3ed84836d0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 158039 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 187125 [IO wait]: internal/poll.runtime_pollWait(0x3f4c318598, 0x72) /usr/lib/go/src/runtime/netpoll.go:343 +0xc4 internal/poll.(*pollDesc).wait(0x3edcdec200?, 0x3ebc984000?, 0x0) /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x44 internal/poll.(*pollDesc).waitRead(...) /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0x3edcdec200, {0x3ebc984000, 0x8000, 0x8000}) /usr/lib/go/src/internal/poll/fd_unix.go:164 +0x23c net.(*netFD).Read(0x3edcdec200, {0x3ebc984000?, 0x3f4a284680?, 0x19768?}) /usr/lib/go/src/net/fd_posix.go:55 +0x4c net.(*conn).Read(0x3ebc982000, {0x3ebc984000?, 0x18?, 0x3?}) /usr/lib/go/src/net/net.go:179 +0x58 bufio.(*Reader).Read(0x3ed8744000, {0x3ebc99c040, 0x9, 0x3ebb28e000?}) /usr/lib/go/src/bufio/bufio.go:244 +0x1cc io.ReadAtLeast({0x281b858, 0x3ed8744000}, {0x3ebc99c040, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd4 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc99c040, 0x9, 0x8d8d3f79de?}, {0x281b858?, 0x3ed8744000?}) /build/loki/src/loki-2.9.2/vendor/golang.org/x/net/http2/frame.go:237 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc99c000) /build/loki/src/loki-2.9.2/vendor/golang.org/x/net/http2/frame.go:498 +0x84 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc9a2000, 0x1?, 0x0?) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/transport/http2_server.go:637 +0x13c google.golang.org/grpc.(*Server).serveStreams(0x3ed8dd8d20, {0x2851a00?, 0x3ebc9a2000}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/server.go:940 +0x1c0 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/server.go:882 +0x48 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 187122 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/server.go:881 +0x19c goroutine 186551 [IO wait, 5 minutes]: internal/poll.runtime_pollWait(0x3f4c0d3b80, 0x72) /usr/lib/go/src/runtime/netpoll.go:343 +0xc4 internal/poll.(*pollDesc).wait(0x3f48d05c80?, 0x3e8ad90?, 0x0) /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x44 internal/poll.(*pollDesc).waitRead(...) /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0x3f48d05c80) /usr/lib/go/src/internal/poll/fd_unix.go:611 +0x260 net.(*netFD).accept(0x3f48d05c80) /usr/lib/go/src/net/fd_unix.go:172 +0x34 net.(*TCPListener).accept(0x3ed8967380) /usr/lib/go/src/net/tcpsock_posix.go:152 +0x34 net.(*TCPListener).Accept(0x3ed8967380) /usr/lib/go/src/net/tcpsock.go:315 +0x38 github.com/grafana/dskit/middleware.(*countingListener).Accept(0x3ed89673a0) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/middleware/counting_listener.go:26 +0x3c google.golang.org/grpc.(*Server).Serve(0x3ed8dd8d20, {0x2835dd0?, 0x3ed89673a0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/server.go:814 +0x3f0 github.com/grafana/dskit/server.(*Server).Run.func3() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:508 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 186548 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:507 +0x198 goroutine 166587 [chan receive, 6 minutes]: github.com/grafana/loki/pkg/loki.ignoreSignalHandler.Loop(0x5ffe90?) /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1499 +0x2c github.com/grafana/dskit/server.(*Server).Run.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:479 +0x38 created by github.com/grafana/dskit/server.(*Server).Run in goroutine 166586 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:478 +0x8c goroutine 167760 [select, 6 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ed95b7540, {0x283dd10, 0x3edb5061e0}) /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:83 +0xf4 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 167758 /build/loki/src/loki-2.9.2/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:55 +0x148 goroutine 186687 [chan receive, 5 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:344 +0x60 created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 186565 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:343 +0x1d0 goroutine 186548 [chan receive, 5 minutes]: github.com/grafana/dskit/server.(*Server).Run(0x3edb219c00) /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/server/server.go:524 +0x258 github.com/grafana/loki/pkg/loki.(*Loki).initServer.NewServerService.func4.1() /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1459 +0x5c created by github.com/grafana/loki/pkg/loki.(*Loki).initServer.NewServerService.func4 in goroutine 186547 /build/loki/src/loki-2.9.2/pkg/loki/modules.go:1457 +0x94 goroutine 186686 [chan receive, 5 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func1() /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:344 +0x60 created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 186565 /build/loki/src/loki-2.9.2/vendor/github.com/grafana/dskit/services/basic_service.go:343 +0x1d0 goroutine 189186 [sync.Cond.Wait, 5 minutes]: sync.runtime_notifyListWait(0x3ed9e8c2d0, 0x32) /usr/lib/go/src/runtime/sema.go:527 +0x16c sync.(*Cond).Wait(0x3ed9e8c2c0) /usr/lib/go/src/sync/cond.go:70 +0xb0 github.com/grafana/loki/pkg/scheduler/queue.contextCond.Wait.func1() /build/loki/src/loki-2.9.2/pkg/scheduler/queue/queue.go:258 +0x44 created by github.com/grafana/loki/pkg/scheduler/queue.contextCond.Wait in goroutine 186536 /build/loki/src/loki-2.9.2/pkg/scheduler/queue/queue.go:254 +0xcc FAIL github.com/grafana/loki/integration 614.437s ok github.com/grafana/loki/pkg/analytics 32.563s ok github.com/grafana/loki/pkg/canary/comparator 7.811s ok github.com/grafana/loki/pkg/canary/writer 8.476s name uncompressed compressed ratio zstd_524 kB 34 MB 1.0 MB 32.297404 zstd_524 kB 34 MB 1.0 MB 32.291436 zstd_524 kB 34 MB 1.0 MB 32.291436 zstd_524 kB 33 MB 1.0 MB 31.756525 zstd_262 kB 26 MB 1.3 MB 20.325232 zstd_262 kB 26 MB 1.3 MB 20.320509 zstd_262 kB 26 MB 1.3 MB 20.320509 zstd_262 kB 26 MB 1.3 MB 19.949691 zstd_66 kB 28 MB 1.5 MB 19.027133 zstd_66 kB 28 MB 1.5 MB 19.010603 zstd_66 kB 28 MB 1.5 MB 19.010603 zstd_66 kB 28 MB 1.5 MB 18.663974 flate_524 kB 19 MB 1.1 MB 17.879185 flate_524 kB 19 MB 1.1 MB 17.877355 flate_524 kB 19 MB 1.1 MB 17.877355 gzip_524 kB 19 MB 1.1 MB 17.867884 gzip_524 kB 19 MB 1.1 MB 17.866057 gzip_524 kB 19 MB 1.1 MB 17.866057 flate_262 kB 23 MB 1.3 MB 17.717022 flate_262 kB 23 MB 1.3 MB 17.713432 flate_262 kB 23 MB 1.3 MB 17.713432 gzip_262 kB 23 MB 1.3 MB 17.694899 gzip_262 kB 23 MB 1.3 MB 17.691317 gzip_262 kB 23 MB 1.3 MB 17.691317 flate_524 kB 18 MB 1.0 MB 17.550032 gzip_524 kB 18 MB 1.0 MB 17.539012 flate_262 kB 23 MB 1.3 MB 17.394876 gzip_262 kB 23 MB 1.3 MB 17.373347 flate_66 kB 24 MB 1.5 MB 16.499708 flate_66 kB 24 MB 1.5 MB 16.487277 flate_66 kB 24 MB 1.5 MB 16.487277 gzip_66 kB 24 MB 1.5 MB 16.391820 gzip_66 kB 24 MB 1.5 MB 16.379549 gzip_66 kB 24 MB 1.5 MB 16.379549 flate_66 kB 24 MB 1.5 MB 16.166782 gzip_66 kB 24 MB 1.5 MB 16.129490 lz4-1M_524 kB 14 MB 1.1 MB 13.174347 lz4_524 kB 14 MB 1.1 MB 13.174347 lz4_524 kB 14 MB 1.1 MB 13.173353 lz4-1M_524 kB 14 MB 1.1 MB 13.173353 lz4-1M_524 kB 14 MB 1.1 MB 13.173353 lz4_524 kB 14 MB 1.1 MB 13.173353 lz4_262 kB 17 MB 1.3 MB 13.031562 lz4-1M_262 kB 17 MB 1.3 MB 13.031562 lz4-1M_262 kB 17 MB 1.3 MB 13.029619 lz4-1M_262 kB 17 MB 1.3 MB 13.029619 lz4_262 kB 17 MB 1.3 MB 13.029619 lz4_262 kB 17 MB 1.3 MB 13.029619 lz4-256k_524 kB 14 MB 1.1 MB 12.940461 lz4-256k_524 kB 14 MB 1.1 MB 12.939504 lz4-256k_524 kB 14 MB 1.1 MB 12.939504 lz4-1M_524 kB 14 MB 1.1 MB 12.910006 lz4_524 kB 14 MB 1.1 MB 12.910006 lz4-256k_262 kB 17 MB 1.3 MB 12.800857 lz4-256k_262 kB 17 MB 1.3 MB 12.798982 lz4-256k_262 kB 17 MB 1.3 MB 12.798982 lz4_262 kB 17 MB 1.3 MB 12.755388 lz4-1M_262 kB 17 MB 1.3 MB 12.755388 lz4-256k_524 kB 14 MB 1.1 MB 12.659275 lz4-256k_262 kB 16 MB 1.3 MB 12.536723 lz4-256k_66 kB 18 MB 1.5 MB 12.241545 lz4_66 kB 18 MB 1.5 MB 12.241545 lz4-1M_66 kB 18 MB 1.5 MB 12.241545 lz4_66 kB 18 MB 1.5 MB 12.234700 lz4-256k_66 kB 18 MB 1.5 MB 12.234700 lz4-1M_66 kB 18 MB 1.5 MB 12.234700 lz4-1M_66 kB 18 MB 1.5 MB 12.234700 lz4-256k_66 kB 18 MB 1.5 MB 12.234700 lz4_66 kB 18 MB 1.5 MB 12.234700 lz4-64k_524 kB 13 MB 1.1 MB 12.188250 lz4-64k_524 kB 13 MB 1.1 MB 12.187399 lz4-64k_524 kB 13 MB 1.1 MB 12.187399 lz4-64k_262 kB 16 MB 1.3 MB 12.078533 lz4-64k_262 kB 16 MB 1.3 MB 12.076864 lz4-64k_262 kB 16 MB 1.3 MB 12.076864 lz4-1M_66 kB 18 MB 1.5 MB 11.977005 lz4_66 kB 18 MB 1.5 MB 11.977005 lz4-256k_66 kB 18 MB 1.5 MB 11.977005 lz4-64k_524 kB 13 MB 1.1 MB 11.927031 lz4-64k_262 kB 16 MB 1.3 MB 11.810968 lz4-64k_66 kB 17 MB 1.5 MB 11.517938 lz4-64k_66 kB 17 MB 1.5 MB 11.511878 lz4-64k_66 kB 17 MB 1.5 MB 11.511878 lz4-64k_66 kB 17 MB 1.5 MB 11.243814 snappy_524 kB 10 MB 1.1 MB 9.255906 snappy_524 kB 10 MB 1.1 MB 9.255416 snappy_524 kB 10 MB 1.1 MB 9.255416 snappy_262 kB 12 MB 1.3 MB 9.204198 snappy_262 kB 12 MB 1.3 MB 9.203228 snappy_262 kB 12 MB 1.3 MB 9.203228 snappy_524 kB 9.9 MB 1.1 MB 9.146319 snappy_262 kB 12 MB 1.3 MB 9.060195 snappy_66 kB 13 MB 1.5 MB 8.872077 snappy_66 kB 13 MB 1.5 MB 8.868481 snappy_66 kB 13 MB 1.5 MB 8.868481 snappy_66 kB 13 MB 1.5 MB 8.706880 none_524 kB 1.5 MB 1.5 MB 0.985492 none_524 kB 1.5 MB 1.5 MB 0.985486 none_524 kB 1.5 MB 1.5 MB 0.985486 none_262 kB 1.5 MB 1.5 MB 0.985456 none_262 kB 1.5 MB 1.5 MB 0.985445 none_262 kB 1.5 MB 1.5 MB 0.985445 none_66 kB 1.5 MB 1.5 MB 0.985269 none_66 kB 1.5 MB 1.5 MB 0.985224 none_66 kB 1.5 MB 1.5 MB 0.985224 none_524 kB 1.5 MB 1.5 MB 0.978206 none_262 kB 1.5 MB 1.5 MB 0.978158 none_66 kB 1.5 MB 1.5 MB 0.977945 panic: test timed out after 10m0s running tests: TestGenerateDataSize (3m0s) TestGenerateDataSize/lz4-1M (27s) goroutine 18258 [running]: testing.(*M).startAlarm.func1() /usr/lib/go/src/testing/testing.go:2259 +0x304 created by time.goFunc /usr/lib/go/src/time/sleep.go:176 +0x50 goroutine 1 [chan receive, 3 minutes]: testing.(*T).Run(0x3f64503d40, {0xc4bccd?, 0xa6c70?}, 0xcdf6b0) /usr/lib/go/src/testing/testing.go:1649 +0x380 testing.runTests.func1(0x1a0?) /usr/lib/go/src/testing/testing.go:2054 +0x54 testing.tRunner(0x3f64503d40, 0x3f648bfc68) /usr/lib/go/src/testing/testing.go:1595 +0x104 testing.runTests(0x3f64752f00?, {0x1842f00, 0x27, 0x27}, {0x20484?, 0x3f64745db0?, 0x18a4b20?}) /usr/lib/go/src/testing/testing.go:2052 +0x414 testing.(*M).Run(0x3f64752f00) /usr/lib/go/src/testing/testing.go:1925 +0x564 main.main() _testmain.go:145 +0x1a8 goroutine 354 [chan receive, 9 minutes]: testing.(*T).Parallel(0x3eea9a21a0) /usr/lib/go/src/testing/testing.go:1403 +0x1ac github.com/grafana/loki/pkg/chunkenc.TestReadFormatV1(0x3eea9a21a0) /build/loki/src/loki-2.9.2/pkg/chunkenc/memchunk_test.go:290 +0x30 testing.tRunner(0x3eea9a21a0, 0xcdf720) /usr/lib/go/src/testing/testing.go:1595 +0x104 created by testing.(*T).Run in goroutine 1 /usr/lib/go/src/testing/testing.go:1648 +0x36c goroutine 5906 [chan receive, 5 minutes]: testing.(*T).Parallel(0x3edae00340) /usr/lib/go/src/testing/testing.go:1403 +0x1ac github.com/grafana/loki/pkg/chunkenc.TestGZIPChunkTargetSize(0x3e544d8000?) /build/loki/src/loki-2.9.2/pkg/chunkenc/memchunk_test.go:542 +0x30 testing.tRunner(0x3edae00340, 0xcdf6a8) /usr/lib/go/src/testing/testing.go:1595 +0x104 created by testing.(*T).Run in goroutine 1 /usr/lib/go/src/testing/testing.go:1648 +0x36c goroutine 18256 [runnable]: github.com/grafana/loki/pkg/chunkenc.TestGenerateDataSize.func1(0x3edec51d40) /build/loki/src/loki-2.9.2/pkg/chunkenc/memchunk_test.go:936 +0x1d4 testing.tRunner(0x3edec51d40, 0x3d04318020) /usr/lib/go/src/testing/testing.go:1595 +0x104 created by testing.(*T).Run in goroutine 18251 /usr/lib/go/src/testing/testing.go:1648 +0x36c goroutine 5907 [chan receive, 5 minutes]: testing.(*T).Parallel(0x3edae004e0) /usr/lib/go/src/testing/testing.go:1403 +0x1ac github.com/grafana/loki/pkg/chunkenc.TestMemChunk_AppendOutOfOrder(0x3e4690a000?) /build/loki/src/loki-2.9.2/pkg/chunkenc/memchunk_test.go:593 +0x30 testing.tRunner(0x3edae004e0, 0xcdf6d0) /usr/lib/go/src/testing/testing.go:1595 +0x104 created by testing.(*T).Run in goroutine 1 /usr/lib/go/src/testing/testing.go:1648 +0x36c goroutine 18251 [chan receive]: testing.(*T).Run(0x3e89c296c0, {0xc3a427?, 0x0?}, 0x3d04318020) /usr/lib/go/src/testing/testing.go:1649 +0x380 github.com/grafana/loki/pkg/chunkenc.TestGenerateDataSize(0x3cd2100000?) /build/loki/src/loki-2.9.2/pkg/chunkenc/memchunk_test.go:925 +0x6c testing.tRunner(0x3e89c296c0, 0xcdf6b0) /usr/lib/go/src/testing/testing.go:1595 +0x104 created by testing.(*T).Run in goroutine 1 /usr/lib/go/src/testing/testing.go:1648 +0x36c FAIL github.com/grafana/loki/pkg/chunkenc 606.087s ok github.com/grafana/loki/pkg/configs/client 14.420s ok github.com/grafana/loki/pkg/configs/userconfig 5.176s --- FAIL: TestDistributor (3.81s) --- FAIL: TestDistributor/[0](lines=10) (2.52s) distributor_test.go:105: expected 1, got 0 FAIL FAIL github.com/grafana/loki/pkg/distributor 23.779s ok github.com/grafana/loki/pkg/distributor/writefailures 6.179s ok github.com/grafana/loki/pkg/ingester 401.439s ok github.com/grafana/loki/pkg/ingester/index 15.254s ok github.com/grafana/loki/pkg/ingester/wal 6.755s ok github.com/grafana/loki/pkg/iter 7.201s ok github.com/grafana/loki/pkg/logcli/client 7.802s ok github.com/grafana/loki/pkg/logcli/output 6.917s ok github.com/grafana/loki/pkg/logcli/print 7.815s ok github.com/grafana/loki/pkg/logcli/query 11.389s ok github.com/grafana/loki/pkg/loghttp 12.991s ok github.com/grafana/loki/pkg/loghttp/push 10.335s ok github.com/grafana/loki/pkg/logproto 6.322s ok github.com/grafana/loki/pkg/logql 80.377s ok github.com/grafana/loki/pkg/logql/log 18.991s ok github.com/grafana/loki/pkg/logql/log/jsonexpr 0.984s ok github.com/grafana/loki/pkg/logql/log/logfmt 0.971s ok github.com/grafana/loki/pkg/logql/log/pattern 1.057s ok github.com/grafana/loki/pkg/logql/sketch 45.911s ok github.com/grafana/loki/pkg/logql/syntax 6.517s ok github.com/grafana/loki/pkg/logqlanalyzer 5.973s ok github.com/grafana/loki/pkg/logqlmodel/metadata 1.807s ok github.com/grafana/loki/pkg/logqlmodel/stats 6.971s ok github.com/grafana/loki/pkg/loki 39.155s ok github.com/grafana/loki/pkg/lokifrontend/frontend/transport 3.475s ok github.com/grafana/loki/pkg/lokifrontend/frontend/v1 5.434s ok github.com/grafana/loki/pkg/lokifrontend/frontend/v2 10.348s ok github.com/grafana/loki/pkg/querier 6.432s ok github.com/grafana/loki/pkg/querier/queryrange 17.771s ok github.com/grafana/loki/pkg/querier/queryrange/queryrangebase 3.583s ok github.com/grafana/loki/pkg/querier/worker 6.454s ok github.com/grafana/loki/pkg/ruler 16.994s ok github.com/grafana/loki/pkg/ruler/base 33.961s ok github.com/grafana/loki/pkg/ruler/rulestore 9.366s ok github.com/grafana/loki/pkg/ruler/rulestore/bucketclient 9.438s ok github.com/grafana/loki/pkg/ruler/rulestore/configdb 8.090s ok github.com/grafana/loki/pkg/ruler/rulestore/local 8.234s ok github.com/grafana/loki/pkg/ruler/rulestore/objectclient 9.359s [no tests to run] ok github.com/grafana/loki/pkg/ruler/storage/cleaner 7.913s ok github.com/grafana/loki/pkg/ruler/storage/instance 8.690s ok github.com/grafana/loki/pkg/ruler/storage/wal 5.883s ok github.com/grafana/loki/pkg/scheduler 13.993s ok github.com/grafana/loki/pkg/scheduler/queue 18.796s ok github.com/grafana/loki/pkg/sizing 1.335s ok github.com/grafana/loki/pkg/storage 22.033s ok github.com/grafana/loki/pkg/storage/bucket 13.174s ok github.com/grafana/loki/pkg/storage/bucket/azure 4.591s ok github.com/grafana/loki/pkg/storage/bucket/http 3.195s ok github.com/grafana/loki/pkg/storage/bucket/s3 7.660s ok github.com/grafana/loki/pkg/storage/chunk 8.024s ok github.com/grafana/loki/pkg/storage/chunk/cache 18.614s ok github.com/grafana/loki/pkg/storage/chunk/client 13.359s ok github.com/grafana/loki/pkg/storage/chunk/client/aws 10.766s ok github.com/grafana/loki/pkg/storage/chunk/client/azure 10.345s ok github.com/grafana/loki/pkg/storage/chunk/client/baidubce 7.087s ok github.com/grafana/loki/pkg/storage/chunk/client/cassandra 8.946s ok github.com/grafana/loki/pkg/storage/chunk/client/gcp 10.574s ok github.com/grafana/loki/pkg/storage/chunk/client/grpc 7.681s ok github.com/grafana/loki/pkg/storage/chunk/client/hedging 3.803s ok github.com/grafana/loki/pkg/storage/chunk/client/ibmcloud 10.718s ok github.com/grafana/loki/pkg/storage/chunk/client/local 9.266s ok github.com/grafana/loki/pkg/storage/chunk/client/openstack 8.740s ok github.com/grafana/loki/pkg/storage/chunk/client/util 7.646s [no tests to run] ok github.com/grafana/loki/pkg/storage/chunk/fetcher 12.095s ok github.com/grafana/loki/pkg/storage/chunk/tests 47.523s ok github.com/grafana/loki/pkg/storage/common/aws 6.765s ok github.com/grafana/loki/pkg/storage/config 7.612s ok github.com/grafana/loki/pkg/storage/stores 9.131s ok github.com/grafana/loki/pkg/storage/stores/index/seriesvolume 6.218s ok github.com/grafana/loki/pkg/storage/stores/index/stats 6.862s ok github.com/grafana/loki/pkg/storage/stores/indexshipper 12.321s ok github.com/grafana/loki/pkg/storage/stores/indexshipper/compactor 40.264s ok github.com/grafana/loki/pkg/storage/stores/indexshipper/compactor/deletion 12.843s ok github.com/grafana/loki/pkg/storage/stores/indexshipper/compactor/deletionmode 1.249s ok github.com/grafana/loki/pkg/storage/stores/indexshipper/compactor/generationnumber 5.359s --- FAIL: Test_expirationChecker_Expired (0.13s) --- FAIL: Test_expirationChecker_Expired/just_expired_tenant (0.05s) expiration_test.go:120: Error Trace: /build/loki/src/loki-2.9.2/pkg/storage/stores/indexshipper/compactor/retention/expiration_test.go:120 Error: Not equal: expected: false actual : true Test: Test_expirationChecker_Expired/just_expired_tenant level=info ts=2023-12-11T20:02:18.600292823Z caller=marker.go:177 msg="mark processor started" workers=10 delay=0s level=debug ts=2023-12-11T20:02:18.602396259Z caller=retention.go:111 msg="starting to process table" level=debug ts=2023-12-11T20:02:18.604166535Z caller=marker.go:207 msg="processing mark file" path=/tmp/Test_Retentionnothing_is_expiring4126846386/002/retention/markers/1702324938602786778 level=info ts=2023-12-11T20:02:18.605365633Z caller=marker.go:77 msg="mark file created" file=/tmp/Test_Retentionnothing_is_expiring4126846386/002/retention/markers/1702324938602786778 level=debug ts=2023-12-11T20:02:18.608441927Z caller=retention.go:109 msg="finished to process table" duration=6.055548ms level=debug ts=2023-12-11T20:02:18.608770906Z caller=retention.go:111 msg="starting to process table" level=info ts=2023-12-11T20:02:18.614777994Z caller=marker.go:77 msg="mark file created" file=/tmp/Test_Retentionnothing_is_expiring4126846386/002/retention/markers/1702324938608915586 level=debug ts=2023-12-11T20:02:18.618435567Z caller=retention.go:109 msg="finished to process table" duration=9.661561ms level=info ts=2023-12-11T20:02:18.643861076Z caller=marker.go:177 msg="mark processor started" workers=10 delay=0s level=debug ts=2023-12-11T20:02:18.644630455Z caller=retention.go:111 msg="starting to process table" level=info ts=2023-12-11T20:02:18.645529693Z caller=marker.go:202 msg="no marks file found" level=info ts=2023-12-11T20:02:18.647467869Z caller=marker.go:77 msg="mark file created" file=/tmp/Test_Retentionone_global_expiration3880134213/002/retention/markers/1702324938645383513 level=debug ts=2023-12-11T20:02:18.651690101Z caller=retention.go:109 msg="finished to process table" duration=7.054906ms level=debug ts=2023-12-11T20:02:18.65199696Z caller=retention.go:111 msg="starting to process table" level=info ts=2023-12-11T20:02:18.654833394Z caller=marker.go:77 msg="mark file created" file=/tmp/Test_Retentionone_global_expiration3880134213/002/retention/markers/1702324938652166320 level=debug ts=2023-12-11T20:02:18.658486667Z caller=retention.go:109 msg="finished to process table" duration=6.489047ms level=debug ts=2023-12-11T20:02:19.647855096Z caller=marker.go:207 msg="processing mark file" path=/tmp/Test_Retentionone_global_expiration3880134213/002/retention/markers/1702324938645383513 level=info ts=2023-12-11T20:02:19.689442913Z caller=marker.go:177 msg="mark processor started" workers=10 delay=0s level=debug ts=2023-12-11T20:02:19.690469271Z caller=retention.go:111 msg="starting to process table" level=debug ts=2023-12-11T20:02:19.721593669Z caller=marker.go:207 msg="processing mark file" path=/tmp/Test_Retentionone_global_expiration_and_stream484660655/002/retention/markers/1702324939690648110 level=info ts=2023-12-11T20:02:19.721710089Z caller=marker.go:77 msg="mark file created" file=/tmp/Test_Retentionone_global_expiration_and_stream484660655/002/retention/markers/1702324939690648110 level=debug ts=2023-12-11T20:02:19.724256103Z caller=retention.go:109 msg="finished to process table" duration=33.786192ms level=debug ts=2023-12-11T20:02:19.724514603Z caller=retention.go:111 msg="starting to process table" level=info ts=2023-12-11T20:02:19.726273019Z caller=marker.go:77 msg="mark file created" file=/tmp/Test_Retentionone_global_expiration_and_stream484660655/002/retention/markers/1702324939724727863 level=debug ts=2023-12-11T20:02:19.728908174Z caller=retention.go:109 msg="finished to process table" duration=4.392591ms level=debug ts=2023-12-11T20:02:19.729177914Z caller=retention.go:111 msg="starting to process table" level=info ts=2023-12-11T20:02:19.730587111Z caller=marker.go:77 msg="mark file created" file=/tmp/Test_Retentionone_global_expiration_and_stream484660655/002/retention/markers/1702324939729334333 level=debug ts=2023-12-11T20:02:19.735661281Z caller=retention.go:109 msg="finished to process table" duration=6.473607ms level=debug ts=2023-12-11T20:02:20.695817247Z caller=marker.go:207 msg="processing mark file" path=/tmp/Test_Retentionone_global_expiration_and_stream484660655/002/retention/markers/1702324939729334333 level=warn ts=2023-12-11T20:02:27.687427356Z caller=retention.go:233 msg="Timed out while running delete" FAIL FAIL github.com/grafana/loki/pkg/storage/stores/indexshipper/compactor/retention 37.273s ok github.com/grafana/loki/pkg/storage/stores/indexshipper/downloads 12.391s ok github.com/grafana/loki/pkg/storage/stores/indexshipper/gatewayclient 10.916s ok github.com/grafana/loki/pkg/storage/stores/indexshipper/storage 10.823s ok github.com/grafana/loki/pkg/storage/stores/indexshipper/uploads 14.683s ok github.com/grafana/loki/pkg/storage/stores/series 61.961s ok github.com/grafana/loki/pkg/storage/stores/series/index 8.785s ok github.com/grafana/loki/pkg/storage/stores/shipper/index 10.969s ok github.com/grafana/loki/pkg/storage/stores/shipper/index/compactor 73.924s ok github.com/grafana/loki/pkg/storage/stores/shipper/indexgateway 9.766s ok github.com/grafana/loki/pkg/storage/stores/shipper/util 8.612s ok github.com/grafana/loki/pkg/storage/stores/tsdb 252.063s ok github.com/grafana/loki/pkg/storage/stores/tsdb/index 87.261s ok github.com/grafana/loki/pkg/util 57.544s ok github.com/grafana/loki/pkg/util/cfg 3.300s ok github.com/grafana/loki/pkg/util/flagext 8.988s ok github.com/grafana/loki/pkg/util/httpreq 6.898s ok github.com/grafana/loki/pkg/util/instrument 5.588s ok github.com/grafana/loki/pkg/util/log 6.425s ok github.com/grafana/loki/pkg/util/loser 0.611s ok github.com/grafana/loki/pkg/util/marshal 12.371s ok github.com/grafana/loki/pkg/util/marshal/legacy 7.931s ok github.com/grafana/loki/pkg/util/pool 1.592s ok github.com/grafana/loki/pkg/util/querylimits 9.634s ok github.com/grafana/loki/pkg/util/server 5.216s ok github.com/grafana/loki/pkg/util/unmarshal 6.281s ok github.com/grafana/loki/pkg/util/unmarshal/legacy 4.656s ok github.com/grafana/loki/pkg/util/validation 5.597s ok github.com/grafana/loki/pkg/validation 7.145s ok github.com/grafana/loki/tools/doc-generator/parse 10.185s ok github.com/grafana/loki/tools/querytee 3.783s ok github.com/grafana/loki/tools/tsdb/migrate-versions 10.854s ok github.com/grafana/loki/tools/tsdb/tsdb-map 4.692s FAIL ==> ERROR: A failure occurred in check().  Aborting... ==> ERROR: Build failed, check /var/lib/archbuild/extra-riscv64/root56/build receiving incremental file list loki-2.9.2-1-riscv64-build.log loki-2.9.2-1-riscv64-check.log loki-2.9.2-1-riscv64-prepare.log sent 81 bytes received 16,028 bytes 32,218.00 bytes/sec total size is 160,891 speedup is 9.99