==> Building on ninetales ==> Checking for remote environment... ==> Syncing package to remote host... sending incremental file list created directory packages/opensearch ./ .SRCINFO 6,615 100% 0.00kB/s 0:00:00 6,615 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=11/13) .nvchecker.toml 126 100% 123.05kB/s 0:00:00 126 100% 123.05kB/s 0:00:00 (xfr#2, to-chk=10/13) PKGBUILD 12,645 100% 12.06MB/s 0:00:00 12,645 100% 12.06MB/s 0:00:00 (xfr#3, to-chk=9/13) README.md 270 100% 263.67kB/s 0:00:00 270 100% 263.67kB/s 0:00:00 (xfr#4, to-chk=8/13) opensearch-2.19.1-2.log 619 100% 604.49kB/s 0:00:00 619 100% 604.49kB/s 0:00:00 (xfr#5, to-chk=7/13) opensearch-keystore.service 243 100% 237.30kB/s 0:00:00 243 100% 237.30kB/s 0:00:00 (xfr#6, to-chk=6/13) opensearch-keystore@.service 290 100% 283.20kB/s 0:00:00 290 100% 283.20kB/s 0:00:00 (xfr#7, to-chk=5/13) opensearch-tmpfile.conf 929 100% 907.23kB/s 0:00:00 929 100% 907.23kB/s 0:00:00 (xfr#8, to-chk=4/13) opensearch-user.conf 34 100% 33.20kB/s 0:00:00 34 100% 33.20kB/s 0:00:00 (xfr#9, to-chk=3/13) opensearch.default 250 100% 244.14kB/s 0:00:00 250 100% 244.14kB/s 0:00:00 (xfr#10, to-chk=2/13) opensearch.service 1,835 100% 1.75MB/s 0:00:00 1,835 100% 1.75MB/s 0:00:00 (xfr#11, to-chk=1/13) opensearch@.service 1,847 100% 1.76MB/s 0:00:00 1,847 100% 1.76MB/s 0:00:00 (xfr#12, to-chk=0/13) sent 9,015 bytes received 289 bytes 18,608.00 bytes/sec total size is 25,280 speedup is 2.72 ==> Applying RISC-V patches... sending incremental file list ./ opensearch-riscv.patch 18,131 100% 0.00kB/s 0:00:00 18,131 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=1/3) riscv64.patch 2,847 100% 2.72MB/s 0:00:00 2,847 100% 2.72MB/s 0:00:00 (xfr#2, to-chk=0/3) sent 5,410 bytes received 57 bytes 10,934.00 bytes/sec total size is 20,978 speedup is 3.84 patching file PKGBUILD Hunk #2 succeeded at 46 (offset -1 lines). Hunk #3 succeeded at 56 with fuzz 1 (offset -2 lines). Hunk #4 succeeded at 81 (offset -2 lines). Hunk #5 succeeded at 168 (offset -2 lines). ==> Patching arch to riscv64... ==> Running pkgctl build --arch riscv64 --repo extra on remote host... ==> WARNING: unsupported architecture: riscv64 ==> Building opensearch  -> repo: extra  -> arch: riscv64  -> worker: felix-1 ==> Building opensearch for [extra] (riscv64) ]2;🔵 Container arch-nspawn-1983547 on ninetales.felixc.at\[?25l:: Synchronizing package databases... core downloading... extra downloading... error: restricting filesystem access failed because landlock is not supported by the kernel! :: Starting full system upgrade... there is nothing to do [?25h==> Building in chroot for [extra] (riscv64)... ==> Locking clean chroot [/var/lib/archbuild/extra-riscv64/root]...done ==> Synchronizing chroot copy [/var/lib/archbuild/extra-riscv64/root] -> [felix-1]...done ==> Making package: opensearch 2.19.1-2 (Thu May 8 15:35:01 2025) ==> Retrieving sources...  -> Downloading OpenSearch-2.19.1.tar.gz... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 3568 0 3568 0 0 2800 0 --:--:-- 0:00:01 --:--:-- 2800 100 100k 0 100k 0 0 47443 0 --:--:-- 0:00:02 --:--:-- 109k 100 2710k 0 2710k 0 0 862k 0 --:--:-- 0:00:03 --:--:-- 1448k 100 3701k 0 3701k 0 0 788k 0 --:--:-- 0:00:04 --:--:-- 1081k 100 4230k 0 4230k 0 0 823k 0 --:--:-- 0:00:05 --:--:-- 1094k 100 6144k 0 6144k 0 0 970k 0 --:--:-- 0:00:06 --:--:-- 1214k 100 6886k 0 6886k 0 0 954k 0 --:--:-- 0:00:07 --:--:-- 1341k 100 7894k 0 7894k 0 0 973k 0 --:--:-- 0:00:08 --:--:-- 1043k 100 10.9M 0 10.9M 0 0 1226k 0 --:--:-- 0:00:09 --:--:-- 1689k 100 16.6M 0 16.6M 0 0 1685k 0 --:--:-- 0:00:10 --:--:-- 2577k 100 19.3M 0 19.3M 0 0 1786k 0 --:--:-- 0:00:11 --:--:-- 2870k 100 22.1M 0 22.1M 0 0 1879k 0 --:--:-- 0:00:12 --:--:-- 3247k 100 22.3M 0 22.3M 0 0 1887k 0 --:--:-- 0:00:12 --:--:-- 3738k  -> Found opensearch.service  -> Found opensearch@.service  -> Found opensearch-keystore.service  -> Found opensearch-keystore@.service  -> Found opensearch-user.conf  -> Found opensearch-tmpfile.conf  -> Found opensearch.default  -> Found opensearch-riscv.patch  -> Downloading protobuf-3.25.5.tar.gz... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7680 0 7680 0 0 6617 0 --:--:-- 0:00:01 --:--:-- 6617 100 2509k 0 2509k 0 0 1343k 0 --:--:-- 0:00:01 --:--:-- 3529k 100 5739k 0 5739k 0 0 2467k 0 --:--:-- 0:00:02 --:--:-- 4920k ==> Validating source files with sha256sums... OpenSearch-2.19.1.tar.gz ... Passed opensearch.service ... Passed opensearch@.service ... Passed opensearch-keystore.service ... Passed opensearch-keystore@.service ... Passed opensearch-user.conf ... Passed opensearch-tmpfile.conf ... Passed opensearch.default ... Passed opensearch-riscv.patch ... Passed protobuf-3.25.5.tar.gz ... Passed ]2;🔵 Container arch-nspawn-2004996 on ninetales.felixc.at\==> Making package: opensearch 2.19.1-2 (Thu May 8 15:51:01 2025) ==> Checking runtime dependencies... ==> Checking buildtime dependencies... ==> Installing missing dependencies... [?25lresolving dependencies... looking for conflicting packages... warning: dependency cycle detected: warning: harfbuzz will be installed before its freetype2 dependency Package (24) New Version Net Change Download Size extra/cppdap 1.58.0-2 1.48 MiB extra/freetype2 2.13.3-3 1.58 MiB extra/giflib 5.2.2-2 0.24 MiB extra/graphite 1:1.3.14-4 0.18 MiB extra/gtest 1.16.0-2 1.50 MiB extra/harfbuzz 11.2.0-1 4.09 MiB extra/hicolor-icon-theme 0.18-1 0.05 MiB extra/java-environment-common 3-6 0.00 MiB extra/java-runtime-common 3-6 0.01 MiB extra/jbigkit 2.1-8 0.13 MiB extra/jsoncpp 1.9.6-3 3.16 MiB extra/lcms2 2.17-1 0.64 MiB extra/libjpeg-turbo 3.1.0-1 1.99 MiB extra/libnet 2:1.3-1 1.46 MiB extra/libpng 1.6.48-1 0.54 MiB extra/libtiff 4.7.0-1 1.64 MiB extra/libuv 1.51.0-1 0.60 MiB core/nspr 4.36-1 0.70 MiB core/nss 3.111-1 4.67 MiB extra/rhash 1.4.4-1 0.31 MiB extra/abseil-cpp 20250127.1-1 5.94 MiB extra/cmake 4.0.1-2 71.23 MiB extra/jdk17-openjdk 17.0.14.u7-1 376.44 MiB 236.65 MiB extra/unzip 6.0-22 0.28 MiB Total Download Size: 236.65 MiB Total Installed Size: 478.85 MiB :: Proceed with installation? [Y/n] :: Retrieving packages... jdk17-openjdk-17.0.14.u7-1-riscv64 downloading... error: restricting filesystem access failed because landlock is not supported by the kernel! checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... installing java-runtime-common... For the complete set of Java binaries to be available in your PATH, you need to re-login or source /etc/profile Please note that this package does not support forcing JAVA_HOME as former package java-common did installing nspr... installing nss... installing libjpeg-turbo... Optional dependencies for libjpeg-turbo java-runtime>11: for TurboJPEG Java wrapper [pending] installing jbigkit... installing libtiff... Optional dependencies for libtiff freeglut: for using tiffgt installing lcms2... installing libnet... installing libpng... installing graphite... Optional dependencies for graphite graphite-docs: Documentation installing harfbuzz... Optional dependencies for harfbuzz harfbuzz-utils: utilities installing freetype2... installing java-environment-common... installing hicolor-icon-theme... installing giflib... installing jdk17-openjdk... Optional dependencies for jdk17-openjdk java-rhino: for some JavaScript support alsa-lib: for basic sound support gtk3: for the Gtk+ 3 look and feel - desktop usage installing unzip... installing cppdap... installing jsoncpp... Optional dependencies for jsoncpp jsoncpp-doc: documentation installing libuv... installing rhash... installing cmake... Optional dependencies for cmake make: for unix Makefile generator [installed] ninja: for ninja generator qt6-base: cmake-gui installing gtest... Optional dependencies for gtest python: gmock generator installing abseil-cpp... :: Running post-transaction hooks... (1/1) Arming ConditionNeedsUpdate... [?25h==> Retrieving sources...  -> Found OpenSearch-2.19.1.tar.gz  -> Found opensearch.service  -> Found opensearch@.service  -> Found opensearch-keystore.service  -> Found opensearch-keystore@.service  -> Found opensearch-user.conf  -> Found opensearch-tmpfile.conf  -> Found opensearch.default  -> Found opensearch-riscv.patch  -> Found protobuf-3.25.5.tar.gz ==> WARNING: Skipping all source file integrity checks. ==> Extracting sources...  -> Extracting OpenSearch-2.19.1.tar.gz with bsdtar  -> Extracting protobuf-3.25.5.tar.gz with bsdtar ==> Starting prepare()... -- The C compiler identification is GNU 14.2.1 -- The CXX compiler identification is GNU 14.2.1 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /usr/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- -- 25.5.0 -- Performing Test protobuf_HAVE_LD_VERSION_SCRIPT -- Performing Test protobuf_HAVE_LD_VERSION_SCRIPT - Success -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success -- Found Threads: TRUE -- Found ZLIB: /usr/lib/libz.so (found version "1.3.1") -- Performing Test protobuf_HAVE_BUILTIN_ATOMICS -- Performing Test protobuf_HAVE_BUILTIN_ATOMICS - Success CMake Deprecation Warning at third_party/utf8_range/CMakeLists.txt:1 (cmake_minimum_required): Compatibility with CMake < 3.10 will be removed from a future version of CMake. Update the VERSION argument value. Or, use the ... syntax to tell CMake that the project requires at least but has been updated to work with policies introduced by or earlier. -- Configuring done (41.6s) -- Generating done (1.9s) -- Build files have been written to: /build/opensearch/src/protobuf-3.25.5 [ 1%] Building C object third_party/utf8_range/CMakeFiles/utf8_range.dir/range2-neon.c.o [ 1%] Building C object third_party/utf8_range/CMakeFiles/utf8_range.dir/naive.c.o [ 1%] Building CXX object third_party/utf8_range/CMakeFiles/utf8_validity.dir/utf8_validity.cc.o [ 1%] Building C object third_party/utf8_range/CMakeFiles/utf8_range.dir/range2-sse.c.o [ 2%] Linking C static library libutf8_range.a [ 2%] Built target utf8_range [ 3%] Linking CXX static library libutf8_validity.a [ 3%] Built target utf8_validity [ 3%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/generated_enum_util.cc.o [ 4%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/arena_align.cc.o [ 3%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/extension_set.cc.o [ 4%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/arenastring.cc.o [ 5%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/implicit_weak_message.cc.o [ 7%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/arenaz_sampler.cc.o [ 7%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/inlined_string_field.cc.o [ 7%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/any_lite.cc.o [ 7%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/arena.cc.o [ 8%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/generated_message_tctable_lite.cc.o [ 8%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/any.pb.cc.o [ 8%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/generated_message_util.cc.o [ 9%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/io/coded_stream.cc.o [ 9%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/api.pb.cc.o [ 9%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/io/io_win32.cc.o [ 10%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/duration.pb.cc.o [ 11%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/io/zero_copy_stream.cc.o [ 11%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/empty.pb.cc.o [ 12%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/field_mask.pb.cc.o [ 12%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/source_context.pb.cc.o [ 12%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/io/zero_copy_stream_impl.cc.o [ 12%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/io/zero_copy_stream_impl_lite.cc.o In file included from /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/generated_message_tctable_decl.h:22, from /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/generated_message_tctable_lite.cc:22: /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/parse_context.h:1125:1: warning: ‘always_inline’ function might not be inlinable unless also declared ‘inline’ [-Wattributes] 1125 | ParseContext::ParseLengthDelimitedInlined(const char* ptr, const Func& func) { | ^~~~~~~~~~~~ /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/parse_context.h:1125:1: warning: ‘always_inline’ function might not be inlinable unless also declared ‘inline’ [-Wattributes] /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/generated_message_tctable_lite.cc:830:36: warning: ‘always_inline’ function might not be inlinable unless also declared ‘inline’ [-Wattributes] 830 | PROTOBUF_ALWAYS_INLINE const char* TcParser::FastVarintS1( | ^~~~~~~~ /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/generated_message_tctable_lite.cc:830:36: warning: ‘always_inline’ function might not be inlinable unless also declared ‘inline’ [-Wattributes] /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/generated_message_tctable_lite.cc:753:29: warning: ‘always_inline’ function might not be inlinable unless also declared ‘inline’ [-Wattributes] 753 | PROTOBUF_ALWAYS_INLINE bool EnumIsValidAux(int32_t val, uint16_t xform_val, | ^~~~~~~~~~~~~~ /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/generated_message_tctable_lite.cc:741:29: warning: ‘always_inline’ function might not be inlinable unless also declared ‘inline’ [-Wattributes] 741 | PROTOBUF_ALWAYS_INLINE void PrefetchEnumData(uint16_t xform_val, | ^~~~~~~~~~~~~~~~ [ 13%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/struct.pb.cc.o [ 14%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/map.cc.o [ 14%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/timestamp.pb.cc.o [ 14%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/message_lite.cc.o [ 15%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/parse_context.cc.o [ 16%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/type.pb.cc.o [ 16%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/port.cc.o [ 16%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/wrappers.pb.cc.o [ 17%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/raw_ptr.cc.o [ 18%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/any_lite.cc.o [ 18%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/any.cc.o [ 18%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/repeated_field.cc.o [ 18%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/arena.cc.o [ 19%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/repeated_ptr_field.cc.o [ 20%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/arena_align.cc.o [ 20%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/stubs/common.cc.o [ 20%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/arenastring.cc.o /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/message_lite.cc: In member function ‘int google::protobuf::MessageLite::GetCachedSize() const’: /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/message_lite.cc:69:70: warning: ‘int google::protobuf::MessageLite::ByteSize() const’ is deprecated: Please use ByteSizeLong() instead [-Wdeprecated-declarations] 69 | if (PROTOBUF_PREDICT_FALSE(cached_size == nullptr)) return ByteSize(); | ~~~~~~~~^~ In file included from /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/message_lite.cc:13: /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/message_lite.h:454:59: note: declared here 454 | [[deprecated("Please use ByteSizeLong() instead")]] int ByteSize() const { | ^~~~~~~~ [ 20%] Building CXX object CMakeFiles/libprotobuf-lite.dir/src/google/protobuf/wire_format_lite.cc.o [ 21%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/arenaz_sampler.cc.o [ 21%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/compiler/importer.cc.o [ 22%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/compiler/parser.cc.o [ 22%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/cpp_features.pb.cc.o [ 22%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/descriptor.cc.o [ 22%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/descriptor_database.cc.o [ 23%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/descriptor.pb.cc.o [ 24%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/dynamic_message.cc.o [ 24%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/extension_set.cc.o [ 25%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/extension_set_heavy.cc.o [ 25%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/feature_resolver.cc.o [ 26%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/generated_enum_util.cc.o [ 27%] Linking CXX static library libprotobuf-lite.a [ 27%] Built target libprotobuf-lite [ 27%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/generated_message_bases.cc.o [ 27%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/generated_message_reflection.cc.o [ 28%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/generated_message_tctable_full.cc.o [ 28%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/generated_message_tctable_gen.cc.o [ 29%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/generated_message_tctable_lite.cc.o [ 29%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/generated_message_util.cc.o [ 30%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/implicit_weak_message.cc.o [ 30%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/inlined_string_field.cc.o [ 31%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/internal_message_util.cc.o [ 31%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/io/coded_stream.cc.o [ 31%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/io/gzip_stream.cc.o [ 32%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/io/io_win32.cc.o [ 32%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/io/printer.cc.o [ 33%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/io/strtod.cc.o [ 33%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/io/tokenizer.cc.o [ 34%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/io/zero_copy_sink.cc.o In file included from /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/generated_message_tctable_decl.h:22, from /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/generated_message_tctable_lite.cc:22: /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/parse_context.h:1125:1: warning: ‘always_inline’ function might not be inlinable unless also declared ‘inline’ [-Wattributes] 1125 | ParseContext::ParseLengthDelimitedInlined(const char* ptr, const Func& func) { | ^~~~~~~~~~~~ /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/parse_context.h:1125:1: warning: ‘always_inline’ function might not be inlinable unless also declared ‘inline’ [-Wattributes] /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/generated_message_tctable_lite.cc:830:36: warning: ‘always_inline’ function might not be inlinable unless also declared ‘inline’ [-Wattributes] 830 | PROTOBUF_ALWAYS_INLINE const char* TcParser::FastVarintS1( | ^~~~~~~~ /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/generated_message_tctable_lite.cc:830:36: warning: ‘always_inline’ function might not be inlinable unless also declared ‘inline’ [-Wattributes] /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/generated_message_tctable_lite.cc:753:29: warning: ‘always_inline’ function might not be inlinable unless also declared ‘inline’ [-Wattributes] 753 | PROTOBUF_ALWAYS_INLINE bool EnumIsValidAux(int32_t val, uint16_t xform_val, | ^~~~~~~~~~~~~~ /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/generated_message_tctable_lite.cc:741:29: warning: ‘always_inline’ function might not be inlinable unless also declared ‘inline’ [-Wattributes] 741 | PROTOBUF_ALWAYS_INLINE void PrefetchEnumData(uint16_t xform_val, | ^~~~~~~~~~~~~~~~ [ 34%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/io/zero_copy_stream.cc.o [ 35%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/io/zero_copy_stream_impl.cc.o [ 35%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/io/zero_copy_stream_impl_lite.cc.o [ 35%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/json/internal/lexer.cc.o [ 36%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/json/internal/message_path.cc.o [ 36%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/json/internal/parser.cc.o [ 37%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/json/internal/unparser.cc.o [ 37%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/json/internal/untyped_message.cc.o [ 38%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/json/internal/writer.cc.o [ 38%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/json/internal/zero_copy_buffered_stream.cc.o [ 39%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/json/json.cc.o [ 39%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/map.cc.o [ 40%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/map_field.cc.o [ 40%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/message.cc.o [ 40%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/message_lite.cc.o [ 41%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/parse_context.cc.o [ 41%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/port.cc.o [ 42%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/raw_ptr.cc.o [ 42%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/reflection_mode.cc.o [ 43%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/reflection_ops.cc.o [ 43%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/repeated_field.cc.o [ 44%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/repeated_ptr_field.cc.o /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/message_lite.cc: In member function ‘int google::protobuf::MessageLite::GetCachedSize() const’: /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/message_lite.cc:69:70: warning: ‘int google::protobuf::MessageLite::ByteSize() const’ is deprecated: Please use ByteSizeLong() instead [-Wdeprecated-declarations] 69 | if (PROTOBUF_PREDICT_FALSE(cached_size == nullptr)) return ByteSize(); | ~~~~~~~~^~ In file included from /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/message_lite.cc:13: /build/opensearch/src/protobuf-3.25.5/src/google/protobuf/message_lite.h:454:59: note: declared here 454 | [[deprecated("Please use ByteSizeLong() instead")]] int ByteSize() const { | ^~~~~~~~ [ 44%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/service.cc.o [ 44%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/stubs/common.cc.o [ 45%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/text_format.cc.o [ 45%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/unknown_field_set.cc.o [ 46%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/util/delimited_message_util.cc.o [ 46%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/util/field_comparator.cc.o [ 47%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/util/field_mask_util.cc.o [ 47%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/util/message_differencer.cc.o [ 48%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/util/time_util.cc.o [ 48%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/util/type_resolver_util.cc.o [ 48%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/wire_format.cc.o [ 49%] Building CXX object CMakeFiles/libprotobuf.dir/src/google/protobuf/wire_format_lite.cc.o [ 49%] Linking CXX static library libprotobuf.a [ 49%] Built target libprotobuf [ 49%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/allowlists/weak_imports.cc.o [ 50%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/code_generator.cc.o [ 51%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/allowlists/unused_imports.cc.o [ 52%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/cpp/field_generators/cord_field.cc.o [ 52%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/allowlists/editions.cc.o [ 52%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/command_line_interface.cc.o [ 52%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/allowlists/open_enum.cc.o [ 53%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/cpp/enum.cc.o [ 53%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/cpp/field_generators/enum_field.cc.o [ 54%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/allowlists/empty_package.cc.o [ 54%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/cpp/extension.cc.o [ 54%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/cpp/field.cc.o [ 55%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/cpp/field_generators/map_field.cc.o [ 55%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/cpp/field_generators/message_field.cc.o [ 56%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/cpp/field_generators/primitive_field.cc.o [ 56%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/cpp/field_generators/string_field.cc.o [ 57%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/cpp/file.cc.o [ 57%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/cpp/generator.cc.o [ 57%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/cpp/helpers.cc.o [ 58%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/cpp/message.cc.o [ 58%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/cpp/padding_optimizer.cc.o [ 59%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/cpp/parse_function_generator.cc.o [ 59%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/cpp/service.cc.o [ 60%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/cpp/tracker.cc.o [ 60%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/csharp/csharp_doc_comment.cc.o [ 61%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/csharp/csharp_enum.cc.o [ 61%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/csharp/csharp_enum_field.cc.o [ 61%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/csharp/csharp_field_base.cc.o [ 62%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/csharp/csharp_generator.cc.o [ 62%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/csharp/csharp_helpers.cc.o [ 63%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/csharp/csharp_map_field.cc.o [ 63%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/csharp/csharp_message.cc.o [ 64%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/csharp/csharp_message_field.cc.o [ 64%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/csharp/csharp_primitive_field.cc.o [ 65%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/csharp/csharp_reflection_class.cc.o [ 65%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/csharp/csharp_repeated_enum_field.cc.o [ 65%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/csharp/csharp_repeated_message_field.cc.o [ 66%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/csharp/csharp_repeated_primitive_field.cc.o [ 66%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/csharp/csharp_source_generator_base.cc.o [ 67%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/csharp/csharp_wrapper_field.cc.o [ 67%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/csharp/names.cc.o [ 68%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/context.cc.o [ 68%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/doc_comment.cc.o [ 69%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/enum.cc.o [ 69%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/enum_field.cc.o [ 69%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/enum_field_lite.cc.o [ 70%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/enum_lite.cc.o [ 70%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/extension.cc.o [ 71%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/extension_lite.cc.o [ 71%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/field.cc.o [ 72%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/file.cc.o [ 72%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/generator.cc.o [ 73%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/generator_factory.cc.o [ 73%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/helpers.cc.o [ 73%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/java_features.pb.cc.o [ 74%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/kotlin_generator.cc.o [ 74%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/map_field.cc.o [ 75%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/map_field_lite.cc.o [ 75%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/message.cc.o [ 76%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/message_builder.cc.o [ 76%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/message_builder_lite.cc.o [ 77%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/message_field.cc.o [ 77%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/message_field_lite.cc.o [ 78%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/message_lite.cc.o [ 78%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/message_serialization.cc.o [ 78%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/name_resolver.cc.o [ 79%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/names.cc.o [ 79%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/primitive_field.cc.o [ 80%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/primitive_field_lite.cc.o [ 80%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/service.cc.o [ 81%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/shared_code_generator.cc.o [ 81%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/string_field.cc.o [ 82%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/java/string_field_lite.cc.o [ 82%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/objectivec/enum.cc.o [ 82%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/objectivec/enum_field.cc.o [ 83%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/objectivec/extension.cc.o [ 83%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/objectivec/field.cc.o [ 84%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/objectivec/file.cc.o [ 84%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/objectivec/generator.cc.o [ 85%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/objectivec/helpers.cc.o [ 85%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/objectivec/import_writer.cc.o [ 86%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/objectivec/line_consumer.cc.o [ 86%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/objectivec/map_field.cc.o [ 86%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/objectivec/message.cc.o [ 87%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/objectivec/message_field.cc.o [ 87%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/objectivec/names.cc.o [ 88%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/objectivec/oneof.cc.o [ 88%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/objectivec/primitive_field.cc.o [ 89%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/objectivec/text_format_decode_data.cc.o [ 89%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/php/names.cc.o [ 90%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/php/php_generator.cc.o [ 90%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/plugin.cc.o [ 90%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/plugin.pb.cc.o [ 91%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/python/generator.cc.o [ 91%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/python/helpers.cc.o [ 92%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/python/pyi_generator.cc.o [ 92%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/retention.cc.o [ 93%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/ruby/ruby_generator.cc.o [ 93%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/rust/accessors/accessors.cc.o [ 94%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/rust/accessors/singular_message.cc.o [ 94%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/rust/accessors/singular_scalar.cc.o [ 94%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/rust/accessors/singular_string.cc.o [ 95%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/rust/accessors/unsupported_field.cc.o [ 95%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/rust/context.cc.o [ 96%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/rust/generator.cc.o [ 96%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/rust/message.cc.o [ 97%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/rust/naming.cc.o [ 97%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/rust/oneof.cc.o [ 98%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/rust/relative_path.cc.o [ 98%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/subprocess.cc.o [ 98%] Building CXX object CMakeFiles/libprotoc.dir/src/google/protobuf/compiler/zip_writer.cc.o [ 99%] Linking CXX static library libprotoc.a [ 99%] Built target libprotoc [ 99%] Building CXX object CMakeFiles/protoc.dir/src/google/protobuf/compiler/main.cc.o [100%] Linking CXX executable protoc [100%] Built target protoc patching file buildSrc/src/main/java/org/opensearch/gradle/Architecture.java patching file buildSrc/src/main/java/org/opensearch/gradle/DistributionDownloadPlugin.java patching file buildSrc/src/main/java/org/opensearch/gradle/JavaVariant.java patching file buildSrc/src/main/java/org/opensearch/gradle/VersionProperties.java patching file buildSrc/src/main/java/org/opensearch/gradle/internal/InternalDistributionBwcSetupPlugin.java patching file buildSrc/src/test/java/org/opensearch/gradle/ArchitectureTests.java patching file buildSrc/src/test/java/org/opensearch/gradle/JdkDownloadPluginTests.java patching file distribution/archives/build.gradle patching file distribution/archives/linux-riscv64-tar/build.gradle patching file distribution/archives/no-jdk-linux-riscv64-tar/build.gradle patching file distribution/build.gradle patching file distribution/docker/build.gradle patching file distribution/docker/docker-riscv64-export/build.gradle patching file server/src/main/java/org/opensearch/bootstrap/SystemCallFilter.java patching file settings.gradle ==> Starting build()... ======================== Building OpenSearch ========================== Downloading https://services.gradle.org/distributions/gradle-8.12.1-all.zip .....................10%......................20%......................30%......................40%......................50%......................60%......................70%......................80%......................90%......................100% Welcome to Gradle 8.12.1! Here are the highlights of this release: - Enhanced error and warning reporting with the Problems API - File-system watching support on Alpine Linux - Build and test Swift 6 libraries and apps For more details see https://docs.gradle.org/8.12.1/release-notes.html Starting a Gradle Daemon (subsequent builds will be faster) > Task :buildSrc:reaper:compileJava > Task :buildSrc:compileJava > Task :buildSrc:reaper:processResources NO-SOURCE > Task :buildSrc:reaper:classes > Task :buildSrc:reaper:jar > Task :buildSrc:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :buildSrc:compileGroovy Note: /build/opensearch/src/OpenSearch-2.19.1/buildSrc/src/main/groovy/org/opensearch/gradle/test/TestWithSslPlugin.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: /build/opensearch/src/OpenSearch-2.19.1/buildSrc/src/main/groovy/org/opensearch/gradle/test/TestWithSslPlugin.java uses unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. > Task :buildSrc:generateVersionProperties > Task :buildSrc:pluginDescriptors > Task :buildSrc:processResources > Task :buildSrc:classes > Task :buildSrc:jar ======================================= OpenSearch Build Hamster says Hello! Gradle Version : 8.12.1 OS Info : Linux 6.13.5-arch1-1 (riscv64) JDK Version : 17 (Arch Linux JDK) JAVA_HOME : /usr/lib/jvm/java-17-openjdk Random Testing Seed : E5CDBDCD7C197CD3 In FIPS 140 mode : false ======================================= > Task :server:protobufDummy UP-TO-DATE > Task :libs:opensearch-geo:processResources NO-SOURCE > Task :libs:opensearch-secure-sm:processResources NO-SOURCE > Task :libs:opensearch-cli:processResources NO-SOURCE > Task :libs:opensearch-plugin-classloader:processResources NO-SOURCE > Task :libs:opensearch-task-commons:processResources NO-SOURCE > Task :modules:systemd:processResources NO-SOURCE > Task :libs:opensearch-telemetry:processResources NO-SOURCE > Task :libs:opensearch-common:processResources > Task :libs:opensearch-compress:processResources > Task :libs:opensearch-x-content:processResources > Task :libs:opensearch-core:processResources > Task :modules:systemd:copyPluginPropertiesTemplate > Task :server:extractProto > Task :libs:opensearch-geo:compileJava > Task :libs:opensearch-secure-sm:compileJava > Task :libs:opensearch-plugin-classloader:compileJava > Task :libs:opensearch-common:compileJava > Task :server:extractIncludeProto > Task :modules:systemd:pluginProperties > Task :server:generateProto > Task :libs:opensearch-common:compileJava Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. > Task :libs:opensearch-common:classes > Task :libs:opensearch-common:jar > Task :libs:opensearch-task-commons:compileJava > Task :libs:opensearch-cli:compileJava > Task :libs:opensearch-core:compileJava > Task :libs:opensearch-telemetry:compileJava > Task :libs:opensearch-core:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :libs:opensearch-compress:compileJava > Task :libs:opensearch-x-content:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :server:compileJava Note: Processing Log4j annotations Note: Annotations processed Note: Processing OpenSearch Api annotations Note: Processing Log4j annotations Note: No elements to process Note: Processing OpenSearch Api annotations Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. > Task :modules:systemd:compileJava Note: /build/opensearch/src/OpenSearch-2.19.1/modules/systemd/src/main/java/org/opensearch/systemd/SystemdPlugin.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:systemd:classes > Task :modules:systemd:jar > Task :modules:systemd:bundlePlugin > Task :distribution:processSystemdOutputs > Task :distribution:buildSystemdModule [Incubating] Problems report is available at: file:///build/opensearch/src/OpenSearch-2.19.1/build/reports/problems/problems-report.html BUILD SUCCESSFUL in 3h 36m 52s 33 actionable tasks: 33 executed > Task :buildSrc:compileJava UP-TO-DATE > Task :buildSrc:reaper:compileJava UP-TO-DATE > Task :buildSrc:reaper:processResources NO-SOURCE > Task :buildSrc:reaper:classes UP-TO-DATE > Task :buildSrc:reaper:jar UP-TO-DATE > Task :buildSrc:compileGroovy UP-TO-DATE > Task :buildSrc:generateVersionProperties UP-TO-DATE > Task :buildSrc:pluginDescriptors UP-TO-DATE > Task :buildSrc:processResources UP-TO-DATE > Task :buildSrc:classes UP-TO-DATE > Task :buildSrc:jar UP-TO-DATE ======================================= OpenSearch Build Hamster says Hello! Gradle Version : 8.12.1 OS Info : Linux 6.13.5-arch1-1 (riscv64) JDK Version : 17 (Arch Linux JDK) JAVA_HOME : /usr/lib/jvm/java-17-openjdk Random Testing Seed : 9155E36D8EE733CD In FIPS 140 mode : false ======================================= > Task :server:protobufDummy UP-TO-DATE > Task :libs:opensearch-telemetry:processResources NO-SOURCE > Task :libs:opensearch-cli:processResources NO-SOURCE > Task :libs:opensearch-plugin-classloader:processResources NO-SOURCE > Task :libs:opensearch-geo:processResources NO-SOURCE > Task :libs:opensearch-secure-sm:processResources NO-SOURCE > Task :libs:opensearch-task-commons:processResources NO-SOURCE > Task :libs:opensearch-common:processResources UP-TO-DATE > Task :libs:opensearch-compress:processResources UP-TO-DATE > Task :libs:opensearch-core:processResources UP-TO-DATE > Task :libs:opensearch-x-content:processResources UP-TO-DATE > Task :server:extractProto UP-TO-DATE > Task :modules:cache-common:processResources NO-SOURCE > Task :modules:geo:processResources NO-SOURCE > Task :modules:aggs-matrix-stats:processResources > Task :libs:opensearch-secure-sm:compileJava UP-TO-DATE > Task :libs:opensearch-plugin-classloader:compileJava UP-TO-DATE > Task :libs:opensearch-geo:compileJava UP-TO-DATE > Task :modules:geo:copyPluginPropertiesTemplate > Task :modules:cache-common:copyPluginPropertiesTemplate > Task :modules:analysis-common:processResources > Task :modules:aggs-matrix-stats:copyPluginPropertiesTemplate > Task :modules:ingest-common:processResources > Task :modules:analysis-common:copyPluginPropertiesTemplate > Task :libs:opensearch-common:compileJava UP-TO-DATE > Task :modules:ingest-common:copyPluginPropertiesTemplate > Task :modules:lang-expression:processResources NO-SOURCE > Task :libs:opensearch-common:classes UP-TO-DATE > Task :modules:ingest-user-agent:processResources > Task :libs:opensearch-task-commons:compileJava UP-TO-DATE > Task :libs:opensearch-cli:compileJava UP-TO-DATE > Task :server:extractIncludeProto UP-TO-DATE > Task :libs:opensearch-common:jar UP-TO-DATE > Task :libs:opensearch-telemetry:compileJava UP-TO-DATE > Task :libs:opensearch-core:compileJava UP-TO-DATE > Task :server:generateProto UP-TO-DATE > Task :modules:ingest-user-agent:copyPluginPropertiesTemplate > Task :modules:lang-expression:copyPluginPropertiesTemplate > Task :modules:aggs-matrix-stats:pluginProperties > Task :modules:cache-common:pluginProperties > Task :modules:geo:pluginProperties > Task :libs:opensearch-compress:compileJava UP-TO-DATE > Task :libs:opensearch-x-content:compileJava UP-TO-DATE > Task :libs:opensearch-dissect:compileJava > Task :libs:opensearch-grok:compileJava > Task :libs:opensearch-cli:classes UP-TO-DATE > Task :libs:opensearch-compress:classes UP-TO-DATE > Task :libs:opensearch-core:classes UP-TO-DATE > Task :modules:ingest-geoip:copyDefaultGeoIp2DatabaseFiles > Task :modules:lang-mustache:processResources NO-SOURCE > Task :modules:lang-mustache:copyPluginPropertiesTemplate > Task :server:generateModulesList > Task :modules:analysis-common:pluginProperties > Task :modules:ingest-common:pluginProperties > Task :libs:opensearch-secure-sm:classes UP-TO-DATE > Task :libs:opensearch-geo:classes UP-TO-DATE > Task :libs:opensearch-compress:jar > Task :libs:opensearch-cli:jar > Task :libs:opensearch-task-commons:classes UP-TO-DATE > Task :libs:opensearch-telemetry:classes UP-TO-DATE > Task :modules:ingest-user-agent:pluginProperties > Task :modules:lang-expression:pluginProperties > Task :libs:opensearch-x-content:classes UP-TO-DATE > Task :libs:opensearch-core:jar > Task :libs:opensearch-secure-sm:jar > Task :libs:opensearch-dissect:processResources NO-SOURCE > Task :libs:opensearch-geo:jar > Task :libs:opensearch-dissect:classes > Task :modules:mapper-extras:processResources NO-SOURCE > Task :modules:lang-painless:spi:processResources NO-SOURCE > Task :libs:opensearch-task-commons:jar > Task :libs:opensearch-telemetry:jar > Task :modules:mapper-extras:copyPluginPropertiesTemplate > Task :modules:lang-painless:processResources > Task :modules:opensearch-dashboards:processResources NO-SOURCE > Task :modules:reindex:processResources NO-SOURCE > Task :modules:lang-mustache:pluginProperties > Task :modules:lang-painless:copyPluginPropertiesTemplate > Task :modules:opensearch-dashboards:copyPluginPropertiesTemplate > Task :server:compileJava > Task :modules:parent-join:processResources > Task :modules:parent-join:copyPluginPropertiesTemplate > Task :libs:opensearch-dissect:jar > Task :libs:opensearch-x-content:jar > Task :modules:transport-netty4:copyPluginPropertiesTemplate > Task :modules:percolator:processResources NO-SOURCE > Task :modules:mapper-extras:pluginProperties > Task :modules:lang-painless:pluginProperties > Task :modules:rank-eval:processResources > Task :modules:opensearch-dashboards:pluginProperties > Task :modules:reindex:copyPluginPropertiesTemplate > Task :modules:search-pipeline-common:processResources NO-SOURCE > Task :modules:percolator:copyPluginPropertiesTemplate > Task :modules:repository-url:processResources NO-SOURCE > Task :modules:rank-eval:copyPluginPropertiesTemplate > Task :modules:repository-url:copyPluginPropertiesTemplate > Task :modules:search-pipeline-common:copyPluginPropertiesTemplate > Task :client:rest:compileJava > Task :libs:opensearch-ssl-config:compileJava > Task :modules:parent-join:pluginProperties > Task :modules:transport-netty4:pluginProperties > Task :modules:percolator:pluginProperties > Task :modules:rank-eval:pluginProperties > Task :modules:reindex:pluginProperties > Task :modules:transport-netty4:processResources NO-SOURCE > Task :modules:systemd:processResources NO-SOURCE > Task :libs:opensearch-plugin-classloader:classes UP-TO-DATE > Task :modules:systemd:copyPluginPropertiesTemplate UP-TO-DATE > Task :modules:reindex:copyParentJoinMetadata > Task :modules:systemd:pluginProperties UP-TO-DATE > Task :modules:repository-url:pluginProperties > Task :modules:search-pipeline-common:pluginProperties > Task :distribution:archives:createJvmOptionsDir > Task :distribution:tools:launchers:processResources NO-SOURCE > Task :distribution:tools:keystore-cli:processResources NO-SOURCE > Task :distribution:archives:createLogsDir > Task :distribution:tools:upgrade-cli:processResources NO-SOURCE > Task :distribution:archives:createPluginsDir > Task :modules:reindex:copyTransportNetty4Metadata > Task :distribution:tools:plugin-cli:processResources > Task :distribution:archives:no-jdk-linux-riscv64-tar:distTar NO-SOURCE > Task :distribution:archives:no-jdk-linux-riscv64-tar:distZip NO-SOURCE > Task :libs:opensearch-plugin-classloader:jar > Task :libs:opensearch-grok:processResources > Task :libs:opensearch-grok:classes > Task :libs:opensearch-grok:jar > Task :libs:opensearch-ssl-config:processResources NO-SOURCE > Task :libs:opensearch-ssl-config:classes > Task :server:compileJava UP-TO-DATE > Task :libs:opensearch-ssl-config:jar > Task :distribution:buildNoJdkNotice > Task :distribution:tools:java-version-checker:compileJava > Task :server:generatePluginsList > Task :client:rest:compileJava Note: /build/opensearch/src/OpenSearch-2.19.1/client/rest/src/main/java/org/opensearch/client/ReactiveRequestProducer.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:lang-painless:spi:compileJava > Task :modules:geo:compileJava > Task :modules:lang-mustache:compileJava > Task :modules:ingest-user-agent:compileJava > Task :modules:cache-common:compileJava > Task :modules:lang-expression:compileJava > Task :modules:aggs-matrix-stats:compileJava > Task :server:processResources > Task :server:classes > Task :modules:mapper-extras:compileJava > Task :server:jar > Task :modules:cache-common:classes > Task :modules:cache-common:jar > Task :modules:ingest-user-agent:classes > Task :client:rest:processResources NO-SOURCE > Task :client:rest:classes > Task :modules:lang-expression:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:ingest-user-agent:jar > Task :client:rest:jar > Task :modules:lang-mustache:compileJava Note: /build/opensearch/src/OpenSearch-2.19.1/modules/lang-mustache/src/main/java/org/opensearch/script/mustache/MustacheScriptEngine.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:cache-common:bundlePlugin > Task :modules:mapper-extras:classes > Task :modules:lang-expression:classes > Task :modules:lang-mustache:classes > Task :modules:mapper-extras:jar > Task :modules:ingest-user-agent:bundlePlugin > Task :modules:reindex:compileJava > Task :modules:lang-mustache:jar > Task :modules:lang-expression:jar > Task :modules:mapper-extras:bundlePlugin > Task :modules:ingest-geoip:compileJava > Task :modules:lang-mustache:bundlePlugin > Task :modules:lang-expression:bundlePlugin > Task :modules:ingest-geoip:processResources NO-SOURCE > Task :modules:ingest-geoip:classes > Task :modules:lang-painless:spi:classes > Task :modules:ingest-geoip:jar > Task :modules:lang-painless:spi:jar > Task :modules:ingest-geoip:copyPluginPropertiesTemplate > Task :modules:lang-painless:compileJava > Task :modules:aggs-matrix-stats:compileJava Note: /build/opensearch/src/OpenSearch-2.19.1/modules/aggs-matrix-stats/src/main/java/org/opensearch/search/aggregations/matrix/MatrixAggregationPlugin.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:ingest-geoip:pluginProperties > Task :modules:aggs-matrix-stats:classes > Task :modules:aggs-matrix-stats:jar > Task :modules:parent-join:compileJava > Task :modules:aggs-matrix-stats:bundlePlugin > Task :distribution:tools:java-version-checker:processResources NO-SOURCE > Task :distribution:tools:java-version-checker:classes > Task :modules:systemd:compileJava UP-TO-DATE > Task :modules:systemd:classes UP-TO-DATE > Task :modules:geo:compileJava Note: /build/opensearch/src/OpenSearch-2.19.1/modules/geo/src/main/java/org/opensearch/geo/GeoModulePlugin.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. > Task :modules:ingest-geoip:bundlePlugin > Task :distribution:tools:java-version-checker:jar > Task :modules:systemd:jar UP-TO-DATE > Task :modules:reindex:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:percolator:compileJava > Task :modules:transport-netty4:compileJava > Task :modules:repository-url:compileJava > Task :modules:rank-eval:compileJava > Task :distribution:tools:keystore-cli:compileJava > Task :modules:systemd:bundlePlugin UP-TO-DATE > Task :distribution:processSystemdOutputs UP-TO-DATE > Task :distribution:buildSystemdModule UP-TO-DATE > Task :modules:lang-painless:compileJava Note: /build/opensearch/src/OpenSearch-2.19.1/modules/lang-painless/src/main/java/org/opensearch/painless/antlr/Walker.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :distribution:tools:launchers:compileJava > Task :modules:geo:classes > Task :modules:percolator:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:parent-join:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:geo:jar > Task :distribution:tools:plugin-cli:compileJava > Task :modules:reindex:classes > Task :modules:percolator:classes > Task :modules:parent-join:classes > Task :modules:opensearch-dashboards:compileJava > Task :modules:parent-join:jar > Task :modules:percolator:jar > Task :modules:reindex:jar > Task :modules:geo:bundlePlugin > Task :modules:repository-url:classes > Task :distribution:tools:launchers:classes > Task :modules:parent-join:bundlePlugin > Task :modules:percolator:bundlePlugin > Task :modules:opensearch-dashboards:classes > Task :modules:repository-url:jar > Task :distribution:tools:launchers:jar > Task :modules:reindex:bundlePlugin > Task :modules:lang-painless:classes > Task :modules:opensearch-dashboards:jar > Task :modules:transport-netty4:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:repository-url:bundlePlugin > Task :modules:opensearch-dashboards:bundlePlugin > Task :distribution:tools:upgrade-cli:compileJava > Task :modules:lang-painless:jar > Task :modules:analysis-common:compileJava > Task :modules:search-pipeline-common:compileJava > Task :modules:ingest-common:compileJava > Task :modules:lang-painless:bundlePlugin > Task :distribution:tools:upgrade-cli:classes > Task :distribution:tools:keystore-cli:classes > Task :distribution:tools:keystore-cli:jar > Task :distribution:tools:upgrade-cli:jar > Task :modules:rank-eval:classes > Task :modules:transport-netty4:classes > Task :modules:rank-eval:jar > Task :modules:transport-netty4:jar > Task :modules:rank-eval:bundlePlugin > Task :modules:transport-netty4:bundlePlugin > Task :modules:search-pipeline-common:classes > Task :distribution:tools:plugin-cli:compileJava Note: /build/opensearch/src/OpenSearch-2.19.1/distribution/tools/plugin-cli/src/main/java/org/opensearch/plugins/InstallPluginCommand.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:search-pipeline-common:jar > Task :modules:search-pipeline-common:bundlePlugin > Task :distribution:tools:plugin-cli:classes > Task :distribution:tools:plugin-cli:jar > Task :modules:ingest-common:classes > Task :modules:ingest-common:jar > Task :modules:ingest-common:bundlePlugin > Task :modules:analysis-common:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:analysis-common:classes > Task :modules:analysis-common:jar > Task :modules:analysis-common:bundlePlugin > Task :distribution:processOutputs > Task :distribution:buildBin > Task :distribution:buildConfig > Task :distribution:buildLog4jConfig > Task :distribution:buildModules > Task :distribution:archives:buildNoJdkLinuxRiscv64 > Task :distribution:archives:buildNoJdkLinuxRiscv64Tar > Task :distribution:archives:no-jdk-linux-riscv64-tar:assemble > Task :distribution:archives:no-jdk-linux-riscv64-tar:checkExtraction > Task :distribution:archives:no-jdk-linux-riscv64-tar:checkLicense > Task :distribution:archives:no-jdk-linux-riscv64-tar:checkNotice > Task :distribution:archives:no-jdk-linux-riscv64-tar:check > Task :distribution:archives:no-jdk-linux-riscv64-tar:build [Incubating] Problems report is available at: file:///build/opensearch/src/OpenSearch-2.19.1/build/reports/problems/problems-report.html BUILD SUCCESSFUL in 30m 47s 184 actionable tasks: 151 executed, 33 up-to-date ======================== Building plugin analysis-icu ======================== > Task :buildSrc:reaper:compileJava UP-TO-DATE > Task :buildSrc:reaper:processResources NO-SOURCE > Task :buildSrc:reaper:classes UP-TO-DATE > Task :buildSrc:reaper:jar UP-TO-DATE > Task :buildSrc:compileJava UP-TO-DATE > Task :buildSrc:compileGroovy UP-TO-DATE > Task :buildSrc:generateVersionProperties UP-TO-DATE > Task :buildSrc:pluginDescriptors UP-TO-DATE > Task :buildSrc:processResources UP-TO-DATE > Task :buildSrc:classes UP-TO-DATE > Task :buildSrc:jar UP-TO-DATE ======================================= OpenSearch Build Hamster says Hello! Gradle Version : 8.12.1 OS Info : Linux 6.13.5-arch1-1 (riscv64) JDK Version : 17 (Arch Linux JDK) JAVA_HOME : /usr/lib/jvm/java-17-openjdk Random Testing Seed : 298D86C07ECB082C In FIPS 140 mode : false ======================================= > Task :server:protobufDummy UP-TO-DATE > Task :libs:opensearch-plugin-classloader:processResources NO-SOURCE > Task :libs:opensearch-cli:processResources NO-SOURCE > Task :libs:opensearch-task-commons:processResources NO-SOURCE > Task :libs:opensearch-secure-sm:processResources NO-SOURCE > Task :libs:opensearch-geo:processResources NO-SOURCE > Task :libs:opensearch-telemetry:processResources NO-SOURCE > Task :libs:opensearch-core:processResources UP-TO-DATE > Task :libs:opensearch-common:processResources UP-TO-DATE > Task :libs:opensearch-compress:processResources UP-TO-DATE > Task :libs:opensearch-x-content:processResources UP-TO-DATE > Task :server:extractProto UP-TO-DATE > Task :test:telemetry:processResources NO-SOURCE > Task :client:sniffer:processResources NO-SOURCE > Task :libs:opensearch-nio:processResources NO-SOURCE > Task :libs:opensearch-plugin-classloader:compileJava UP-TO-DATE > Task :libs:opensearch-geo:compileJava UP-TO-DATE > Task :libs:opensearch-secure-sm:compileJava UP-TO-DATE > Task :client:rest:compileJava UP-TO-DATE > Task :libs:opensearch-common:compileJava UP-TO-DATE > Task :libs:opensearch-common:classes UP-TO-DATE > Task :test:framework:processResources > Task :server:extractIncludeProto UP-TO-DATE > Task :modules:aggs-matrix-stats:processResources UP-TO-DATE > Task :plugins:analysis-icu:generateNotice > Task :libs:opensearch-cli:compileJava UP-TO-DATE > Task :libs:opensearch-task-commons:compileJava UP-TO-DATE > Task :libs:opensearch-telemetry:compileJava UP-TO-DATE > Task :plugins:analysis-icu:processResources NO-SOURCE > Task :client:rest:processResources NO-SOURCE > Task :libs:opensearch-secure-sm:classes UP-TO-DATE > Task :libs:opensearch-cli:classes UP-TO-DATE > Task :client:rest:classes UP-TO-DATE > Task :libs:opensearch-geo:classes UP-TO-DATE > Task :libs:opensearch-core:compileJava UP-TO-DATE > Task :server:generateProto UP-TO-DATE > Task :libs:opensearch-common:jar UP-TO-DATE > Task :plugins:analysis-icu:copyPluginPropertiesTemplate > Task :libs:opensearch-cli:jar UP-TO-DATE > Task :libs:opensearch-geo:jar UP-TO-DATE > Task :client:rest:jar UP-TO-DATE > Task :libs:opensearch-secure-sm:jar UP-TO-DATE > Task :libs:opensearch-core:classes UP-TO-DATE > Task :libs:opensearch-telemetry:classes UP-TO-DATE > Task :libs:opensearch-task-commons:classes UP-TO-DATE > Task :libs:opensearch-compress:compileJava UP-TO-DATE > Task :libs:opensearch-x-content:compileJava UP-TO-DATE > Task :server:generateModulesList UP-TO-DATE > Task :libs:opensearch-compress:classes UP-TO-DATE > Task :libs:opensearch-x-content:classes UP-TO-DATE > Task :test:logger-usage:compileJava > Task :libs:opensearch-core:jar UP-TO-DATE > Task :libs:opensearch-x-content:jar UP-TO-DATE > Task :libs:opensearch-task-commons:jar UP-TO-DATE > Task :libs:opensearch-telemetry:jar UP-TO-DATE > Task :libs:opensearch-compress:jar UP-TO-DATE > Task :spotlessInternalRegisterDependencies > Task :modules:cache-common:processResources NO-SOURCE > Task :modules:geo:processResources NO-SOURCE > Task :client:sniffer:compileJava > Task :libs:opensearch-nio:compileJava > Task :modules:cache-common:copyPluginPropertiesTemplate UP-TO-DATE > Task :modules:geo:copyPluginPropertiesTemplate UP-TO-DATE > Task :modules:analysis-common:processResources UP-TO-DATE > Task :modules:cache-common:pluginProperties UP-TO-DATE > Task :client:sniffer:compileJava Note: /build/opensearch/src/OpenSearch-2.19.1/client/sniffer/src/main/java/org/opensearch/client/sniff/OpenSearchNodesSniffer.java uses or overrides a deprecated API. > Task :modules:geo:pluginProperties UP-TO-DATE > Task :modules:analysis-common:copyPluginPropertiesTemplate UP-TO-DATE > Task :client:sniffer:compileJava Note: Recompile with -Xlint:deprecation for details. > Task :missing-doclet:compileJava > Task :modules:analysis-common:pluginProperties UP-TO-DATE > Task :libs:opensearch-dissect:compileJava UP-TO-DATE > Task :modules:ingest-common:processResources UP-TO-DATE > Task :libs:opensearch-dissect:processResources NO-SOURCE > Task :modules:ingest-user-agent:processResources UP-TO-DATE > Task :modules:ingest-common:copyPluginPropertiesTemplate UP-TO-DATE > Task :libs:opensearch-dissect:classes UP-TO-DATE > Task :modules:ingest-user-agent:copyPluginPropertiesTemplate UP-TO-DATE > Task :test:telemetry:compileJava > Task :plugins:analysis-icu:pluginProperties > Task :modules:ingest-user-agent:pluginProperties UP-TO-DATE > Task :modules:ingest-common:pluginProperties UP-TO-DATE > Task :libs:opensearch-grok:compileJava UP-TO-DATE > Task :server:compileJava > Task :modules:lang-expression:processResources NO-SOURCE > Task :modules:lang-mustache:processResources NO-SOURCE > Task :modules:ingest-geoip:copyDefaultGeoIp2DatabaseFiles UP-TO-DATE > Task :modules:lang-mustache:copyPluginPropertiesTemplate UP-TO-DATE > Task :modules:lang-expression:copyPluginPropertiesTemplate UP-TO-DATE > Task :modules:ingest-geoip:processResources NO-SOURCE > Task :libs:opensearch-grok:processResources UP-TO-DATE > Task :modules:lang-expression:pluginProperties UP-TO-DATE > Task :modules:lang-mustache:pluginProperties UP-TO-DATE > Task :libs:opensearch-dissect:jar UP-TO-DATE > Task :libs:opensearch-grok:classes UP-TO-DATE > Task :libs:opensearch-nio:classes > Task :client:sniffer:classes > Task :modules:ingest-geoip:copyPluginPropertiesTemplate UP-TO-DATE > Task :modules:lang-painless:spi:processResources NO-SOURCE > Task :modules:mapper-extras:processResources NO-SOURCE > Task :modules:mapper-extras:copyPluginPropertiesTemplate UP-TO-DATE > Task :modules:aggs-matrix-stats:copyPluginPropertiesTemplate > Task :modules:ingest-geoip:pluginProperties UP-TO-DATE > Task :modules:lang-painless:processResources UP-TO-DATE > Task :modules:mapper-extras:pluginProperties UP-TO-DATE > Task :modules:lang-painless:copyPluginPropertiesTemplate UP-TO-DATE > Task :modules:opensearch-dashboards:processResources NO-SOURCE > Task :modules:reindex:processResources NO-SOURCE > Task :modules:lang-painless:pluginProperties UP-TO-DATE > Task :modules:opensearch-dashboards:copyPluginPropertiesTemplate UP-TO-DATE > Task :modules:opensearch-dashboards:pluginProperties UP-TO-DATE > Task :libs:opensearch-grok:jar UP-TO-DATE > Task :modules:parent-join:processResources UP-TO-DATE > Task :libs:opensearch-ssl-config:compileJava UP-TO-DATE > Task :modules:parent-join:copyPluginPropertiesTemplate UP-TO-DATE > Task :modules:transport-netty4:copyPluginPropertiesTemplate UP-TO-DATE > Task :modules:rank-eval:processResources UP-TO-DATE > Task :modules:parent-join:pluginProperties UP-TO-DATE > Task :modules:transport-netty4:pluginProperties UP-TO-DATE > Task :libs:opensearch-ssl-config:processResources NO-SOURCE > Task :modules:rank-eval:copyPluginPropertiesTemplate UP-TO-DATE > Task :libs:opensearch-ssl-config:classes UP-TO-DATE > Task :modules:repository-url:processResources NO-SOURCE > Task :modules:rank-eval:pluginProperties UP-TO-DATE > Task :modules:repository-url:copyPluginPropertiesTemplate UP-TO-DATE > Task :modules:reindex:copyParentJoinMetadata UP-TO-DATE > Task :modules:search-pipeline-common:processResources NO-SOURCE > Task :modules:repository-url:pluginProperties UP-TO-DATE > Task :modules:reindex:copyTransportNetty4Metadata UP-TO-DATE > Task :modules:search-pipeline-common:copyPluginPropertiesTemplate UP-TO-DATE > Task :modules:transport-netty4:processResources NO-SOURCE > Task :modules:reindex:copyPluginPropertiesTemplate UP-TO-DATE > Task :modules:search-pipeline-common:pluginProperties UP-TO-DATE > Task :libs:opensearch-ssl-config:jar UP-TO-DATE > Task :libs:opensearch-plugin-classloader:classes UP-TO-DATE > Task :libs:opensearch-nio:jar > Task :client:sniffer:jar > Task :distribution:archives:createJvmOptionsDir > Task :distribution:archives:createLogsDir > Task :distribution:archives:createPluginsDir > Task :distribution:tools:keystore-cli:processResources NO-SOURCE > Task :distribution:tools:launchers:processResources NO-SOURCE > Task :modules:percolator:processResources NO-SOURCE > Task :modules:percolator:copyPluginPropertiesTemplate UP-TO-DATE > Task :modules:aggs-matrix-stats:copyPluginPropertiesTemplate UP-TO-DATE > Task :distribution:tools:upgrade-cli:processResources NO-SOURCE > Task :distribution:tools:plugin-cli:processResources UP-TO-DATE > Task :modules:aggs-matrix-stats:pluginProperties UP-TO-DATE > Task :modules:percolator:pluginProperties UP-TO-DATE > Task :distribution:tools:java-version-checker:compileJava UP-TO-DATE > Task :distribution:tools:java-version-checker:processResources NO-SOURCE > Task :distribution:buildNoJdkNotice UP-TO-DATE > Task :distribution:tools:java-version-checker:classes UP-TO-DATE > Task :libs:opensearch-plugin-classloader:jar UP-TO-DATE > Task :plugins:analysis-icu:generatePomFileForNebulaPublication > Task :modules:reindex:pluginProperties UP-TO-DATE > Task :distribution:tools:launchers:compileJava UP-TO-DATE > Task :distribution:tools:launchers:classes UP-TO-DATE > Task :distribution:tools:java-version-checker:jar UP-TO-DATE > Task :test:telemetry:classes > Task :distribution:tools:launchers:jar UP-TO-DATE > Task :test:telemetry:jar > Task :plugins:analysis-icu:generatePom > Task :server:compileJava UP-TO-DATE > Task :server:generatePluginsList UP-TO-DATE > Task :test:logger-usage:processResources NO-SOURCE > Task :test:logger-usage:classes > Task :modules:lang-painless:spi:compileJava UP-TO-DATE > Task :modules:ingest-geoip:compileJava UP-TO-DATE > Task :modules:cache-common:compileJava UP-TO-DATE > Task :modules:aggs-matrix-stats:compileJava UP-TO-DATE > Task :modules:geo:compileJava UP-TO-DATE > Task :modules:ingest-user-agent:compileJava UP-TO-DATE > Task :modules:geo:classes UP-TO-DATE > Task :modules:ingest-user-agent:classes UP-TO-DATE > Task :modules:ingest-geoip:classes UP-TO-DATE > Task :modules:aggs-matrix-stats:classes UP-TO-DATE > Task :modules:cache-common:classes UP-TO-DATE > Task :modules:lang-expression:compileJava UP-TO-DATE > Task :modules:aggs-matrix-stats:jar UP-TO-DATE > Task :modules:ingest-user-agent:jar UP-TO-DATE > Task :modules:cache-common:jar UP-TO-DATE > Task :modules:geo:jar UP-TO-DATE > Task :modules:ingest-geoip:jar UP-TO-DATE > Task :modules:lang-expression:classes UP-TO-DATE > Task :plugins:analysis-icu:sourcesJar > Task :test:logger-usage:jar > Task :modules:lang-painless:spi:classes UP-TO-DATE > Task :server:processResources UP-TO-DATE > Task :server:classes UP-TO-DATE > Task :modules:lang-painless:compileJava UP-TO-DATE > Task :modules:lang-mustache:compileJava UP-TO-DATE > Task :modules:lang-painless:spi:jar UP-TO-DATE > Task :modules:lang-expression:jar UP-TO-DATE > Task :modules:lang-mustache:classes UP-TO-DATE > Task :server:jar UP-TO-DATE > Task :modules:lang-painless:classes UP-TO-DATE > Task :modules:ingest-user-agent:bundlePlugin UP-TO-DATE > Task :modules:cache-common:bundlePlugin UP-TO-DATE > Task :modules:ingest-geoip:bundlePlugin UP-TO-DATE > Task :modules:geo:bundlePlugin UP-TO-DATE > Task :modules:lang-mustache:jar UP-TO-DATE > Task :modules:ingest-common:compileJava UP-TO-DATE > Task :modules:lang-painless:jar UP-TO-DATE > Task :modules:ingest-common:classes UP-TO-DATE > Task :modules:percolator:compileJava UP-TO-DATE > Task :modules:reindex:compileJava UP-TO-DATE > Task :modules:mapper-extras:compileJava UP-TO-DATE > Task :modules:analysis-common:compileJava UP-TO-DATE > Task :modules:parent-join:compileJava UP-TO-DATE > Task :modules:analysis-common:classes UP-TO-DATE > Task :modules:mapper-extras:classes UP-TO-DATE > Task :modules:parent-join:classes UP-TO-DATE > Task :modules:reindex:classes UP-TO-DATE > Task :test:framework:compileJava > Task :modules:opensearch-dashboards:compileJava UP-TO-DATE > Task :modules:ingest-common:jar UP-TO-DATE > Task :modules:parent-join:jar UP-TO-DATE > Task :modules:mapper-extras:jar UP-TO-DATE > Task :modules:analysis-common:jar UP-TO-DATE > Task :modules:reindex:jar UP-TO-DATE > Task :modules:opensearch-dashboards:classes UP-TO-DATE > Task :modules:percolator:classes UP-TO-DATE > Task :modules:lang-painless:bundlePlugin UP-TO-DATE > Task :modules:lang-expression:bundlePlugin UP-TO-DATE > Task :modules:lang-mustache:bundlePlugin UP-TO-DATE > Task :modules:aggs-matrix-stats:bundlePlugin > Task :plugins:analysis-icu:compileJava > Task :modules:rank-eval:compileJava UP-TO-DATE > Task :modules:repository-url:compileJava UP-TO-DATE > Task :modules:opensearch-dashboards:jar UP-TO-DATE > Task :modules:aggs-matrix-stats:bundlePlugin UP-TO-DATE > Task :modules:parent-join:bundlePlugin UP-TO-DATE > Task :modules:mapper-extras:bundlePlugin UP-TO-DATE > Task :modules:percolator:jar UP-TO-DATE > Task :modules:reindex:bundlePlugin UP-TO-DATE > Task :modules:repository-url:classes UP-TO-DATE > Task :modules:rank-eval:classes UP-TO-DATE > Task :modules:analysis-common:bundlePlugin UP-TO-DATE > Task :modules:search-pipeline-common:compileJava UP-TO-DATE > Task :distribution:tools:keystore-cli:compileJava UP-TO-DATE > Task :distribution:tools:plugin-cli:compileJava UP-TO-DATE > Task :distribution:tools:keystore-cli:classes UP-TO-DATE > Task :modules:search-pipeline-common:classes UP-TO-DATE > Task :distribution:tools:plugin-cli:classes UP-TO-DATE > Task :modules:percolator:bundlePlugin UP-TO-DATE > Task :modules:opensearch-dashboards:bundlePlugin UP-TO-DATE > Task :modules:ingest-common:bundlePlugin UP-TO-DATE > Task :modules:transport-netty4:compileJava UP-TO-DATE > Task :modules:transport-netty4:classes UP-TO-DATE > Task :modules:rank-eval:jar UP-TO-DATE > Task :distribution:tools:upgrade-cli:compileJava UP-TO-DATE > Task :modules:repository-url:jar UP-TO-DATE > Task :distribution:tools:upgrade-cli:classes UP-TO-DATE > Task :distribution:tools:keystore-cli:jar UP-TO-DATE > Task :distribution:tools:plugin-cli:jar UP-TO-DATE > Task :modules:search-pipeline-common:jar UP-TO-DATE > Task :distribution:tools:upgrade-cli:jar UP-TO-DATE > Task :modules:transport-netty4:jar UP-TO-DATE > Task :modules:search-pipeline-common:bundlePlugin UP-TO-DATE > Task :modules:transport-netty4:bundlePlugin UP-TO-DATE > Task :modules:rank-eval:bundlePlugin UP-TO-DATE > Task :modules:repository-url:bundlePlugin UP-TO-DATE > Task :distribution:processOutputs UP-TO-DATE > Task :distribution:buildBin UP-TO-DATE > Task :distribution:buildConfig UP-TO-DATE > Task :distribution:buildLog4jConfig UP-TO-DATE > Task :distribution:processTransportOutputs > Task :distribution:buildTransportModules > Task :missing-doclet:processResources NO-SOURCE > Task :missing-doclet:classes > Task :missing-doclet:jar > Task :plugins:analysis-icu:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :libs:opensearch-plugin-classloader:missingJavadoc > Task :distribution:archives:buildIntegTest > Task :server:missingJavadoc > Task :libs:opensearch-plugin-classloader:javadoc > Task :plugins:analysis-icu:classes > Task :plugins:analysis-icu:jar > Task :plugins:analysis-icu:bundlePlugin > Task :plugins:analysis-icu:missingJavadoc > Task :plugins:analysis-icu:copyYamlTestsTask NO-SOURCE > Task :plugins:analysis-icu:copyRestApiSpecsTask > Task :plugins:analysis-icu:processTestResources > Task :plugins:analysis-icu:forbiddenApisResources > Task :plugins:analysis-icu:processInternalClusterTestResources NO-SOURCE > Task :plugins:analysis-icu:forbiddenApisMain > Task :plugins:analysis-icu:processYamlRestTestResources > Task :plugins:analysis-icu:dependencyLicenses > Task :plugins:analysis-icu:filepermissions > Task :plugins:analysis-icu:forbiddenPatterns > Task :test:framework:compileJava Note: Processing Log4j annotations Note: Annotations processed Note: Processing Log4j annotations Note: No elements to process > Task :plugins:analysis-icu:licenseHeaders > Task :plugins:analysis-icu:spotlessJava > Task :plugins:analysis-icu:spotlessJavaCheck > Task :plugins:analysis-icu:thirdPartyAuditResources > Task :plugins:analysis-icu:thirdPartyAudit > Task :plugins:analysis-icu:validateNebulaPom > Task :plugins:analysis-icu:validatePom > Task :server:javadoc > Task :plugins:analysis-icu:spotlessMisc > Task :plugins:analysis-icu:spotlessMiscCheck > Task :plugins:analysis-icu:spotlessCheck > Task :test:framework:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. > Task :test:framework:classes > Task :test:framework:jar > Task :plugins:analysis-icu:compileTestJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :plugins:analysis-icu:testClasses > Task :plugins:analysis-icu:compileInternalClusterTestJava > Task :plugins:analysis-icu:internalClusterTestClasses > Task :plugins:analysis-icu:forbiddenApisInternalClusterTest > Task :plugins:analysis-icu:forbiddenApisTest > Task :plugins:analysis-icu:compileYamlRestTestJava > Task :plugins:analysis-icu:yamlRestTestClasses > Task :plugins:analysis-icu:forbiddenApisYamlRestTest > Task :plugins:analysis-icu:forbiddenApis > Task :plugins:analysis-icu:jarHell > Task :plugins:analysis-icu:loggerUsageCheck > Task :plugins:analysis-icu:testingConventions > Task :plugins:analysis-icu:precommit > Task :plugins:analysis-icu:test May 08, 2025 9:26:40 PM sun.util.locale.provider.LocaleProviderAdapter May 08, 2025 9:26:40 PM sun.util.locale.provider.LocaleProviderAdapter WARNING: COMPAT locale provider will be removed in a future release WARNING: COMPAT locale provider will be removed in a future release May 08, 2025 9:26:40 PM sun.util.locale.provider.LocaleProviderAdapter WARNING: COMPAT locale provider will be removed in a future release May 08, 2025 9:26:40 PM sun.util.locale.provider.LocaleProviderAdapter WARNING: COMPAT locale provider will be removed in a future release May 08, 2025 9:26:40 PM sun.util.locale.provider.LocaleProviderAdapter WARNING: COMPAT locale provider will be removed in a future release May 08, 2025 9:26:40 PM sun.util.locale.provider.LocaleProviderAdapter WARNING: COMPAT locale provider will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/build/opensearch/src/OpenSearch-2.19.1/test/framework/build/distributions/framework-2.19.1.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting WARNING: System::setSecurityManager will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/build/opensearch/src/OpenSearch-2.19.1/test/framework/build/distributions/framework-2.19.1.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting WARNING: System::setSecurityManager will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/build/opensearch/src/OpenSearch-2.19.1/test/framework/build/distributions/framework-2.19.1.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting WARNING: System::setSecurityManager will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/build/opensearch/src/OpenSearch-2.19.1/test/framework/build/distributions/framework-2.19.1.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting WARNING: System::setSecurityManager will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/build/opensearch/src/OpenSearch-2.19.1/test/framework/build/distributions/framework-2.19.1.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting WARNING: System::setSecurityManager will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/build/opensearch/src/OpenSearch-2.19.1/test/framework/build/distributions/framework-2.19.1.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting WARNING: System::setSecurityManager will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/build/.gradle/wrapper/dists/gradle-8.12.1-all/5akij8gnr65tdxupapyu22pqa/gradle-8.12.1/lib/plugins/gradle-testing-base-infrastructure-8.12.1.jar) WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker WARNING: System::setSecurityManager will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/build/.gradle/wrapper/dists/gradle-8.12.1-all/5akij8gnr65tdxupapyu22pqa/gradle-8.12.1/lib/plugins/gradle-testing-base-infrastructure-8.12.1.jar) WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker WARNING: System::setSecurityManager will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/build/.gradle/wrapper/dists/gradle-8.12.1-all/5akij8gnr65tdxupapyu22pqa/gradle-8.12.1/lib/plugins/gradle-testing-base-infrastructure-8.12.1.jar) WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker WARNING: System::setSecurityManager will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/build/.gradle/wrapper/dists/gradle-8.12.1-all/5akij8gnr65tdxupapyu22pqa/gradle-8.12.1/lib/plugins/gradle-testing-base-infrastructure-8.12.1.jar) WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker WARNING: System::setSecurityManager will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/build/.gradle/wrapper/dists/gradle-8.12.1-all/5akij8gnr65tdxupapyu22pqa/gradle-8.12.1/lib/plugins/gradle-testing-base-infrastructure-8.12.1.jar) WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker WARNING: System::setSecurityManager will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/build/.gradle/wrapper/dists/gradle-8.12.1-all/5akij8gnr65tdxupapyu22pqa/gradle-8.12.1/lib/plugins/gradle-testing-base-infrastructure-8.12.1.jar) WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker WARNING: System::setSecurityManager will be removed in a future release > Task :plugins:analysis-icu:internalClusterTest May 08, 2025 9:31:24 PM sun.util.locale.provider.LocaleProviderAdapter WARNING: COMPAT locale provider will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/build/opensearch/src/OpenSearch-2.19.1/test/framework/build/distributions/framework-2.19.1.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting WARNING: System::setSecurityManager will be removed in a future release > Task :plugins:analysis-icu:internalClusterTest REPRODUCE WITH: ./gradlew ':plugins:analysis-icu:internalClusterTest' --tests "org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT.testNormalization" -Dtests.seed=298D86C07ECB082C -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=hi-IN -Dtests.timezone=SystemV/PST8 -Druntime.java=21 ICUCollationKeywordFieldMapperIT > testNormalization FAILED java.lang.Exception: Test abandoned because suite timeout was reached. at __randomizedtesting.SeedInfo.seed([298D86C07ECB082C]:0) REPRODUCE WITH: ./gradlew ':plugins:analysis-icu:internalClusterTest' --tests "org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT.testNormalization" -Dtests.seed=298D86C07ECB082C -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=hi-IN -Dtests.timezone=SystemV/PST8 -Druntime.java=21 ICUCollationKeywordFieldMapperIT > classMethod FAILED java.lang.Exception: Suite timeout exceeded (>= 1200000 msec). at __randomizedtesting.SeedInfo.seed([298D86C07ECB082C]:0) Suite: Test class org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT 1> [2025-05-08T21:32:07,263][WARN ][o.o.b.JNANatives ] [[SUITE-ICUCollationKeywordFieldMapperIT-seed#[298D86C07ECB082C]]] unable to install syscall filter: 1> java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed 1> at org.opensearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:376) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:667) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:281) [opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:128) [opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:129) [opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.bootstrap.BootstrapForTesting.(BootstrapForTesting.java:108) [framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchTestCase.(OpenSearchTestCase.java:285) [framework-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Class.forName0(Native Method) ~[?:?] 1> at java.base/java.lang.Class.forName(Class.java:534) [?:?] 1> at java.base/java.lang.Class.forName(Class.java:513) [?:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:623) [randomizedtesting-runner-2.7.1.jar:?] 2> May 08, 2025 9:32:28 PM org.apache.lucene.internal.vectorization.VectorizationProvider lookup 2> WARNING: Java vector incubator module is not readable. For optimal vector performance, pass '--add-modules jdk.incubator.vector' to enable Vector API. 1> [2025-05-08T13:32:39,564][INFO ][o.o.t.OpenSearchTestClusterRule] [testIgnoreWhitespace] [ICUCollationKeywordFieldMapperIT#testIgnoreWhitespace]: setting up test 1> [2025-05-08T13:32:41,650][INFO ][o.o.t.InternalTestCluster] [testIgnoreWhitespace] Setup InternalTestCluster [SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster] with seed [29149C9C7DA617EA] using [3] dedicated cluster-managers, [2] (data) nodes and [1] coord only nodes (min_cluster_manager_nodes are [auto-managed]) 1> [2025-05-08T13:32:54,760][INFO ][o.o.n.Node ] [testIgnoreWhitespace] version[2.19.1], pid[46898], build[unknown/unknown/2025-05-08T15:34:16Z], OS[Linux/6.13.5-arch1-1/riscv64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.6/21.0.6+7-LTS] 1> [2025-05-08T13:32:54,790][INFO ][o.o.n.Node ] [testIgnoreWhitespace] JVM home [/build/.gradle/caches/8.12.1/transforms/ca9a730a464aebead200f2dfe1b82576/transformed/linux-21.0.6-riscv64.tar.gz] 1> [2025-05-08T13:32:54,965][DEPRECATION][o.o.d.n.Node ] [testIgnoreWhitespace] no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release 1> [2025-05-08T13:32:54,978][INFO ][o.o.n.Node ] [testIgnoreWhitespace] JVM arguments [-Dgradle.dist.lib=/build/.gradle/wrapper/dists/gradle-8.12.1-all/5akij8gnr65tdxupapyu22pqa/gradle-8.12.1/lib, -Dgradle.user.home=/build/.gradle, -Dgradle.worker.jar=/build/.gradle/caches/8.12.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djava.security.manager=allow, -Djna.nosys=true, -Dopensearch.scripting.update.ctx_in_params=false, -Dopensearch.search.rewrite_sort=true, -Dopensearch.transport.cname_in_publish_address=true, -Dorg.gradle.internal.worker.tmpdir=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/tmp/internalClusterTest/work, -Dtests.artifact=analysis-icu, -Dtests.gradle=true, -Dtests.jvm.argline=-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=298D86C07ECB082C, -Dtests.task=:plugins:analysis-icu:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:TieredStopAtLevel=1, -XX:ReservedCodeCacheSize=64m, -esa, -XX:HeapDumpPath=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/heapdump, -javaagent:/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/tmp/.cache/expanded/zip_ff7bf8f04f99dc306508d1c81e47a68b/jacocoagent.jar=destfile=../../jacoco/internalClusterTest.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false, -Xms512m, -Xmx3g, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp, -Duser.country, -Duser.language=en, -Duser.variant, -ea] 1> [2025-05-08T13:32:55,582][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] no modules loaded 1> [2025-05-08T13:32:55,607][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.cluster.service.applicationtemplates.TestSystemTemplatesRepositoryPlugin] 1> [2025-05-08T13:32:55,613][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.index.mapper.MockFieldFilterPlugin] 1> [2025-05-08T13:32:55,617][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.node.NodeMocksPlugin] 1> [2025-05-08T13:32:55,620][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.plugin.analysis.icu.AnalysisICUPlugin] 1> [2025-05-08T13:32:55,667][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.script.MockScriptService$TestPlugin] 1> [2025-05-08T13:32:55,676][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin] 1> [2025-05-08T13:32:55,691][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$AssertActionNamePlugin] 1> [2025-05-08T13:32:55,696][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$TestSeedPlugin] WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/build/.gradle/wrapper/dists/gradle-8.12.1-all/5akij8gnr65tdxupapyu22pqa/gradle-8.12.1/lib/plugins/gradle-testing-base-infrastructure-8.12.1.jar) WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker WARNING: System::setSecurityManager will be removed in a future release > Task :plugins:analysis-icu:internalClusterTest 1> [2025-05-08T13:32:55,700][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.TestGeoShapeFieldMapperPlugin] 1> [2025-05-08T13:32:55,705][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.store.MockFSIndexStore$TestPlugin] 1> [2025-05-08T13:32:55,710][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.telemetry.MockTelemetryPlugin] 1> [2025-05-08T13:32:55,722][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.transport.MockTransportService$TestPlugin] 1> [2025-05-08T13:32:55,730][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin] 2> WARNING: A restricted method in 2> java.lang.foreign.Linker 2> has been called 2> WARNING: 2> java.lang.foreign.Linker::downcallHandle 2> has been called by 2> the unnamed module 2> WARNING: Use --enable-native-access= 2> ALL-UNNAMED 2> to avoid a warning for this module 2> मई ०८, २०२५ १:३२:५९ अपराह्न org.apache.lucene.store.MemorySegmentIndexInputProvider 2> INFO: Using MemorySegmentIndexInput and native madvise support with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false 1> [2025-05-08T13:32:59,970][INFO ][o.o.e.NodeEnvironment ] [testIgnoreWhitespace] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [172.1gb], net total_space [914.7gb], types [ext4] 1> [2025-05-08T13:32:59,984][INFO ][o.o.e.NodeEnvironment ] [testIgnoreWhitespace] heap size [3gb], compressed ordinary object pointers [true] 1> [2025-05-08T13:33:01,851][INFO ][o.o.n.Node ] [testIgnoreWhitespace] node name [node_s0], node ID [48nm_5e6SW27203haHcNtA], cluster name [SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster], roles [ingest, remote_cluster_client, cluster_manager] 1> [2025-05-08T13:33:02,725][INFO ][o.o.e.ExtensionsManager ] [testIgnoreWhitespace] ExtensionsManager initialized 1> [2025-05-08T13:34:21,391][INFO ][o.o.d.DiscoveryModule ] [testIgnoreWhitespace] using discovery type [zen] and seed hosts providers [settings, file] 1> [2025-05-08T13:34:39,634][WARN ][o.o.g.DanglingIndicesState] [testIgnoreWhitespace] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually 1> [2025-05-08T13:34:52,482][INFO ][o.o.n.Node ] [testIgnoreWhitespace] initialized 1> [2025-05-08T13:34:52,825][INFO ][o.o.n.Node ] [testIgnoreWhitespace] version[2.19.1], pid[46898], build[unknown/unknown/2025-05-08T15:34:16Z], OS[Linux/6.13.5-arch1-1/riscv64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.6/21.0.6+7-LTS] 1> [2025-05-08T13:34:52,878][INFO ][o.o.n.Node ] [testIgnoreWhitespace] JVM home [/build/.gradle/caches/8.12.1/transforms/ca9a730a464aebead200f2dfe1b82576/transformed/linux-21.0.6-riscv64.tar.gz] 1> [2025-05-08T13:34:52,895][INFO ][o.o.n.Node ] [testIgnoreWhitespace] JVM arguments [-Dgradle.dist.lib=/build/.gradle/wrapper/dists/gradle-8.12.1-all/5akij8gnr65tdxupapyu22pqa/gradle-8.12.1/lib, -Dgradle.user.home=/build/.gradle, -Dgradle.worker.jar=/build/.gradle/caches/8.12.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djava.security.manager=allow, -Djna.nosys=true, -Dopensearch.scripting.update.ctx_in_params=false, -Dopensearch.search.rewrite_sort=true, -Dopensearch.transport.cname_in_publish_address=true, -Dorg.gradle.internal.worker.tmpdir=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/tmp/internalClusterTest/work, -Dtests.artifact=analysis-icu, -Dtests.gradle=true, -Dtests.jvm.argline=-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=298D86C07ECB082C, -Dtests.task=:plugins:analysis-icu:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:TieredStopAtLevel=1, -XX:ReservedCodeCacheSize=64m, -esa, -XX:HeapDumpPath=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/heapdump, -javaagent:/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/tmp/.cache/expanded/zip_ff7bf8f04f99dc306508d1c81e47a68b/jacocoagent.jar=destfile=../../jacoco/internalClusterTest.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false, -Xms512m, -Xmx3g, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp, -Duser.country, -Duser.language=en, -Duser.variant, -ea] 1> [2025-05-08T13:34:53,028][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] no modules loaded 1> [2025-05-08T13:34:53,589][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.cluster.service.applicationtemplates.TestSystemTemplatesRepositoryPlugin] 1> [2025-05-08T13:34:53,668][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.index.mapper.MockFieldFilterPlugin] 1> [2025-05-08T13:34:53,831][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.node.NodeMocksPlugin] 1> [2025-05-08T13:34:53,883][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.plugin.analysis.icu.AnalysisICUPlugin] 1> [2025-05-08T13:34:53,956][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.script.MockScriptService$TestPlugin] 1> [2025-05-08T13:34:54,027][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin] 1> [2025-05-08T13:34:54,054][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$AssertActionNamePlugin] 1> [2025-05-08T13:34:54,059][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$TestSeedPlugin] 1> [2025-05-08T13:34:54,062][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.TestGeoShapeFieldMapperPlugin] 1> [2025-05-08T13:34:54,066][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.store.MockFSIndexStore$TestPlugin] 1> [2025-05-08T13:34:54,070][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.telemetry.MockTelemetryPlugin] 1> [2025-05-08T13:34:54,084][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.transport.MockTransportService$TestPlugin] 1> [2025-05-08T13:34:54,098][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin] 1> [2025-05-08T13:34:54,669][INFO ][o.o.e.NodeEnvironment ] [testIgnoreWhitespace] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [172.1gb], net total_space [914.7gb], types [ext4] 1> [2025-05-08T13:34:54,687][INFO ][o.o.e.NodeEnvironment ] [testIgnoreWhitespace] heap size [3gb], compressed ordinary object pointers [true] 1> [2025-05-08T13:34:55,029][INFO ][o.o.n.Node ] [testIgnoreWhitespace] node name [node_s1], node ID [yZZ3wNx4St-kTZad4B03Jg], cluster name [SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster], roles [ingest, remote_cluster_client, cluster_manager] 1> [2025-05-08T13:34:55,210][INFO ][o.o.e.ExtensionsManager ] [testIgnoreWhitespace] ExtensionsManager initialized 1> [2025-05-08T13:34:56,741][INFO ][o.o.d.DiscoveryModule ] [testIgnoreWhitespace] using discovery type [zen] and seed hosts providers [settings, file] 1> [2025-05-08T13:34:57,986][WARN ][o.o.g.DanglingIndicesState] [testIgnoreWhitespace] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually 1> [2025-05-08T13:34:58,731][INFO ][o.o.n.Node ] [testIgnoreWhitespace] initialized 1> [2025-05-08T13:34:58,841][INFO ][o.o.n.Node ] [testIgnoreWhitespace] version[2.19.1], pid[46898], build[unknown/unknown/2025-05-08T15:34:16Z], OS[Linux/6.13.5-arch1-1/riscv64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.6/21.0.6+7-LTS] 1> [2025-05-08T13:34:58,856][INFO ][o.o.n.Node ] [testIgnoreWhitespace] JVM home [/build/.gradle/caches/8.12.1/transforms/ca9a730a464aebead200f2dfe1b82576/transformed/linux-21.0.6-riscv64.tar.gz] 1> [2025-05-08T13:34:58,865][INFO ][o.o.n.Node ] [testIgnoreWhitespace] JVM arguments [-Dgradle.dist.lib=/build/.gradle/wrapper/dists/gradle-8.12.1-all/5akij8gnr65tdxupapyu22pqa/gradle-8.12.1/lib, -Dgradle.user.home=/build/.gradle, -Dgradle.worker.jar=/build/.gradle/caches/8.12.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djava.security.manager=allow, -Djna.nosys=true, -Dopensearch.scripting.update.ctx_in_params=false, -Dopensearch.search.rewrite_sort=true, -Dopensearch.transport.cname_in_publish_address=true, -Dorg.gradle.internal.worker.tmpdir=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/tmp/internalClusterTest/work, -Dtests.artifact=analysis-icu, -Dtests.gradle=true, -Dtests.jvm.argline=-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=298D86C07ECB082C, -Dtests.task=:plugins:analysis-icu:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:TieredStopAtLevel=1, -XX:ReservedCodeCacheSize=64m, -esa, -XX:HeapDumpPath=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/heapdump, -javaagent:/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/tmp/.cache/expanded/zip_ff7bf8f04f99dc306508d1c81e47a68b/jacocoagent.jar=destfile=../../jacoco/internalClusterTest.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false, -Xms512m, -Xmx3g, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp, -Duser.country, -Duser.language=en, -Duser.variant, -ea] 1> [2025-05-08T13:34:58,944][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] no modules loaded 1> [2025-05-08T13:34:58,953][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.cluster.service.applicationtemplates.TestSystemTemplatesRepositoryPlugin] 1> [2025-05-08T13:34:58,957][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.index.mapper.MockFieldFilterPlugin] 1> [2025-05-08T13:34:58,960][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.node.NodeMocksPlugin] 1> [2025-05-08T13:34:58,964][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.plugin.analysis.icu.AnalysisICUPlugin] 1> [2025-05-08T13:34:58,970][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.script.MockScriptService$TestPlugin] 1> [2025-05-08T13:34:58,976][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin] 1> [2025-05-08T13:34:58,996][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$AssertActionNamePlugin] 1> [2025-05-08T13:34:59,002][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$TestSeedPlugin] 1> [2025-05-08T13:34:59,011][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.TestGeoShapeFieldMapperPlugin] 1> [2025-05-08T13:34:59,016][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.store.MockFSIndexStore$TestPlugin] 1> [2025-05-08T13:34:59,020][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.telemetry.MockTelemetryPlugin] 1> [2025-05-08T13:34:59,025][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.transport.MockTransportService$TestPlugin] 1> [2025-05-08T13:34:59,029][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin] 1> [2025-05-08T13:34:59,252][INFO ][o.o.e.NodeEnvironment ] [testIgnoreWhitespace] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [172.1gb], net total_space [914.7gb], types [ext4] 1> [2025-05-08T13:34:59,263][INFO ][o.o.e.NodeEnvironment ] [testIgnoreWhitespace] heap size [3gb], compressed ordinary object pointers [true] 1> [2025-05-08T13:34:59,506][INFO ][o.o.n.Node ] [testIgnoreWhitespace] node name [node_s2], node ID [3DHw9gajTnaQD82aEN-0fw], cluster name [SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster], roles [ingest, remote_cluster_client, cluster_manager] 1> [2025-05-08T13:34:59,586][INFO ][o.o.e.ExtensionsManager ] [testIgnoreWhitespace] ExtensionsManager initialized 1> [2025-05-08T13:35:00,540][INFO ][o.o.d.DiscoveryModule ] [testIgnoreWhitespace] using discovery type [zen] and seed hosts providers [settings, file] 1> [2025-05-08T13:35:01,328][WARN ][o.o.g.DanglingIndicesState] [testIgnoreWhitespace] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually 1> [2025-05-08T13:35:01,892][INFO ][o.o.n.Node ] [testIgnoreWhitespace] initialized 1> [2025-05-08T13:35:01,975][INFO ][o.o.n.Node ] [testIgnoreWhitespace] version[2.19.1], pid[46898], build[unknown/unknown/2025-05-08T15:34:16Z], OS[Linux/6.13.5-arch1-1/riscv64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.6/21.0.6+7-LTS] 1> [2025-05-08T13:35:02,000][INFO ][o.o.n.Node ] [testIgnoreWhitespace] JVM home [/build/.gradle/caches/8.12.1/transforms/ca9a730a464aebead200f2dfe1b82576/transformed/linux-21.0.6-riscv64.tar.gz] 1> [2025-05-08T13:35:02,006][INFO ][o.o.n.Node ] [testIgnoreWhitespace] JVM arguments [-Dgradle.dist.lib=/build/.gradle/wrapper/dists/gradle-8.12.1-all/5akij8gnr65tdxupapyu22pqa/gradle-8.12.1/lib, -Dgradle.user.home=/build/.gradle, -Dgradle.worker.jar=/build/.gradle/caches/8.12.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djava.security.manager=allow, -Djna.nosys=true, -Dopensearch.scripting.update.ctx_in_params=false, -Dopensearch.search.rewrite_sort=true, -Dopensearch.transport.cname_in_publish_address=true, -Dorg.gradle.internal.worker.tmpdir=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/tmp/internalClusterTest/work, -Dtests.artifact=analysis-icu, -Dtests.gradle=true, -Dtests.jvm.argline=-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=298D86C07ECB082C, -Dtests.task=:plugins:analysis-icu:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:TieredStopAtLevel=1, -XX:ReservedCodeCacheSize=64m, -esa, -XX:HeapDumpPath=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/heapdump, -javaagent:/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/tmp/.cache/expanded/zip_ff7bf8f04f99dc306508d1c81e47a68b/jacocoagent.jar=destfile=../../jacoco/internalClusterTest.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false, -Xms512m, -Xmx3g, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp, -Duser.country, -Duser.language=en, -Duser.variant, -ea] 1> [2025-05-08T13:35:02,072][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] no modules loaded 1> [2025-05-08T13:35:02,077][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.cluster.service.applicationtemplates.TestSystemTemplatesRepositoryPlugin] 1> [2025-05-08T13:35:02,081][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.index.mapper.MockFieldFilterPlugin] 1> [2025-05-08T13:35:02,084][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.node.NodeMocksPlugin] 1> [2025-05-08T13:35:02,088][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.plugin.analysis.icu.AnalysisICUPlugin] 1> [2025-05-08T13:35:02,095][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.script.MockScriptService$TestPlugin] 1> [2025-05-08T13:35:02,104][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin] 1> [2025-05-08T13:35:02,112][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$AssertActionNamePlugin] 1> [2025-05-08T13:35:02,119][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$TestSeedPlugin] 1> [2025-05-08T13:35:02,123][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.TestGeoShapeFieldMapperPlugin] 1> [2025-05-08T13:35:02,126][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.store.MockFSIndexStore$TestPlugin] 1> [2025-05-08T13:35:02,129][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.telemetry.MockTelemetryPlugin] 1> [2025-05-08T13:35:02,131][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.transport.MockTransportService$TestPlugin] 1> [2025-05-08T13:35:02,134][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin] 1> [2025-05-08T13:35:02,356][INFO ][o.o.e.NodeEnvironment ] [testIgnoreWhitespace] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [172.1gb], net total_space [914.7gb], types [ext4] 1> [2025-05-08T13:35:02,363][INFO ][o.o.e.NodeEnvironment ] [testIgnoreWhitespace] heap size [3gb], compressed ordinary object pointers [true] 1> [2025-05-08T13:35:02,622][INFO ][o.o.n.Node ] [testIgnoreWhitespace] node name [node_s3], node ID [RyCADUdWTC-RdM7abPqxWQ], cluster name [SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster], roles [ingest, remote_cluster_client, data] 1> [2025-05-08T13:35:03,015][DEPRECATION][o.o.d.c.s.Settings ] [testIgnoreWhitespace] [thread_pool.search.min_queue_size] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version. 1> [2025-05-08T13:35:03,224][INFO ][o.o.e.ExtensionsManager ] [testIgnoreWhitespace] ExtensionsManager initialized 1> [2025-05-08T13:35:04,140][INFO ][o.o.d.DiscoveryModule ] [testIgnoreWhitespace] using discovery type [zen] and seed hosts providers [settings, file] 1> [2025-05-08T13:35:04,745][WARN ][o.o.g.DanglingIndicesState] [testIgnoreWhitespace] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually 1> [2025-05-08T13:35:05,362][INFO ][o.o.n.Node ] [testIgnoreWhitespace] initialized 1> [2025-05-08T13:35:05,438][INFO ][o.o.n.Node ] [testIgnoreWhitespace] version[2.19.1], pid[46898], build[unknown/unknown/2025-05-08T15:34:16Z], OS[Linux/6.13.5-arch1-1/riscv64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.6/21.0.6+7-LTS] 1> [2025-05-08T13:35:05,447][INFO ][o.o.n.Node ] [testIgnoreWhitespace] JVM home [/build/.gradle/caches/8.12.1/transforms/ca9a730a464aebead200f2dfe1b82576/transformed/linux-21.0.6-riscv64.tar.gz] 1> [2025-05-08T13:35:05,453][INFO ][o.o.n.Node ] [testIgnoreWhitespace] JVM arguments [-Dgradle.dist.lib=/build/.gradle/wrapper/dists/gradle-8.12.1-all/5akij8gnr65tdxupapyu22pqa/gradle-8.12.1/lib, -Dgradle.user.home=/build/.gradle, -Dgradle.worker.jar=/build/.gradle/caches/8.12.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djava.security.manager=allow, -Djna.nosys=true, -Dopensearch.scripting.update.ctx_in_params=false, -Dopensearch.search.rewrite_sort=true, -Dopensearch.transport.cname_in_publish_address=true, -Dorg.gradle.internal.worker.tmpdir=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/tmp/internalClusterTest/work, -Dtests.artifact=analysis-icu, -Dtests.gradle=true, -Dtests.jvm.argline=-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=298D86C07ECB082C, -Dtests.task=:plugins:analysis-icu:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:TieredStopAtLevel=1, -XX:ReservedCodeCacheSize=64m, -esa, -XX:HeapDumpPath=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/heapdump, -javaagent:/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/tmp/.cache/expanded/zip_ff7bf8f04f99dc306508d1c81e47a68b/jacocoagent.jar=destfile=../../jacoco/internalClusterTest.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false, -Xms512m, -Xmx3g, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp, -Duser.country, -Duser.language=en, -Duser.variant, -ea] 1> [2025-05-08T13:35:05,518][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] no modules loaded 1> [2025-05-08T13:35:05,525][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.cluster.service.applicationtemplates.TestSystemTemplatesRepositoryPlugin] 1> [2025-05-08T13:35:05,529][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.index.mapper.MockFieldFilterPlugin] 1> [2025-05-08T13:35:05,531][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.node.NodeMocksPlugin] 1> [2025-05-08T13:35:05,534][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.plugin.analysis.icu.AnalysisICUPlugin] 1> [2025-05-08T13:35:05,540][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.script.MockScriptService$TestPlugin] 1> [2025-05-08T13:35:05,546][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin] 1> [2025-05-08T13:35:05,551][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$AssertActionNamePlugin] 1> [2025-05-08T13:35:05,554][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$TestSeedPlugin] 1> [2025-05-08T13:35:05,557][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.TestGeoShapeFieldMapperPlugin] 1> [2025-05-08T13:35:05,559][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.store.MockFSIndexStore$TestPlugin] 1> [2025-05-08T13:35:05,567][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.telemetry.MockTelemetryPlugin] 1> [2025-05-08T13:35:05,572][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.transport.MockTransportService$TestPlugin] 1> [2025-05-08T13:35:05,575][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin] 1> [2025-05-08T13:35:05,813][INFO ][o.o.e.NodeEnvironment ] [testIgnoreWhitespace] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [172.1gb], net total_space [914.7gb], types [ext4] 1> [2025-05-08T13:35:05,824][INFO ][o.o.e.NodeEnvironment ] [testIgnoreWhitespace] heap size [3gb], compressed ordinary object pointers [true] 1> [2025-05-08T13:35:06,049][INFO ][o.o.n.Node ] [testIgnoreWhitespace] node name [node_s4], node ID [MxhvN93ARFSXSbr2W4-NUQ], cluster name [SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster], roles [ingest, remote_cluster_client, data] 1> [2025-05-08T13:35:06,117][INFO ][o.o.e.ExtensionsManager ] [testIgnoreWhitespace] ExtensionsManager initialized 1> [2025-05-08T13:35:06,965][INFO ][o.o.d.DiscoveryModule ] [testIgnoreWhitespace] using discovery type [zen] and seed hosts providers [settings, file] 1> [2025-05-08T13:35:07,547][WARN ][o.o.g.DanglingIndicesState] [testIgnoreWhitespace] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually 1> [2025-05-08T13:35:08,087][INFO ][o.o.n.Node ] [testIgnoreWhitespace] initialized 1> [2025-05-08T13:35:08,167][INFO ][o.o.n.Node ] [testIgnoreWhitespace] version[2.19.1], pid[46898], build[unknown/unknown/2025-05-08T15:34:16Z], OS[Linux/6.13.5-arch1-1/riscv64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.6/21.0.6+7-LTS] 1> [2025-05-08T13:35:08,176][INFO ][o.o.n.Node ] [testIgnoreWhitespace] JVM home [/build/.gradle/caches/8.12.1/transforms/ca9a730a464aebead200f2dfe1b82576/transformed/linux-21.0.6-riscv64.tar.gz] 1> [2025-05-08T13:35:08,183][INFO ][o.o.n.Node ] [testIgnoreWhitespace] JVM arguments [-Dgradle.dist.lib=/build/.gradle/wrapper/dists/gradle-8.12.1-all/5akij8gnr65tdxupapyu22pqa/gradle-8.12.1/lib, -Dgradle.user.home=/build/.gradle, -Dgradle.worker.jar=/build/.gradle/caches/8.12.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djava.security.manager=allow, -Djna.nosys=true, -Dopensearch.scripting.update.ctx_in_params=false, -Dopensearch.search.rewrite_sort=true, -Dopensearch.transport.cname_in_publish_address=true, -Dorg.gradle.internal.worker.tmpdir=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/tmp/internalClusterTest/work, -Dtests.artifact=analysis-icu, -Dtests.gradle=true, -Dtests.jvm.argline=-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=298D86C07ECB082C, -Dtests.task=:plugins:analysis-icu:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:TieredStopAtLevel=1, -XX:ReservedCodeCacheSize=64m, -esa, -XX:HeapDumpPath=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/heapdump, -javaagent:/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/tmp/.cache/expanded/zip_ff7bf8f04f99dc306508d1c81e47a68b/jacocoagent.jar=destfile=../../jacoco/internalClusterTest.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false, -Xms512m, -Xmx3g, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp, -Duser.country, -Duser.language=en, -Duser.variant, -ea] 1> [2025-05-08T13:35:08,251][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] no modules loaded 1> [2025-05-08T13:35:08,263][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.cluster.service.applicationtemplates.TestSystemTemplatesRepositoryPlugin] 1> [2025-05-08T13:35:08,269][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.index.mapper.MockFieldFilterPlugin] 1> [2025-05-08T13:35:08,272][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.node.NodeMocksPlugin] 1> [2025-05-08T13:35:08,274][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.plugin.analysis.icu.AnalysisICUPlugin] 1> [2025-05-08T13:35:08,279][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.script.MockScriptService$TestPlugin] 1> [2025-05-08T13:35:08,283][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin] 1> [2025-05-08T13:35:08,287][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$AssertActionNamePlugin] 1> [2025-05-08T13:35:08,290][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$TestSeedPlugin] 1> [2025-05-08T13:35:08,293][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.TestGeoShapeFieldMapperPlugin] 1> [2025-05-08T13:35:08,295][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.store.MockFSIndexStore$TestPlugin] 1> [2025-05-08T13:35:08,298][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.telemetry.MockTelemetryPlugin] 1> [2025-05-08T13:35:08,300][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.test.transport.MockTransportService$TestPlugin] 1> [2025-05-08T13:35:08,302][INFO ][o.o.p.PluginsService ] [testIgnoreWhitespace] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin] 1> [2025-05-08T13:35:08,519][INFO ][o.o.e.NodeEnvironment ] [testIgnoreWhitespace] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [172.1gb], net total_space [914.7gb], types [ext4] 1> [2025-05-08T13:35:08,525][INFO ][o.o.e.NodeEnvironment ] [testIgnoreWhitespace] heap size [3gb], compressed ordinary object pointers [true] 1> [2025-05-08T13:35:08,754][INFO ][o.o.n.Node ] [testIgnoreWhitespace] node name [node_s5], node ID [7Bx5ewP5Q42abT6xeM4uBA], cluster name [SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster], roles [] 1> [2025-05-08T13:35:08,817][INFO ][o.o.e.ExtensionsManager ] [testIgnoreWhitespace] ExtensionsManager initialized 1> [2025-05-08T13:35:09,752][INFO ][o.o.d.DiscoveryModule ] [testIgnoreWhitespace] using discovery type [zen] and seed hosts providers [settings, file] 1> [2025-05-08T13:35:10,338][WARN ][o.o.g.DanglingIndicesState] [testIgnoreWhitespace] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually 1> [2025-05-08T13:35:10,934][INFO ][o.o.n.Node ] [testIgnoreWhitespace] initialized 1> [2025-05-08T13:35:11,268][INFO ][o.o.n.Node ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#1]]] starting ... 1> [2025-05-08T13:35:11,270][INFO ][o.o.n.Node ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#2]]] starting ... 1> [2025-05-08T13:35:11,273][INFO ][o.o.n.Node ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#3]]] starting ... 1> [2025-05-08T13:35:11,282][INFO ][o.o.n.Node ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#4]]] starting ... 1> [2025-05-08T13:35:11,309][INFO ][o.o.n.Node ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#6]]] starting ... 1> [2025-05-08T13:35:11,291][INFO ][o.o.n.Node ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#5]]] starting ... 1> [2025-05-08T13:35:17,348][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [813 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:17,345][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [816 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:17,343][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [816 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:18,130][INFO ][o.o.t.TransportService ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#3]]] publish_address {127.0.0.1:39703}, bound_addresses {[::1]:39611}, {127.0.0.1:39703} 1> [2025-05-08T13:35:18,126][INFO ][o.o.t.TransportService ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#4]]] publish_address {127.0.0.1:45155}, bound_addresses {[::1]:44885}, {127.0.0.1:45155} 1> [2025-05-08T13:35:18,127][INFO ][o.o.t.TransportService ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#1]]] publish_address {127.0.0.1:40859}, bound_addresses {[::1]:44915}, {127.0.0.1:40859} 1> [2025-05-08T13:35:18,126][INFO ][o.o.t.TransportService ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#5]]] publish_address {127.0.0.1:45869}, bound_addresses {[::1]:33467}, {127.0.0.1:45869} 1> [2025-05-08T13:35:18,126][INFO ][o.o.t.TransportService ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#2]]] publish_address {127.0.0.1:43491}, bound_addresses {[::1]:44433}, {127.0.0.1:43491} 1> [2025-05-08T13:35:18,126][INFO ][o.o.t.TransportService ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#6]]] publish_address {127.0.0.1:41581}, bound_addresses {[::1]:45611}, {127.0.0.1:41581} 1> [2025-05-08T13:35:17,348][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [800 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:17,345][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [800 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:17,343][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [800 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:18,504][INFO ][o.o.t.TransportService ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#1]]] Remote clusters initialized successfully. 1> [2025-05-08T13:35:18,503][INFO ][o.o.t.TransportService ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#3]]] Remote clusters initialized successfully. 1> [2025-05-08T13:35:18,504][INFO ][o.o.t.TransportService ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#4]]] Remote clusters initialized successfully. 1> [2025-05-08T13:35:18,504][INFO ][o.o.t.TransportService ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#2]]] Remote clusters initialized successfully. 1> [2025-05-08T13:35:18,501][INFO ][o.o.t.TransportService ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#5]]] Remote clusters initialized successfully. 1> [2025-05-08T13:35:28,629][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][15] overhead, spent [597ms] collecting in the last [1.3s] 1> [2025-05-08T13:35:28,634][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][15] overhead, spent [597ms] collecting in the last [1.1s] 1> [2025-05-08T13:35:28,634][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][15] overhead, spent [597ms] collecting in the last [1.2s] 1> [2025-05-08T13:35:28,617][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][15] overhead, spent [597ms] collecting in the last [1.3s] 1> [2025-05-08T13:35:28,617][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][15] overhead, spent [597ms] collecting in the last [1.2s] 1> [2025-05-08T13:35:28,617][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][15] overhead, spent [597ms] collecting in the last [1.3s] 1> [2025-05-08T13:35:36,241][WARN ][o.o.g.PersistedClusterStateService] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#2]]] writing cluster state took [13903ms] which is above the warn threshold of [10s]; wrote full state with [0] indices 1> [2025-05-08T13:35:36,242][WARN ][o.o.g.PersistedClusterStateService] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#5]]] writing cluster state took [13907ms] which is above the warn threshold of [10s]; wrote full state with [0] indices 1> [2025-05-08T13:35:36,242][WARN ][o.o.g.PersistedClusterStateService] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#1]]] writing cluster state took [13907ms] which is above the warn threshold of [10s]; wrote full state with [0] indices 1> [2025-05-08T13:35:36,242][WARN ][o.o.g.PersistedClusterStateService] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#3]]] writing cluster state took [13909ms] which is above the warn threshold of [10s]; wrote full state with [0] indices 1> [2025-05-08T13:35:36,241][WARN ][o.o.g.PersistedClusterStateService] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#4]]] writing cluster state took [13896ms] which is above the warn threshold of [10s]; wrote full state with [0] indices 1> [2025-05-08T13:35:36,241][WARN ][o.o.g.PersistedClusterStateService] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#6]]] writing cluster state took [13906ms] which is above the warn threshold of [10s]; wrote full state with [0] indices 1> [2025-05-08T13:35:39,726][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Potentially blocked execution on network thread [opensearch[node_s0][transport_worker][T#1]] [RUNNABLE] [400 milliseconds]: 1> org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:394) 1> org.opensearch.nio.SocketChannelContext.register(SocketChannelContext.java:115) 1> org.opensearch.nio.EventHandler.handleRegistration(EventHandler.java:78) 1> org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:91) 1> org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) 1> org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) 1> org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:35:39,721][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Potentially blocked execution on network thread [opensearch[node_s5][transport_worker][T#1]] [RUNNABLE] [401 milliseconds]: 1> java.base/java.util.Arrays.spliterator(Arrays.java:5396) 1> java.base/java.util.Arrays.stream(Arrays.java:5547) 1> java.base/java.util.Arrays.stream(Arrays.java:5528) 1> org.opensearch.secure_sm.SecureSM$1.hasTrustedCallerChain(SecureSM.java:144) 1> org.opensearch.secure_sm.SecureSM$1.checkConnect(SecureSM.java:130) 1> java.base/sun.nio.ch.SocketChannelImpl.checkRemote(SocketChannelImpl.java:846) 1> java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:865) 1> org.opensearch.nio.SocketChannelContext.lambda$connect$0(SocketChannelContext.java:394) 1> java.base/java.security.AccessController.doPrivileged(AccessController.java:571) 1> org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:394) 1> org.opensearch.nio.SocketChannelContext.register(SocketChannelContext.java:115) 1> org.opensearch.nio.EventHandler.handleRegistration(EventHandler.java:78) 1> org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:91) 1> org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) 1> org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) 1> org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:35:39,739][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Potentially blocked execution on network thread [opensearch[node_s3][transport_worker][T#2]] [BLOCKED] [400 milliseconds]: 1> java.base/java.net.SocketPermission.impliesIgnoreMask(SocketPermission.java:979) 1> java.base/java.net.SocketPermissionCollection.implies(SocketPermission.java:1441) 1> java.base/java.security.Permissions.implies(Permissions.java:179) 1> org.opensearch.bootstrap.OpenSearchPolicy.implies(OpenSearchPolicy.java:134) 1> org.opensearch.bootstrap.BootstrapForTesting$1.implies(BootstrapForTesting.java:174) 1> java.base/java.security.ProtectionDomain.implies(ProtectionDomain.java:330) 1> java.base/java.security.ProtectionDomain.impliesWithAltFilePerm(ProtectionDomain.java:362) 1> java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:466) 1> java.base/java.security.AccessController.checkPermission(AccessController.java:1071) 1> java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411) 1> java.base/java.lang.SecurityManager.checkConnect(SecurityManager.java:905) 1> org.opensearch.secure_sm.SecureSM$1.checkConnect(SecureSM.java:131) 1> java.base/sun.nio.ch.SocketChannelImpl.checkRemote(SocketChannelImpl.java:846) 1> java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:865) 1> org.opensearch.nio.SocketChannelContext.lambda$connect$0(SocketChannelContext.java:394) 1> java.base/java.security.AccessController.doPrivileged(AccessController.java:571) 1> org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:394) 1> org.opensearch.nio.SocketChannelContext.register(SocketChannelContext.java:115) 1> org.opensearch.nio.EventHandler.handleRegistration(EventHandler.java:78) 1> org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:91) 1> org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) 1> org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) 1> org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:35:39,926][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:39,904][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:39,911][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [601 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:39,904][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [602 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:39,899][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:39,894][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:39,905][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:39,892][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:39,879][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [602 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:39,879][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [401 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:39,882][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:39,884][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:47,083][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:47,080][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:48,296][INFO ][o.o.c.c.Coordinator ] [node_s1] setting initial configuration to VotingConfiguration{yZZ3wNx4St-kTZad4B03Jg,48nm_5e6SW27203haHcNtA,3DHw9gajTnaQD82aEN-0fw} 1> [2025-05-08T13:35:48,753][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node_s3] cluster-manager not discovered yet: have discovered [{node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true}, {node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}, {node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}, {node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:40859, 127.0.0.1:43491, 127.0.0.1:39703] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0 1> [2025-05-08T13:35:48,761][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node_s4] cluster-manager not discovered yet: have discovered [{node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true}, {node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}, {node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}, {node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:40859, 127.0.0.1:43491, 127.0.0.1:39703] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0 1> [2025-05-08T13:35:48,968][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node_s2] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial_cluster_manager_nodes] is empty on this node: have discovered [{node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}, {node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}, {node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:40859, 127.0.0.1:43491] from hosts providers and [{node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 1> [2025-05-08T13:35:48,962][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node_s0] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial_cluster_manager_nodes] is empty on this node: have discovered [{node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}, {node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}, {node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:43491, 127.0.0.1:39703] from hosts providers and [{node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 1> [2025-05-08T13:35:55,415][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [1817 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:55,417][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [800 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:55,428][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:35:58,971][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node_s3] cluster-manager not discovered yet: have discovered [{node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true}, {node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}, {node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}, {node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:40859, 127.0.0.1:43491, 127.0.0.1:39703] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0 1> [2025-05-08T13:35:58,972][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node_s4] cluster-manager not discovered yet: have discovered [{node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true}, {node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}, {node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}, {node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:40859, 127.0.0.1:43491, 127.0.0.1:39703] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0 1> [2025-05-08T13:35:59,324][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node_s2] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial_cluster_manager_nodes] is empty on this node: have discovered [{node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}, {node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}, {node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:40859, 127.0.0.1:43491] from hosts providers and [{node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 1> [2025-05-08T13:35:59,326][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node_s0] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial_cluster_manager_nodes] is empty on this node: have discovered [{node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}, {node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}, {node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:43491, 127.0.0.1:39703] from hosts providers and [{node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 1> [2025-05-08T13:36:00,106][WARN ][o.o.g.PersistedClusterStateService] [node_s1] writing cluster state took [11647ms] which is above the warn threshold of [10s]; wrote global metadata [true] and metadata for [0] indices and skipped [0] unchanged indices 1> [2025-05-08T13:36:00,249][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node_s1] cluster-manager not discovered or elected yet, an election requires at least 2 nodes with ids from [yZZ3wNx4St-kTZad4B03Jg, 48nm_5e6SW27203haHcNtA, 3DHw9gajTnaQD82aEN-0fw], have discovered [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}, {node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}, {node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}] which is a quorum; discovery will continue using [127.0.0.1:40859, 127.0.0.1:39703] from hosts providers and [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 1> [2025-05-08T13:36:02,949][INFO ][o.o.c.s.MasterService ] [node_s1] Tasks batched with key: org.opensearch.cluster.coordination.JoinHelper, count:4 and sample tasks: elected-as-cluster-manager ([2] nodes joined)[{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true} elect leader, {node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: cluster-manager node changed {previous [], current [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}]}, added {{node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}} 1> [2025-05-08T13:36:07,540][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:07,261][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:07,262][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:07,301][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:07,280][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:09,120][WARN ][o.o.n.Node ] [node_s4] timed out while waiting for initial discovery state - timeout: 30s 1> [2025-05-08T13:36:09,123][WARN ][o.o.n.Node ] [node_s1] timed out while waiting for initial discovery state - timeout: 30s 1> [2025-05-08T13:36:09,128][WARN ][o.o.n.Node ] [node_s2] timed out while waiting for initial discovery state - timeout: 30s 1> [2025-05-08T13:36:09,169][WARN ][o.o.n.Node ] [node_s0] timed out while waiting for initial discovery state - timeout: 30s 1> [2025-05-08T13:36:09,186][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node_s3] cluster-manager not discovered yet: have discovered [{node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true}, {node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}, {node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}, {node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:40859, 127.0.0.1:43491, 127.0.0.1:39703] from hosts providers and [] from last-known cluster state; node term 1, last-accepted version 0 in term 0 1> [2025-05-08T13:36:09,120][WARN ][o.o.n.Node ] [node_s5] timed out while waiting for initial discovery state - timeout: 30s 1> [2025-05-08T13:36:09,122][WARN ][o.o.n.Node ] [node_s3] timed out while waiting for initial discovery state - timeout: 30s 1> [2025-05-08T13:36:09,248][INFO ][o.o.n.Node ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#5]]] started 1> [2025-05-08T13:36:09,248][INFO ][o.o.n.Node ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#2]]] started 1> [2025-05-08T13:36:09,181][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node_s4] cluster-manager not discovered yet: have discovered [{node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true}, {node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}, {node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}, {node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:40859, 127.0.0.1:43491, 127.0.0.1:39703] from hosts providers and [] from last-known cluster state; node term 1, last-accepted version 0 in term 0 1> [2025-05-08T13:36:09,276][INFO ][o.o.n.Node ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#4]]] started 1> [2025-05-08T13:36:09,266][INFO ][o.o.n.Node ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#6]]] started 1> [2025-05-08T13:36:09,249][INFO ][o.o.n.Node ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#3]]] started 1> [2025-05-08T13:36:09,251][INFO ][o.o.n.Node ] [[test_SUITE-TEST_WORKER_VM=[16]-CLUSTER_SEED=[2960163051026126826]-HASH=[1258DAB6F35A2C]-cluster[T#1]]] started 1> [2025-05-08T13:36:09,859][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node_s0] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial_cluster_manager_nodes] is empty on this node: have discovered [{node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}, {node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}, {node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:43491, 127.0.0.1:39703] from hosts providers and [{node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 1, last-accepted version 0 in term 0 1> [2025-05-08T13:36:10,347][INFO ][o.o.c.c.CoordinationState] [node_s2] cluster UUID set to [ZoJObGYLS0yUg3PLncoSFw] 1> [2025-05-08T13:36:10,349][INFO ][o.o.c.c.CoordinationState] [node_s1] cluster UUID set to [ZoJObGYLS0yUg3PLncoSFw] 1> [2025-05-08T13:36:12,846][INFO ][o.o.c.s.ClusterApplierService] [node_s2] cluster-manager node changed {previous [], current [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}]}, added {{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}}, term: 1, version: 1, reason: ApplyCommitRequest{term=1, version=1, sourceNode={node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}} 1> [2025-05-08T13:36:13,374][INFO ][o.o.c.s.ClusterApplierService] [node_s1] cluster-manager node changed {previous [], current [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}]}, added {{node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}}, term: 1, version: 1, reason: Publication{term=1, version=1} 1> [2025-05-08T13:36:13,789][INFO ][o.o.d.PeerFinder ] [node_s1] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}] 1> [2025-05-08T13:36:13,809][INFO ][o.o.d.PeerFinder ] [node_s2] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}] 1> [2025-05-08T13:36:13,895][INFO ][o.o.c.s.MasterService ] [node_s1] Tasks batched with key: org.opensearch.cluster.coordination.JoinHelper, count:6 and sample tasks: node-join[{node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true} join existing leader, {node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true} join existing leader, {node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true} join existing leader, {node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true} join existing leader, {node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true} join existing leader, {node_s5}{7Bx5ewP5Q42abT6xeM4uBA}{OVcy3CfWRJeYN8aHI5nZRg}{127.0.0.1}{127.0.0.1:41581}{shard_indexing_pressure_enabled=true} join existing leader], term: 1, version: 2, delta: added {{node_s5}{7Bx5ewP5Q42abT6xeM4uBA}{OVcy3CfWRJeYN8aHI5nZRg}{127.0.0.1}{127.0.0.1:41581}{shard_indexing_pressure_enabled=true},{node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true},{node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true},{node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true}} 1> [2025-05-08T13:36:14,426][INFO ][o.o.c.s.ClusterApplierService] [node_s2] added {{node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true},{node_s5}{7Bx5ewP5Q42abT6xeM4uBA}{OVcy3CfWRJeYN8aHI5nZRg}{127.0.0.1}{127.0.0.1:41581}{shard_indexing_pressure_enabled=true},{node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true},{node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}} 1> [2025-05-08T13:36:14,430][INFO ][o.o.c.s.ClusterApplierService] [node_s3] cluster-manager node changed {previous [], current [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}]}, added {{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true},{node_s5}{7Bx5ewP5Q42abT6xeM4uBA}{OVcy3CfWRJeYN8aHI5nZRg}{127.0.0.1}{127.0.0.1:41581}{shard_indexing_pressure_enabled=true},{node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true},{node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true},{node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}} 1> [2025-05-08T13:36:14,430][INFO ][o.o.c.s.ClusterApplierService] [node_s4] cluster-manager node changed {previous [], current [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}]}, added {{node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true},{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true},{node_s5}{7Bx5ewP5Q42abT6xeM4uBA}{OVcy3CfWRJeYN8aHI5nZRg}{127.0.0.1}{127.0.0.1:41581}{shard_indexing_pressure_enabled=true},{node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true},{node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}} 1> [2025-05-08T13:36:14,431][INFO ][o.o.c.s.ClusterApplierService] [node_s5] cluster-manager node changed {previous [], current [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}]}, added {{node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true},{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true},{node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true},{node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true},{node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}} 1> [2025-05-08T13:36:15,874][INFO ][o.o.c.s.ClusterApplierService] [node_s0] cluster-manager node changed {previous [], current [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}]}, added {{node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true},{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true},{node_s5}{7Bx5ewP5Q42abT6xeM4uBA}{OVcy3CfWRJeYN8aHI5nZRg}{127.0.0.1}{127.0.0.1:41581}{shard_indexing_pressure_enabled=true},{node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true},{node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}} 1> [2025-05-08T13:36:17,259][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][60] overhead, spent [343ms] collecting in the last [1.2s] 1> [2025-05-08T13:36:17,443][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][60] overhead, spent [343ms] collecting in the last [1.2s] 1> [2025-05-08T13:36:17,439][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][60] overhead, spent [343ms] collecting in the last [1.1s] 1> [2025-05-08T13:36:17,247][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][60] overhead, spent [343ms] collecting in the last [1.2s] 1> [2025-05-08T13:36:16,538][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [612 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.acceptChannel(TestEventHandler.java:70) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:246) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:16,501][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [401 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:145) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:16,368][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [401 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleActive(TestEventHandler.java:118) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.channelActive(NioSelector.java:483) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:467) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:17,694][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [716 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:17,688][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [742 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRegistration(TestEventHandler.java:94) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:466) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:17,656][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [928 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:19,820][INFO ][o.o.c.s.ClusterApplierService] [node_s1] added {{node_s5}{7Bx5ewP5Q42abT6xeM4uBA}{OVcy3CfWRJeYN8aHI5nZRg}{127.0.0.1}{127.0.0.1:41581}{shard_indexing_pressure_enabled=true},{node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true},{node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true},{node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true}}, term: 1, version: 2, reason: Publication{term=1, version=2} 1> [2025-05-08T13:36:20,054][INFO ][o.o.d.PeerFinder ] [node_s5] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_s5}{7Bx5ewP5Q42abT6xeM4uBA}{OVcy3CfWRJeYN8aHI5nZRg}{127.0.0.1}{127.0.0.1:41581}{shard_indexing_pressure_enabled=true}] 1> [2025-05-08T13:36:20,084][INFO ][o.o.d.PeerFinder ] [node_s0] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}] 1> [2025-05-08T13:36:20,068][INFO ][o.o.d.PeerFinder ] [node_s2] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}] 1> [2025-05-08T13:36:20,070][INFO ][o.o.d.PeerFinder ] [node_s3] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true}] 1> [2025-05-08T13:36:20,067][INFO ][o.o.d.PeerFinder ] [node_s4] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true}] 1> [2025-05-08T13:36:20,316][INFO ][o.o.d.PeerFinder ] [node_s0] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}] 1> [2025-05-08T13:36:20,356][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:21,149][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:36:22,021][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:36:22,177][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:22,287][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:22,760][INFO ][o.o.g.GatewayService ] [node_s1] recovered [0] indices into cluster_state 1> [2025-05-08T13:36:28,364][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:36:29,949][DEPRECATION][o.o.d.c.s.Settings ] [node_s1] [index.force_memory_term_dictionary] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version. 1> [2025-05-08T13:36:31,124][INFO ][o.o.p.PluginsService ] [node_s1] PluginService:onIndexModule index:[CQyIwDtlRXKpKJGAnmJB5Q/w-udCortSPeqcXcrEHVawg] 1> [2025-05-08T13:36:37,431][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [407 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:37,431][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [616 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:37,501][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [616 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:37,428][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [418 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:51,928][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][90] overhead, spent [566ms] collecting in the last [1.1s] 1> [2025-05-08T13:36:51,940][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][90] overhead, spent [566ms] collecting in the last [2s] 1> [2025-05-08T13:36:51,935][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][89] overhead, spent [566ms] collecting in the last [1.2s] 1> [2025-05-08T13:36:51,937][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][90] overhead, spent [566ms] collecting in the last [1.1s] 1> [2025-05-08T13:36:51,945][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][90] overhead, spent [566ms] collecting in the last [1.2s] 1> [2025-05-08T13:36:51,747][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:51,798][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [605 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:52,414][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:36:54,577][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][91] overhead, spent [302ms] collecting in the last [1s] 1> [2025-05-08T13:36:54,577][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][92] overhead, spent [302ms] collecting in the last [1s] 1> [2025-05-08T13:36:54,624][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][92] overhead, spent [302ms] collecting in the last [1.1s] 1> [2025-05-08T13:36:54,621][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][92] overhead, spent [302ms] collecting in the last [1s] 1> [2025-05-08T13:36:54,624][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][92] overhead, spent [302ms] collecting in the last [1s] 1> [2025-05-08T13:37:02,695][INFO ][o.o.c.m.MetadataIndexTemplateService] [node_s1] adding template [random_index_template] for index patterns [*] 1> [2025-05-08T13:37:02,823][WARN ][o.o.c.s.MasterService ] [node_s1] took [32.6s], which is over [10s], to compute cluster state update for [Tasks batched with key: org.opensearch.cluster.metadata.MetadataIndexTemplateService, count:1 and sample tasks: create-index-template [random_index_template], cause [api]] 1> [2025-05-08T13:37:09,252][INFO ][o.o.t.OpenSearchTestClusterRule] [testIgnoreWhitespace] [ICUCollationKeywordFieldMapperIT#testIgnoreWhitespace]: all set up test 1> [2025-05-08T13:37:09,455][INFO ][o.o.i.m.ICUCollationKeywordFieldMapperIT] [testIgnoreWhitespace] before test 1> [2025-05-08T13:37:11,208][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:37:12,069][INFO ][o.o.p.PluginsService ] [node_s1] PluginService:onIndexModule index:[foo/M4aeEAbRQ9KFiC8xFN55PQ] 1> [2025-05-08T13:37:15,053][INFO ][o.o.c.m.MetadataCreateIndexService] [node_s1] [foo] creating index, cause [api], templates [random_index_template], shards [8]/[0] 1> [2025-05-08T13:37:15,640][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:37:23,637][INFO ][o.o.p.PluginsService ] [node_s3] PluginService:onIndexModule index:[foo/M4aeEAbRQ9KFiC8xFN55PQ] 1> [2025-05-08T13:37:23,637][INFO ][o.o.p.PluginsService ] [node_s4] PluginService:onIndexModule index:[foo/M4aeEAbRQ9KFiC8xFN55PQ] 1> [2025-05-08T13:37:31,836][INFO ][o.o.c.c.C.CoordinatorPublication] [node_s1] after [11.6s] publication of cluster state version [6] is still waiting for {node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT], {node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT] 1> [2025-05-08T13:37:44,096][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [418 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:37:44,099][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [416 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:37:44,112][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [425 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:37:46,800][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Potentially blocked execution on network thread [opensearch[node_s4][transport_worker][T#1]] [RUNNABLE] [1284 milliseconds]: 1> java.base/java.util.HashMap.hash(HashMap.java:338) 1> java.base/java.util.HashMap.put(HashMap.java:618) 1> org.opensearch.telemetry.tracing.attributes.Attributes.addAttribute(Attributes.java:54) 1> org.opensearch.telemetry.tracing.SpanBuilder.buildSpanAttributes(SpanBuilder.java:181) 1> org.opensearch.telemetry.tracing.SpanBuilder.from(SpanBuilder.java:171) 1> org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:211) 1> org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) 1> org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 1> org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 1> org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:37:47,347][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:37:52,575][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:37:55,047][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:37:55,096][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [800 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:37:58,984][INFO ][o.o.c.r.a.AllocationService] [node_s1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[foo][1], [foo][0], [foo][5], [foo][3], [foo][2], [foo][4], [foo][6]]]). 1> [2025-05-08T13:38:06,019][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:38:06,625][INFO ][o.o.i.m.ICUCollationKeywordFieldMapperIT] [testIgnoreWhitespace] Index [5] docs async: [true] bulk: [false] 1> [2025-05-08T13:38:07,969][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [800 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:38:07,969][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [800 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:38:07,931][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:38:30,826][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:38:43,911][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [800 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:38:43,289][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [498 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:38:44,811][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:38:44,503][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Potentially blocked execution on network thread [opensearch[node_s2][transport_worker][T#2]] [RUNNABLE] [1785 milliseconds]: 1> org.opensearch.telemetry.tracing.channels.TraceableTcpTransportChannel.create(TraceableTcpTransportChannel.java:70) 1> org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:260) 1> org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) 1> org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 1> org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 1> org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:38:43,314][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [917 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:38:56,707][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [1076 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:38:56,707][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [876 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:39:15,609][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [3355 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:39:16,858][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:39:16,598][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:39:31,242][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:39:31,251][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:39:38,748][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Potentially blocked execution on network thread [opensearch[node_s4][transport_worker][T#2]] [BLOCKED] [400 milliseconds]: 1> java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:651) 1> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639) 1> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) 1> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526) 1> org.opensearch.search.builder.SearchSourceBuilder.(SearchSourceBuilder.java:253) 1> org.opensearch.core.common.io.stream.StreamInput.readOptionalWriteable(StreamInput.java:1013) 1> org.opensearch.search.internal.ShardSearchRequest.(ShardSearchRequest.java:245) 1> org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:87) 1> org.opensearch.transport.NativeMessageHandler.newRequest(NativeMessageHandler.java:316) 1> org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:271) 1> org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) 1> org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 1> org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 1> org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:39:38,745][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Potentially blocked execution on network thread [opensearch[node_s3][transport_worker][T#2]] [BLOCKED] [400 milliseconds]: 1> java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:651) 1> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639) 1> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) 1> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526) 1> org.opensearch.search.builder.SearchSourceBuilder.(SearchSourceBuilder.java:253) 1> org.opensearch.core.common.io.stream.StreamInput.readOptionalWriteable(StreamInput.java:1013) 1> org.opensearch.search.internal.ShardSearchRequest.(ShardSearchRequest.java:245) 1> org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:87) 1> org.opensearch.transport.NativeMessageHandler.newRequest(NativeMessageHandler.java:316) 1> org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:271) 1> org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) 1> org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 1> org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 1> org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:39:38,976][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Potentially blocked execution on network thread [opensearch[node_s3][transport_worker][T#1]] [BLOCKED] [600 milliseconds]: 1> java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:651) 1> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639) 1> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) 1> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526) 1> org.opensearch.search.builder.SearchSourceBuilder.(SearchSourceBuilder.java:274) 1> org.opensearch.core.common.io.stream.StreamInput.readOptionalWriteable(StreamInput.java:1013) 1> org.opensearch.search.internal.ShardSearchRequest.(ShardSearchRequest.java:245) 1> org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:87) 1> org.opensearch.transport.NativeMessageHandler.newRequest(NativeMessageHandler.java:316) 1> org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:271) 1> org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) 1> org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 1> org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 1> org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:39:44,479][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Potentially blocked execution on network thread [opensearch[node_s5][transport_worker][T#1]] [BLOCKED] [1422 milliseconds]: 1> org.opensearch.action.search.SearchQueryThenFetchAsyncAction.onShardResult(SearchQueryThenFetchAsyncAction.java:156) 1> org.opensearch.action.search.AbstractSearchAsyncAction$1.innerOnResponse(AbstractSearchAsyncAction.java:303) 1> org.opensearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:59) 1> org.opensearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:44) 1> org.opensearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:99) 1> org.opensearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:52) 1> org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:70) 1> org.opensearch.action.search.SearchTransportService$ConnectionCountingHandler.handleResponse(SearchTransportService.java:757) 1> org.opensearch.telemetry.tracing.handler.TraceableTransportResponseHandler.handleResponse(TraceableTransportResponseHandler.java:72) 1> org.opensearch.transport.TransportService$9.handleResponse(TransportService.java:1735) 1> org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1517) 1> org.opensearch.transport.NativeMessageHandler.doHandleResponse(NativeMessageHandler.java:427) 1> org.opensearch.transport.NativeMessageHandler.handleResponse(NativeMessageHandler.java:419) 1> org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:174) 1> org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 1> org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 1> org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:39:45,109][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Potentially blocked execution on network thread [opensearch[node_s5][transport_worker][T#2]] [RUNNABLE] [2623 milliseconds]: 1> com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:759) 1> org.opensearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:66) 1> org.opensearch.core.xcontent.ObjectParser.parse(ObjectParser.java:319) 1> org.opensearch.core.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:188) 1> org.opensearch.core.xcontent.ConstructingObjectParser.apply(ConstructingObjectParser.java:180) 1> org.opensearch.tasks.TaskResourceTrackingService.getTaskResourceUsageFromThreadContext(TaskResourceTrackingService.java:355) 1> org.opensearch.action.search.AbstractSearchAsyncAction.setPhaseResourceUsages(AbstractSearchAsyncAction.java:632) 1> org.opensearch.action.search.AbstractSearchAsyncAction.onShardResult(AbstractSearchAsyncAction.java:627) 1> org.opensearch.action.search.SearchQueryThenFetchAsyncAction.onShardResult(SearchQueryThenFetchAsyncAction.java:164) 1> org.opensearch.action.search.AbstractSearchAsyncAction$1.innerOnResponse(AbstractSearchAsyncAction.java:303) 1> org.opensearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:59) 1> org.opensearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:44) 1> org.opensearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:99) 1> org.opensearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:52) 1> org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:70) 1> org.opensearch.action.search.SearchTransportService$ConnectionCountingHandler.handleResponse(SearchTransportService.java:757) 1> org.opensearch.telemetry.tracing.handler.TraceableTransportResponseHandler.handleResponse(TraceableTransportResponseHandler.java:72) 1> org.opensearch.transport.TransportService$9.handleResponse(TransportService.java:1735) 1> org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1517) 1> org.opensearch.transport.NativeMessageHandler.doHandleResponse(NativeMessageHandler.java:427) 1> org.opensearch.transport.NativeMessageHandler.handleResponse(NativeMessageHandler.java:419) 1> org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:174) 1> org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 1> org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 1> org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:39:45,953][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [3424 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:39:46,052][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [3624 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:39:46,612][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:39:46,635][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:39:47,081][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:39:48,994][INFO ][o.o.i.m.ICUCollationKeywordFieldMapperIT] [testIgnoreWhitespace] after test 1> [2025-05-08T13:39:49,549][INFO ][o.o.t.OpenSearchTestClusterRule] [testIgnoreWhitespace] [ICUCollationKeywordFieldMapperIT#testIgnoreWhitespace]: cleaning up after test 1> [2025-05-08T13:39:52,167][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][250] overhead, spent [266ms] collecting in the last [1s] 1> [2025-05-08T13:39:52,165][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][249] overhead, spent [266ms] collecting in the last [1s] 1> [2025-05-08T13:39:52,164][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][249] overhead, spent [266ms] collecting in the last [1s] 1> [2025-05-08T13:39:52,164][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][241] overhead, spent [266ms] collecting in the last [1s] 1> [2025-05-08T13:39:52,166][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][249] overhead, spent [266ms] collecting in the last [1s] 1> [2025-05-08T13:39:58,129][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:39:59,362][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_s1] [foo/M4aeEAbRQ9KFiC8xFN55PQ] deleting index 1> [2025-05-08T13:39:59,374][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:40:03,118][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][1] start check index 1> [2025-05-08T13:40:03,118][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][0] start check index 1> [2025-05-08T13:40:08,236][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:40:08,480][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][1] end check index 1> [2025-05-08T13:40:08,480][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][0] end check index 1> [2025-05-08T13:40:09,481][INFO ][o.o.c.c.C.CoordinatorPublication] [node_s1] after [10s] publication of cluster state version [10] is still waiting for {node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT], {node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT] 1> [2025-05-08T13:40:09,999][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][3] start check index 1> [2025-05-08T13:40:10,165][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][2] start check index 1> [2025-05-08T13:40:10,245][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][3] end check index 1> [2025-05-08T13:40:10,332][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][2] end check index 1> [2025-05-08T13:40:15,172][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][267] overhead, spent [621ms] collecting in the last [2.3s] 1> [2025-05-08T13:40:15,187][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][267] overhead, spent [621ms] collecting in the last [2.4s] 1> [2025-05-08T13:40:15,172][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][267] overhead, spent [621ms] collecting in the last [2.4s] 1> [2025-05-08T13:40:15,183][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][267] overhead, spent [621ms] collecting in the last [2.4s] 1> [2025-05-08T13:40:15,112][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][268] overhead, spent [636ms] collecting in the last [2s] 1> [2025-05-08T13:40:16,580][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:40:16,744][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [800 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:40:15,079][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [430 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:40:15,926][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:40:17,514][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][5] start check index 1> [2025-05-08T13:40:17,515][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][4] start check index 1> [2025-05-08T13:40:18,480][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][4] end check index 1> [2025-05-08T13:40:18,479][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][5] end check index 1> [2025-05-08T13:40:19,744][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][6] start check index 1> [2025-05-08T13:40:20,062][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][7] start check index 1> [2025-05-08T13:40:20,088][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][6] end check index 1> [2025-05-08T13:40:20,233][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][7] end check index 1> [2025-05-08T13:40:22,565][INFO ][o.o.c.m.MetadataIndexTemplateService] [node_s1] removing template [random_index_template] 1> [2025-05-08T13:40:29,049][INFO ][o.o.t.OpenSearchTestClusterRule] [testIgnoreWhitespace] [ICUCollationKeywordFieldMapperIT#testIgnoreWhitespace]: cleaned up after test 1> [2025-05-08T13:40:30,599][INFO ][o.o.t.OpenSearchTestClusterRule] [testUpperCaseFirst] [ICUCollationKeywordFieldMapperIT#testUpperCaseFirst]: setting up test 1> [2025-05-08T13:40:30,874][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:40:31,716][DEPRECATION][o.o.d.c.s.Settings ] [node_s1] [index.force_memory_term_dictionary] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version. 1> [2025-05-08T13:40:32,006][INFO ][o.o.p.PluginsService ] [node_s1] PluginService:onIndexModule index:[6ewR7nu7SGONKRmE7Mii5w/zky5P_UmRhWM038vyNQ8xw] 1> [2025-05-08T13:40:32,311][INFO ][o.o.c.m.MetadataIndexTemplateService] [node_s1] adding template [random_index_template] for index patterns [*] 1> [2025-05-08T13:40:34,433][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][283] overhead, spent [282ms] collecting in the last [1s] 1> [2025-05-08T13:40:34,418][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][283] overhead, spent [282ms] collecting in the last [1s] 1> [2025-05-08T13:40:34,944][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:40:37,636][INFO ][o.o.t.OpenSearchTestClusterRule] [testUpperCaseFirst] [ICUCollationKeywordFieldMapperIT#testUpperCaseFirst]: all set up test 1> [2025-05-08T13:40:37,672][INFO ][o.o.i.m.ICUCollationKeywordFieldMapperIT] [testUpperCaseFirst] before test 1> [2025-05-08T13:40:38,703][INFO ][o.o.p.PluginsService ] [node_s1] PluginService:onIndexModule index:[foo/8FjOAzM7Sk6YAWNm1jmvIw] 1> [2025-05-08T13:40:39,093][INFO ][o.o.c.m.MetadataCreateIndexService] [node_s1] [foo] creating index, cause [api], templates [random_index_template], shards [9]/[0] 1> [2025-05-08T13:40:39,141][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:40:41,827][INFO ][o.o.p.PluginsService ] [node_s3] PluginService:onIndexModule index:[foo/8FjOAzM7Sk6YAWNm1jmvIw] 1> [2025-05-08T13:40:41,828][INFO ][o.o.p.PluginsService ] [node_s4] PluginService:onIndexModule index:[foo/8FjOAzM7Sk6YAWNm1jmvIw] 1> [2025-05-08T13:40:44,009][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][282] overhead, spent [322ms] collecting in the last [1.2s] 1> [2025-05-08T13:40:44,098][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][291] overhead, spent [322ms] collecting in the last [1s] 1> [2025-05-08T13:40:44,101][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][291] overhead, spent [322ms] collecting in the last [1s] 1> [2025-05-08T13:40:44,176][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][292] overhead, spent [322ms] collecting in the last [1s] 1> [2025-05-08T13:40:44,178][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][292] overhead, spent [322ms] collecting in the last [1s] 1> [2025-05-08T13:40:44,177][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][293] overhead, spent [322ms] collecting in the last [1s] 1> [2025-05-08T13:40:54,011][WARN ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][299] overhead, spent [856ms] collecting in the last [1.6s] 1> [2025-05-08T13:40:54,006][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][297] overhead, spent [856ms] collecting in the last [2.1s] 1> [2025-05-08T13:40:54,004][WARN ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][298] overhead, spent [856ms] collecting in the last [1.6s] 1> [2025-05-08T13:40:53,996][WARN ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][298] overhead, spent [856ms] collecting in the last [1.6s] 1> [2025-05-08T13:40:53,986][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][297] overhead, spent [856ms] collecting in the last [2.1s] 1> [2025-05-08T13:40:56,326][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [1059 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:40:57,411][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:40:56,321][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [1021 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:41:04,525][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][306] overhead, spent [303ms] collecting in the last [1.1s] 1> [2025-05-08T13:41:04,526][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][295] overhead, spent [303ms] collecting in the last [1.1s] 1> [2025-05-08T13:41:04,526][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][307] overhead, spent [303ms] collecting in the last [1s] 1> [2025-05-08T13:41:04,526][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][305] overhead, spent [303ms] collecting in the last [1.1s] 1> [2025-05-08T13:41:04,527][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][305] overhead, spent [303ms] collecting in the last [1.1s] 1> [2025-05-08T13:41:04,531][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][306] overhead, spent [303ms] collecting in the last [1s] 1> [2025-05-08T13:41:06,444][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:41:10,881][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:41:13,166][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][312] overhead, spent [351ms] collecting in the last [1.3s] 1> [2025-05-08T13:41:13,168][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][302] overhead, spent [351ms] collecting in the last [1.3s] 1> [2025-05-08T13:41:13,373][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][314] overhead, spent [351ms] collecting in the last [1s] 1> [2025-05-08T13:41:13,427][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][313] overhead, spent [351ms] collecting in the last [1s] 1> [2025-05-08T13:41:13,384][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][315] overhead, spent [351ms] collecting in the last [1s] 1> [2025-05-08T13:41:13,379][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][314] overhead, spent [351ms] collecting in the last [1s] 1> [2025-05-08T13:41:15,165][INFO ][o.o.i.m.ICUCollationKeywordFieldMapperIT] [testUpperCaseFirst] Index [4] docs async: [true] bulk: [false] 1> [2025-05-08T13:41:18,157][INFO ][o.o.c.r.a.AllocationService] [node_s1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[foo][8]]]). 1> [2025-05-08T13:41:24,243][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:41:28,528][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][327] overhead, spent [358ms] collecting in the last [1s] 1> [2025-05-08T13:41:28,530][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][328] overhead, spent [358ms] collecting in the last [1.1s] 1> [2025-05-08T13:41:28,538][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][325] overhead, spent [358ms] collecting in the last [1.1s] 1> [2025-05-08T13:41:28,530][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][326] overhead, spent [358ms] collecting in the last [1s] 1> [2025-05-08T13:41:28,533][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][327] overhead, spent [358ms] collecting in the last [1s] 1> [2025-05-08T13:41:29,957][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [911 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:41:32,996][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Potentially blocked execution on network thread [opensearch[node_s2][transport_worker][T#2]] [RUNNABLE] [2980 milliseconds]: 1> org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:586) 1> org.opensearch.cluster.node.DiscoveryNode.(DiscoveryNode.java:344) 1> org.opensearch.cluster.coordination.FollowersChecker$FollowerCheckRequest.(FollowersChecker.java:505) 1> org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:87) 1> org.opensearch.transport.NativeMessageHandler.newRequest(NativeMessageHandler.java:316) 1> org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:271) 1> org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) 1> org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 1> org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 1> org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:41:41,732][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][338] overhead, spent [311ms] collecting in the last [1s] 1> [2025-05-08T13:41:41,733][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][324] overhead, spent [311ms] collecting in the last [1s] 1> [2025-05-08T13:41:41,732][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][337] overhead, spent [311ms] collecting in the last [1s] 1> [2025-05-08T13:41:41,732][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][335] overhead, spent [311ms] collecting in the last [1s] 1> [2025-05-08T13:41:41,734][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][337] overhead, spent [311ms] collecting in the last [1s] 1> [2025-05-08T13:41:41,732][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][336] overhead, spent [311ms] collecting in the last [1s] 1> [2025-05-08T13:41:52,097][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][344] overhead, spent [338ms] collecting in the last [1s] 1> [2025-05-08T13:41:52,095][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][347] overhead, spent [338ms] collecting in the last [1s] 1> [2025-05-08T13:41:52,096][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][345] overhead, spent [338ms] collecting in the last [1s] 1> [2025-05-08T13:41:52,095][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][333] overhead, spent [338ms] collecting in the last [1s] 1> [2025-05-08T13:41:52,096][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][346] overhead, spent [338ms] collecting in the last [1s] 1> [2025-05-08T13:41:52,094][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][346] overhead, spent [338ms] collecting in the last [1s] 1> [2025-05-08T13:42:09,657][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [930 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:42:10,142][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [800 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:42:09,500][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [516 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:42:12,519][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][363] overhead, spent [606ms] collecting in the last [1.5s] 1> [2025-05-08T13:42:12,513][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][362] overhead, spent [606ms] collecting in the last [1.4s] 1> [2025-05-08T13:42:12,516][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][361] overhead, spent [606ms] collecting in the last [1.4s] 1> [2025-05-08T13:42:12,514][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][363] overhead, spent [606ms] collecting in the last [1.5s] 1> [2025-05-08T13:42:12,512][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][364] overhead, spent [606ms] collecting in the last [1.4s] 1> [2025-05-08T13:42:14,206][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:42:14,146][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:42:14,257][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [1623 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:42:13,947][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [624 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:42:23,871][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][371] overhead, spent [265ms] collecting in the last [1s] 1> [2025-05-08T13:42:23,871][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][372] overhead, spent [265ms] collecting in the last [1s] 1> [2025-05-08T13:42:23,876][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][373] overhead, spent [265ms] collecting in the last [1s] 1> [2025-05-08T13:42:23,870][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][373] overhead, spent [265ms] collecting in the last [1s] 1> [2025-05-08T13:42:24,335][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][374] overhead, spent [265ms] collecting in the last [1s] 1> [2025-05-08T13:42:24,340][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][358] overhead, spent [265ms] collecting in the last [1s] 1> [2025-05-08T13:42:45,163][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][393] overhead, spent [288ms] collecting in the last [1s] 1> [2025-05-08T13:42:45,164][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][393] overhead, spent [288ms] collecting in the last [1s] 1> [2025-05-08T13:42:45,156][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][378] overhead, spent [288ms] collecting in the last [1s] 1> [2025-05-08T13:42:45,163][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][394] overhead, spent [288ms] collecting in the last [1s] 1> [2025-05-08T13:42:45,163][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][392] overhead, spent [288ms] collecting in the last [1s] 1> [2025-05-08T13:42:45,162][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][391] overhead, spent [288ms] collecting in the last [1s] 1> [2025-05-08T13:42:45,069][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [500 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:42:57,567][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][402] overhead, spent [522ms] collecting in the last [1.7s] 1> [2025-05-08T13:42:57,475][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Potentially blocked execution on network thread [opensearch[node_s1][transport_worker][T#2]] [RUNNABLE] [1815 milliseconds]: 1> org.opensearch.tasks.Task.(Task.java:115) 1> org.opensearch.tasks.TaskAwareRequest.createTask(TaskAwareRequest.java:67) 1> org.opensearch.tasks.TaskManager.register(TaskManager.java:206) 1> org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:91) 1> org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:278) 1> org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) 1> org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 1> org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 1> org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:42:57,531][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][401] overhead, spent [522ms] collecting in the last [1.4s] 1> [2025-05-08T13:42:57,553][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][400] overhead, spent [522ms] collecting in the last [1.4s] 1> [2025-05-08T13:42:57,608][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][402] overhead, spent [522ms] collecting in the last [1.5s] 1> [2025-05-08T13:42:57,562][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][387] overhead, spent [522ms] collecting in the last [1.1s] 1> [2025-05-08T13:42:57,458][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Potentially blocked execution on network thread [opensearch[node_s4][transport_worker][T#1]] [RUNNABLE] [600 milliseconds]: 1> org.opensearch.core.common.io.stream.InputStreamStreamInput.readByte(InputStreamStreamInput.java:73) 1> org.opensearch.core.common.io.stream.FilterStreamInput.readByte(FilterStreamInput.java:55) 1> org.opensearch.core.common.io.stream.StreamInput.readVInt(StreamInput.java:219) 1> org.opensearch.core.common.io.stream.StreamInput.readArraySize(StreamInput.java:1319) 1> org.opensearch.core.common.io.stream.StreamInput.readString(StreamInput.java:428) 1> org.opensearch.core.tasks.TaskId.readFromStream(TaskId.java:98) 1> org.opensearch.transport.TransportRequest.(TransportRequest.java:74) 1> org.opensearch.cluster.coordination.FollowersChecker$FollowerCheckRequest.(FollowersChecker.java:503) 1> org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:87) 1> org.opensearch.transport.NativeMessageHandler.newRequest(NativeMessageHandler.java:316) 1> org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:271) 1> org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) 1> org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 1> org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 1> org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:42:58,331][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:43:12,138][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:43:12,064][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:43:12,444][INFO ][o.o.i.m.ICUCollationKeywordFieldMapperIT] [testUpperCaseFirst] after test 1> [2025-05-08T13:43:12,587][INFO ][o.o.t.OpenSearchTestClusterRule] [testUpperCaseFirst] [ICUCollationKeywordFieldMapperIT#testUpperCaseFirst]: cleaning up after test 1> [2025-05-08T13:43:17,294][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:43:18,848][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_s1] [foo/8FjOAzM7Sk6YAWNm1jmvIw] deleting index 1> [2025-05-08T13:43:18,882][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:43:20,474][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][422] overhead, spent [275ms] collecting in the last [1s] 1> [2025-05-08T13:43:20,474][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][407] overhead, spent [275ms] collecting in the last [1s] 1> [2025-05-08T13:43:20,474][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][421] overhead, spent [275ms] collecting in the last [1s] 1> [2025-05-08T13:43:20,474][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][424] overhead, spent [275ms] collecting in the last [1s] 1> [2025-05-08T13:43:20,475][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][421] overhead, spent [275ms] collecting in the last [1s] 1> [2025-05-08T13:43:20,476][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][420] overhead, spent [275ms] collecting in the last [1s] 1> [2025-05-08T13:43:22,922][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][1] start check index 1> [2025-05-08T13:43:23,037][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][0] start check index 1> [2025-05-08T13:43:23,277][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][0] end check index 1> [2025-05-08T13:43:23,277][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][1] end check index 1> [2025-05-08T13:43:24,486][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][3] start check index 1> [2025-05-08T13:43:24,624][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][8] start check index 1> [2025-05-08T13:43:24,676][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][3] end check index 1> [2025-05-08T13:43:24,783][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][8] end check index 1> [2025-05-08T13:43:25,747][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][2] start check index 1> [2025-05-08T13:43:25,807][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][5] start check index 1> [2025-05-08T13:43:25,933][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][2] end check index 1> [2025-05-08T13:43:25,967][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][5] end check index 1> [2025-05-08T13:43:26,925][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][4] start check index 1> [2025-05-08T13:43:26,984][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][7] start check index 1> [2025-05-08T13:43:27,106][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][4] end check index 1> [2025-05-08T13:43:27,148][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][7] end check index 1> [2025-05-08T13:43:28,044][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][6] start check index 1> [2025-05-08T13:43:28,210][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][6] end check index 1> [2025-05-08T13:43:29,340][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [514 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:43:32,549][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][433] overhead, spent [469ms] collecting in the last [1.8s] 1> [2025-05-08T13:43:32,574][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][416] overhead, spent [469ms] collecting in the last [1.2s] 1> [2025-05-08T13:43:33,419][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:43:33,414][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:43:32,886][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Potentially blocked execution on network thread [opensearch[node_s1][transport_worker][T#2]] [RUNNABLE] [600 milliseconds]: 1> org.opensearch.core.common.io.stream.InputStreamStreamInput.read(InputStreamStreamInput.java:114) 1> org.opensearch.core.common.io.stream.FilterStreamInput.read(FilterStreamInput.java:95) 1> org.opensearch.transport.NativeMessageHandler.checkStreamIsFullyConsumed(NativeMessageHandler.java:340) 1> org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:273) 1> org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) 1> org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 1> org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 1> org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:43:33,195][INFO ][o.o.c.c.C.CoordinatorPublication] [node_s1] after [14s] publication of cluster state version [20] is still waiting for {node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT] 1> [2025-05-08T13:43:35,580][INFO ][o.o.c.m.MetadataIndexTemplateService] [node_s1] removing template [random_index_template] 1> [2025-05-08T13:43:41,396][INFO ][o.o.t.OpenSearchTestClusterRule] [testUpperCaseFirst] [ICUCollationKeywordFieldMapperIT#testUpperCaseFirst]: cleaned up after test 1> [2025-05-08T13:43:42,399][INFO ][o.o.t.OpenSearchTestClusterRule] [testIgnorePunctuation] [ICUCollationKeywordFieldMapperIT#testIgnorePunctuation]: setting up test 1> [2025-05-08T13:43:42,537][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:43:43,441][DEPRECATION][o.o.d.c.s.Settings ] [node_s1] [index.force_memory_term_dictionary] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version. 1> [2025-05-08T13:43:43,751][INFO ][o.o.p.PluginsService ] [node_s1] PluginService:onIndexModule index:[1Vm94OZ7QHaPi5JfY35XLQ/0c5_VEB0T2O2crfjZ2TMmQ] 1> [2025-05-08T13:43:44,093][INFO ][o.o.c.m.MetadataIndexTemplateService] [node_s1] adding template [random_index_template] for index patterns [*] 1> [2025-05-08T13:43:46,838][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][443] overhead, spent [253ms] collecting in the last [1s] 1> [2025-05-08T13:43:46,839][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][428] overhead, spent [253ms] collecting in the last [1s] 1> [2025-05-08T13:43:46,841][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][445] overhead, spent [253ms] collecting in the last [1s] 1> [2025-05-08T13:43:46,838][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][444] overhead, spent [253ms] collecting in the last [1s] 1> [2025-05-08T13:43:46,955][INFO ][o.o.t.OpenSearchTestClusterRule] [testIgnorePunctuation] [ICUCollationKeywordFieldMapperIT#testIgnorePunctuation]: all set up test 1> [2025-05-08T13:43:46,972][INFO ][o.o.i.m.ICUCollationKeywordFieldMapperIT] [testIgnorePunctuation] before test 1> [2025-05-08T13:43:47,373][INFO ][o.o.p.PluginsService ] [node_s1] PluginService:onIndexModule index:[foo/Na7tN12aQBGy8_iboP23SQ] 1> [2025-05-08T13:43:47,603][INFO ][o.o.c.m.MetadataCreateIndexService] [node_s1] [foo] creating index, cause [api], templates [random_index_template], shards [9]/[0] 1> [2025-05-08T13:43:47,647][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:43:49,715][INFO ][o.o.p.PluginsService ] [node_s4] PluginService:onIndexModule index:[foo/Na7tN12aQBGy8_iboP23SQ] 1> [2025-05-08T13:43:49,715][INFO ][o.o.p.PluginsService ] [node_s3] PluginService:onIndexModule index:[foo/Na7tN12aQBGy8_iboP23SQ] 1> [2025-05-08T13:44:01,785][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:44:01,632][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:44:01,608][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:44:02,246][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:44:01,616][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:44:01,646][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:44:02,239][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [865 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:44:01,632][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:44:04,068][INFO ][o.o.c.c.C.CoordinatorPublication] [node_s1] after [16s] publication of cluster state version [24] is still waiting for {node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT], {node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT] 1> [2025-05-08T13:44:07,933][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][460] overhead, spent [269ms] collecting in the last [1s] 1> [2025-05-08T13:44:07,932][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][461] overhead, spent [269ms] collecting in the last [1s] 1> [2025-05-08T13:44:07,933][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][461] overhead, spent [269ms] collecting in the last [1s] 1> [2025-05-08T13:44:07,941][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][442] overhead, spent [269ms] collecting in the last [1s] 1> [2025-05-08T13:44:07,932][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][459] overhead, spent [269ms] collecting in the last [1s] 1> [2025-05-08T13:44:18,098][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][470] overhead, spent [308ms] collecting in the last [1s] 1> [2025-05-08T13:44:18,098][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][469] overhead, spent [308ms] collecting in the last [1s] 1> [2025-05-08T13:44:18,095][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][468] overhead, spent [308ms] collecting in the last [1s] 1> [2025-05-08T13:44:18,097][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][470] overhead, spent [308ms] collecting in the last [1s] 1> [2025-05-08T13:44:18,097][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][468] overhead, spent [308ms] collecting in the last [1s] 1> [2025-05-08T13:44:18,097][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][451] overhead, spent [308ms] collecting in the last [1s] 1> [2025-05-08T13:44:18,738][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:44:25,330][INFO ][o.o.c.r.a.AllocationService] [node_s1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[foo][8]]]). 1> [2025-05-08T13:44:29,711][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][476] overhead, spent [513ms] collecting in the last [2s] 1> [2025-05-08T13:44:30,970][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:44:30,968][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [601 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:44:35,029][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:44:35,104][INFO ][o.o.i.m.ICUCollationKeywordFieldMapperIT] [testIgnorePunctuation] Index [2] docs async: [true] bulk: [false] 1> [2025-05-08T13:44:37,328][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [404 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:44:37,607][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [2020 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:44:46,284][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][492] overhead, spent [286ms] collecting in the last [1s] 1> [2025-05-08T13:44:46,284][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][494] overhead, spent [286ms] collecting in the last [1s] 1> [2025-05-08T13:44:46,283][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][490] overhead, spent [286ms] collecting in the last [1s] 1> [2025-05-08T13:44:46,285][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][493] overhead, spent [286ms] collecting in the last [1s] 1> [2025-05-08T13:44:46,456][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][473] overhead, spent [286ms] collecting in the last [1s] 1> [2025-05-08T13:44:55,229][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][500] overhead, spent [319ms] collecting in the last [1s] 1> [2025-05-08T13:44:55,222][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][502] overhead, spent [319ms] collecting in the last [1s] 1> [2025-05-08T13:44:55,233][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][498] overhead, spent [319ms] collecting in the last [1s] 1> [2025-05-08T13:44:55,228][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][501] overhead, spent [319ms] collecting in the last [1s] 1> [2025-05-08T13:44:55,227][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][502] overhead, spent [319ms] collecting in the last [1.1s] 1> [2025-05-08T13:45:11,103][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [425 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:45:11,189][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:45:11,103][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:45:12,850][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Potentially blocked execution on network thread [opensearch[node_s1][transport_worker][T#2]] [RUNNABLE] [2001 milliseconds]: 1> java.base/java.util.HashMap.hash(HashMap.java:338) 1> java.base/java.util.HashMap.getNode(HashMap.java:576) 1> java.base/java.util.HashMap.get(HashMap.java:564) 1> java.base/java.util.Collections$UnmodifiableMap.get(Collections.java:1656) 1> org.opensearch.cluster.node.DiscoveryNodes.nodeExists(DiscoveryNodes.java:273) 1> org.opensearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:228) 1> org.opensearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:152) 1> org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) 1> org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) 1> org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:278) 1> org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) 1> org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 1> org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 1> org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:45:12,855][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Potentially blocked execution on network thread [opensearch[node_s0][transport_worker][T#2]] [RUNNABLE] [1400 milliseconds]: 1> java.base/java.util.HashMap.resize(HashMap.java:741) 1> java.base/java.util.HashMap.putVal(HashMap.java:669) 1> java.base/java.util.HashMap.putIfAbsent(HashMap.java:1152) 1> org.opensearch.common.util.concurrent.ThreadContext$ThreadContextStruct.putSingleHeader(ThreadContext.java:755) 1> org.opensearch.common.util.concurrent.ThreadContext$ThreadContextStruct.putTransient(ThreadContext.java:906) 1> org.opensearch.common.util.concurrent.ThreadContext.putTransient(ThreadContext.java:511) 1> org.opensearch.telemetry.tracing.ThreadContextBasedTracerContextStorage.put(ThreadContextBasedTracerContextStorage.java:47) 1> org.opensearch.telemetry.tracing.ThreadContextBasedTracerContextStorage.put(ThreadContextBasedTracerContextStorage.java:25) 1> org.opensearch.telemetry.tracing.DefaultSpanScope.attach(DefaultSpanScope.java:66) 1> org.opensearch.telemetry.tracing.DefaultTracer.withSpanInScope(DefaultTracer.java:84) 1> org.opensearch.telemetry.tracing.WrappedTracer.withSpanInScope(WrappedTracer.java:59) 1> org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:212) 1> org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) 1> org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 1> org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 1> org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:45:28,458][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Potentially blocked execution on network thread [opensearch[node_s3][transport_worker][T#2]] [RUNNABLE] [323 milliseconds]: 1> org.opensearch.action.DocWriteResponse.(DocWriteResponse.java:163) 1> org.opensearch.action.index.IndexResponse.(IndexResponse.java:60) 1> org.opensearch.action.bulk.BulkItemResponse.(BulkItemResponse.java:401) 1> org.opensearch.action.bulk.BulkShardResponse.lambda$new$0(BulkShardResponse.java:62) 1> org.opensearch.core.common.io.stream.StreamInput.readArray(StreamInput.java:1001) 1> org.opensearch.action.bulk.BulkShardResponse.(BulkShardResponse.java:61) 1> org.opensearch.action.bulk.TransportShardBulkAction.newResponseInstance(TransportShardBulkAction.java:406) 1> org.opensearch.action.bulk.TransportShardBulkAction.newResponseInstance(TransportShardBulkAction.java:125) 1> org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.read(TransportReplicationAction.java:1140) 1> org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.read(TransportReplicationAction.java:1136) 1> org.opensearch.telemetry.tracing.handler.TraceableTransportResponseHandler.read(TraceableTransportResponseHandler.java:66) 1> org.opensearch.telemetry.tracing.handler.TraceableTransportResponseHandler.read(TraceableTransportResponseHandler.java:26) 1> org.opensearch.transport.TransportService$9.read(TransportService.java:1751) 1> org.opensearch.transport.TransportService$9.read(TransportService.java:1731) 1> org.opensearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1508) 1> org.opensearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1495) 1> org.opensearch.transport.NativeMessageHandler.handleResponse(NativeMessageHandler.java:405) 1> org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:174) 1> org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 1> org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 1> org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:45:28,448][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [523 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:45:28,686][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][530] overhead, spent [264ms] collecting in the last [1s] 1> [2025-05-08T13:45:28,682][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][528] overhead, spent [264ms] collecting in the last [1s] 1> [2025-05-08T13:45:28,686][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][509] overhead, spent [264ms] collecting in the last [1s] 1> [2025-05-08T13:45:28,686][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][531] overhead, spent [264ms] collecting in the last [1s] 1> [2025-05-08T13:45:28,685][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][527] overhead, spent [264ms] collecting in the last [1s] 1> [2025-05-08T13:45:28,689][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][531] overhead, spent [264ms] collecting in the last [1s] 1> [2025-05-08T13:45:39,843][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [468 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:45:42,600][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [932 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:45:42,586][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [903 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:45:42,589][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [901 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:45:42,583][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [916 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:45:42,637][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [1077 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:45:42,582][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [909 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:45:43,522][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Potentially blocked execution on network thread [opensearch[node_s3][transport_worker][T#2]] [RUNNABLE] [1032 milliseconds]: 1> org.opensearch.transport.InboundDecoder.readHeader(InboundDecoder.java:195) 1> org.opensearch.transport.InboundDecoder.internalDecode(InboundDecoder.java:93) 1> org.opensearch.transport.InboundDecoder.decode(InboundDecoder.java:71) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:61) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:45:53,677][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [1630 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:46:05,436][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [338 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:46:05,379][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [342 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:46:30,968][INFO ][o.o.i.m.ICUCollationKeywordFieldMapperIT] [testIgnorePunctuation] after test 1> [2025-05-08T13:46:30,890][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:46:31,358][INFO ][o.o.t.OpenSearchTestClusterRule] [testIgnorePunctuation] [ICUCollationKeywordFieldMapperIT#testIgnorePunctuation]: cleaning up after test 1> [2025-05-08T13:46:33,999][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [401 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:46:36,014][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][589] overhead, spent [638ms] collecting in the last [2.4s] 1> [2025-05-08T13:46:36,007][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][591] overhead, spent [638ms] collecting in the last [2.4s] 1> [2025-05-08T13:46:36,017][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][587] overhead, spent [638ms] collecting in the last [2.4s] 1> [2025-05-08T13:46:36,009][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][588] overhead, spent [638ms] collecting in the last [2.4s] 1> [2025-05-08T13:46:36,021][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][591] overhead, spent [638ms] collecting in the last [2.4s] 1> [2025-05-08T13:46:33,997][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:46:37,890][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:46:37,973][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [1063 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:46:37,845][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [411 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:46:37,845][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [407 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:46:37,893][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [628 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:46:39,892][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:46:41,830][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:46:43,395][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_s1] [foo/Na7tN12aQBGy8_iboP23SQ] deleting index 1> [2025-05-08T13:46:43,417][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:46:47,402][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][1] start check index 1> [2025-05-08T13:46:47,401][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][0] start check index 1> [2025-05-08T13:46:47,772][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][0] end check index 1> [2025-05-08T13:46:47,773][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][1] end check index 1> [2025-05-08T13:46:51,946][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][3] start check index 1> [2025-05-08T13:46:52,200][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][8] start check index 1> [2025-05-08T13:46:52,493][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][8] end check index 1> [2025-05-08T13:46:52,493][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][3] end check index 1> [2025-05-08T13:46:53,527][INFO ][o.o.c.c.C.CoordinatorPublication] [node_s1] after [10.1s] publication of cluster state version [30] is still waiting for {node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT], {node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT] 1> [2025-05-08T13:46:53,778][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][2] start check index 1> [2025-05-08T13:46:53,912][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][5] start check index 1> [2025-05-08T13:46:53,977][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][2] end check index 1> [2025-05-08T13:46:54,067][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][5] end check index 1> [2025-05-08T13:46:55,073][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][4] start check index 1> [2025-05-08T13:46:55,157][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][7] start check index 1> [2025-05-08T13:46:55,243][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][4] end check index 1> [2025-05-08T13:46:55,315][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][7] end check index 1> [2025-05-08T13:46:56,341][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][6] start check index 1> [2025-05-08T13:46:56,500][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][6] end check index 1> [2025-05-08T13:46:57,636][INFO ][o.o.c.m.MetadataIndexTemplateService] [node_s1] removing template [random_index_template] 1> [2025-05-08T13:47:00,763][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][611] overhead, spent [317ms] collecting in the last [1s] 1> [2025-05-08T13:47:00,758][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][613] overhead, spent [317ms] collecting in the last [1s] 1> [2025-05-08T13:47:00,810][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][588] overhead, spent [317ms] collecting in the last [1s] 1> [2025-05-08T13:47:00,758][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][613] overhead, spent [317ms] collecting in the last [1s] 1> [2025-05-08T13:47:00,761][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][610] overhead, spent [317ms] collecting in the last [1s] 1> [2025-05-08T13:47:00,763][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][609] overhead, spent [317ms] collecting in the last [1s] 1> [2025-05-08T13:47:01,004][INFO ][o.o.t.OpenSearchTestClusterRule] [testIgnorePunctuation] [ICUCollationKeywordFieldMapperIT#testIgnorePunctuation]: cleaned up after test 1> [2025-05-08T13:47:01,811][INFO ][o.o.t.OpenSearchTestClusterRule] [testSecondaryStrength] [ICUCollationKeywordFieldMapperIT#testSecondaryStrength]: setting up test 1> [2025-05-08T13:47:01,934][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:47:04,969][DEPRECATION][o.o.d.c.s.Settings ] [node_s1] [index.force_memory_term_dictionary] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version. 1> [2025-05-08T13:47:05,240][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:47:05,631][INFO ][o.o.p.PluginsService ] [node_s1] PluginService:onIndexModule index:[ZKkg0ycZSGSUbEUhs1fp_Q/c0LXBt6pTgOX-3BvOz8Taw] 1> [2025-05-08T13:47:06,094][INFO ][o.o.c.m.MetadataIndexTemplateService] [node_s1] adding template [random_index_template] for index patterns [*] 1> [2025-05-08T13:47:09,347][INFO ][o.o.t.OpenSearchTestClusterRule] [testSecondaryStrength] [ICUCollationKeywordFieldMapperIT#testSecondaryStrength]: all set up test 1> [2025-05-08T13:47:09,382][INFO ][o.o.i.m.ICUCollationKeywordFieldMapperIT] [testSecondaryStrength] before test 1> [2025-05-08T13:47:09,851][INFO ][o.o.p.PluginsService ] [node_s1] PluginService:onIndexModule index:[foo/CMMQHqNGS2CxSFShmab1JA] 1> [2025-05-08T13:47:10,078][INFO ][o.o.c.m.MetadataCreateIndexService] [node_s1] [foo] creating index, cause [api], templates [random_index_template], shards [9]/[0] 1> [2025-05-08T13:47:10,137][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:47:17,324][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][622] overhead, spent [519ms] collecting in the last [1.6s] 1> [2025-05-08T13:47:17,335][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][623] overhead, spent [519ms] collecting in the last [1.6s] 1> [2025-05-08T13:47:17,324][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][625] overhead, spent [519ms] collecting in the last [1.6s] 1> [2025-05-08T13:47:17,340][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][621] overhead, spent [519ms] collecting in the last [1.7s] 1> [2025-05-08T13:47:17,325][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][625] overhead, spent [519ms] collecting in the last [1.7s] 1> [2025-05-08T13:47:17,235][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [2017 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:47:16,113][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:47:15,759][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:47:15,113][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:47:20,041][INFO ][o.o.p.PluginsService ] [node_s3] PluginService:onIndexModule index:[foo/CMMQHqNGS2CxSFShmab1JA] 1> [2025-05-08T13:47:20,041][INFO ][o.o.p.PluginsService ] [node_s4] PluginService:onIndexModule index:[foo/CMMQHqNGS2CxSFShmab1JA] 1> [2025-05-08T13:47:20,546][INFO ][o.o.c.c.C.CoordinatorPublication] [node_s1] after [9.9s] publication of cluster state version [34] is still waiting for {node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT], {node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT] 1> [2025-05-08T13:47:24,641][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [1429 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:47:24,226][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [828 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:47:28,495][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [364 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:47:37,235][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:47:39,727][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:47:46,448][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][648] overhead, spent [762ms] collecting in the last [2.9s] 1> [2025-05-08T13:47:46,449][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][644] overhead, spent [762ms] collecting in the last [2.9s] 1> [2025-05-08T13:47:46,454][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][648] overhead, spent [762ms] collecting in the last [2.8s] 1> [2025-05-08T13:47:46,027][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [800 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:47:46,101][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [2095 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:47:48,728][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:47:48,741][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:47:49,235][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [1007 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:47:49,130][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Potentially blocked execution on network thread [opensearch[node_s2][transport_worker][T#1]] [RUNNABLE] [607 milliseconds]: 1> org.opensearch.common.util.concurrent.ThreadContext.setHeaders(ThreadContext.java:389) 1> org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:143) 1> org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 1> org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 1> org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:47:50,805][INFO ][o.o.c.c.C.CoordinatorPublication] [node_s1] after [10.6s] publication of cluster state version [37] is still waiting for {node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT], {node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT], {node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true} [SENT_PUBLISH_REQUEST], {node_s5}{7Bx5ewP5Q42abT6xeM4uBA}{OVcy3CfWRJeYN8aHI5nZRg}{127.0.0.1}{127.0.0.1:41581}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT], {node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT], {node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT] 1> [2025-05-08T13:47:50,967][WARN ][o.o.g.PersistedClusterStateService] [node_s2] writing cluster state took [10640ms] which is above the warn threshold of [10s]; wrote global metadata [false] and metadata for [1] indices and skipped [0] unchanged indices 1> [2025-05-08T13:47:59,566][INFO ][o.o.i.m.ICUCollationKeywordFieldMapperIT] [testSecondaryStrength] Index [2] docs async: [true] bulk: [false] 1> [2025-05-08T13:48:00,570][INFO ][o.o.c.r.a.AllocationService] [node_s1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[foo][8]]]). 1> [2025-05-08T13:48:02,579][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][661] overhead, spent [281ms] collecting in the last [1s] 1> [2025-05-08T13:48:02,576][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][661] overhead, spent [281ms] collecting in the last [1s] 1> [2025-05-08T13:48:02,583][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][661] overhead, spent [281ms] collecting in the last [1s] 1> [2025-05-08T13:48:02,575][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][657] overhead, spent [281ms] collecting in the last [1s] 1> [2025-05-08T13:48:02,579][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][660] overhead, spent [281ms] collecting in the last [1s] 1> [2025-05-08T13:48:02,593][INFO ][o.o.m.j.JvmGcMonitorService] [node_s1] [gc][634] overhead, spent [281ms] collecting in the last [1s] 1> [2025-05-08T13:48:05,622][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:48:11,067][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Potentially blocked execution on network thread [opensearch[node_s4][transport_worker][T#2]] [RUNNABLE] [432 milliseconds]: 1> org.opensearch.common.util.concurrent.AbstractRefCounted.incRef(AbstractRefCounted.java:54) 1> org.opensearch.common.bytes.ReleasableBytesReference.(ReleasableBytesReference.java:66) 1> org.opensearch.common.bytes.ReleasableBytesReference.retainedSlice(ReleasableBytesReference.java:83) 1> org.opensearch.transport.InboundBytesHandler.releasePendingBytes(InboundBytesHandler.java:111) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:63) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:48:11,355][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [2435 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:48:11,353][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:48:20,336][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [619 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:48:20,228][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:48:20,368][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:48:20,276][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [809 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:48:21,186][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [845 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:48:20,273][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [426 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:48:20,809][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [1236 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.1 1> 9.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:48:23,267][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:48:38,630][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:48:59,659][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [1196 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:48:59,766][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:49:00,620][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [1010 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:48:59,625][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [1148 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:48:59,755][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:48:59,696][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [1159 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:49:01,161][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Potentially blocked execution on network thread [opensearch[node_s0][transport_worker][T#1]] [RUNNABLE] [1983 milliseconds]: 1> org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:137) 1> org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 1> org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 1> org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:49:02,263][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:49:02,510][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:49:31,706][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [467 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:49:33,072][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:49:31,573][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [467 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:49:39,933][INFO ][o.o.i.m.ICUCollationKeywordFieldMapperIT] [testSecondaryStrength] after test 1> [2025-05-08T13:49:40,129][INFO ][o.o.t.OpenSearchTestClusterRule] [testSecondaryStrength] [ICUCollationKeywordFieldMapperIT#testSecondaryStrength]: cleaning up after test 1> [2025-05-08T13:49:41,605][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:49:42,494][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_s1] [foo/CMMQHqNGS2CxSFShmab1JA] deleting index 1> [2025-05-08T13:49:42,506][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:49:49,451][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:49:46,993][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:49:49,441][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:49:49,451][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:49:49,445][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:49:49,454][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:49:49,821][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Potentially blocked execution on network thread [opensearch[node_s5][transport_worker][T#2]] [RUNNABLE] [400 milliseconds]: 1> org.opensearch.common.bytes.ReleasableBytesReference.get(ReleasableBytesReference.java:93) 1> org.opensearch.transport.TcpTransport.readHeaderBuffer(TcpTransport.java:794) 1> org.opensearch.transport.TcpTransport.readMessageLength(TcpTransport.java:789) 1> org.opensearch.transport.InboundDecoder.internalDecode(InboundDecoder.java:80) 1> org.opensearch.transport.InboundDecoder.decode(InboundDecoder.java:71) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:61) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:49:50,070][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [1000 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:49:52,637][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][1] start check index 1> [2025-05-08T13:49:52,684][INFO ][o.o.c.c.C.CoordinatorPublication] [node_s1] after [10.3s] publication of cluster state version [40] is still waiting for {node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT], {node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT] 1> [2025-05-08T13:49:52,685][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][0] start check index 1> [2025-05-08T13:49:53,187][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][0] end check index 1> [2025-05-08T13:49:53,186][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][1] end check index 1> [2025-05-08T13:49:54,890][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [623 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:49:56,747][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][3] start check index 1> [2025-05-08T13:49:57,040][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][8] start check index 1> [2025-05-08T13:49:57,436][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][3] end check index 1> [2025-05-08T13:49:57,436][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][8] end check index 1> [2025-05-08T13:49:58,594][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][2] start check index 1> [2025-05-08T13:49:58,693][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][5] start check index 1> [2025-05-08T13:49:58,776][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][2] end check index 1> [2025-05-08T13:49:58,854][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][5] end check index 1> [2025-05-08T13:49:59,685][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][4] start check index 1> [2025-05-08T13:49:59,778][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][7] start check index 1> [2025-05-08T13:49:59,848][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][4] end check index 1> [2025-05-08T13:49:59,931][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][7] end check index 1> [2025-05-08T13:50:00,758][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][6] start check index 1> [2025-05-08T13:50:00,910][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][6] end check index 1> [2025-05-08T13:50:02,116][INFO ][o.o.c.m.MetadataIndexTemplateService] [node_s1] removing template [random_index_template] 1> [2025-05-08T13:50:11,960][INFO ][o.o.t.OpenSearchTestClusterRule] [testSecondaryStrength] [ICUCollationKeywordFieldMapperIT#testSecondaryStrength]: cleaned up after test 1> [2025-05-08T13:50:12,876][INFO ][o.o.t.OpenSearchTestClusterRule] [testNormalization] [ICUCollationKeywordFieldMapperIT#testNormalization]: setting up test 1> [2025-05-08T13:50:13,088][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:50:14,480][DEPRECATION][o.o.d.c.s.Settings ] [node_s1] [index.force_memory_term_dictionary] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version. 1> [2025-05-08T13:50:14,782][INFO ][o.o.p.PluginsService ] [node_s1] PluginService:onIndexModule index:[d35upYiOQkyvWeq18Vbjgg/--hYC5bHQZy4oi1r7uY44w] 1> [2025-05-08T13:50:15,113][INFO ][o.o.c.m.MetadataIndexTemplateService] [node_s1] adding template [random_index_template] for index patterns [*] 1> [2025-05-08T13:50:17,689][INFO ][o.o.t.OpenSearchTestClusterRule] [testNormalization] [ICUCollationKeywordFieldMapperIT#testNormalization]: all set up test 1> [2025-05-08T13:50:17,698][INFO ][o.o.i.m.ICUCollationKeywordFieldMapperIT] [testNormalization] before test 1> [2025-05-08T13:50:19,668][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:50:20,621][INFO ][o.o.p.PluginsService ] [node_s1] PluginService:onIndexModule index:[foo/AlCiHCCqRqmjoEspiGpRzg] 1> [2025-05-08T13:50:22,376][INFO ][o.o.c.m.MetadataCreateIndexService] [node_s1] [foo] creating index, cause [api], templates [random_index_template], shards [9]/[0] 1> [2025-05-08T13:50:22,510][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:50:30,004][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][786] overhead, spent [903ms] collecting in the last [1.8s] 1> [2025-05-08T13:50:30,002][WARN ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][790] overhead, spent [903ms] collecting in the last [1.6s] 1> [2025-05-08T13:50:30,014][WARN ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][790] overhead, spent [903ms] collecting in the last [1.6s] 1> [2025-05-08T13:50:30,006][WARN ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][790] overhead, spent [903ms] collecting in the last [1.5s] 1> [2025-05-08T13:50:30,008][WARN ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][788] overhead, spent [903ms] collecting in the last [1.6s] 1> [2025-05-08T13:50:30,672][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:50:30,680][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [404 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:50:33,292][INFO ][o.o.c.c.C.CoordinatorPublication] [node_s1] after [10.3s] publication of cluster state version [44] is still waiting for {node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT], {node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT], {node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT], {node_s5}{7Bx5ewP5Q42abT6xeM4uBA}{OVcy3CfWRJeYN8aHI5nZRg}{127.0.0.1}{127.0.0.1:41581}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT], {node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT], {node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true} [SENT_APPLY_COMMIT] 1> [2025-05-08T13:50:34,209][INFO ][o.o.p.PluginsService ] [node_s4] PluginService:onIndexModule index:[foo/AlCiHCCqRqmjoEspiGpRzg] 1> [2025-05-08T13:50:34,209][INFO ][o.o.p.PluginsService ] [node_s3] PluginService:onIndexModule index:[foo/AlCiHCCqRqmjoEspiGpRzg] 1> [2025-05-08T13:50:40,995][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:50:52,608][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:50:53,896][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [1424 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:50:53,893][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [1838 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:50:52,589][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:50:52,607][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:50:52,590][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:50:55,723][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:50:56,400][WARN ][o.o.g.PersistedClusterStateService] [node_s0] writing cluster state took [11201ms] which is above the warn threshold of [10s]; wrote global metadata [false] and metadata for [1] indices and skipped [0] unchanged indices 1> [2025-05-08T13:50:56,401][WARN ][o.o.g.PersistedClusterStateService] [node_s2] writing cluster state took [11189ms] which is above the warn threshold of [10s]; wrote global metadata [false] and metadata for [1] indices and skipped [0] unchanged indices 1> [2025-05-08T13:50:56,399][WARN ][o.o.g.PersistedClusterStateService] [node_s3] writing cluster state took [11199ms] which is above the warn threshold of [10s]; wrote global metadata [false] and metadata for [1] indices and skipped [0] unchanged indices 1> [2025-05-08T13:50:56,398][WARN ][o.o.g.PersistedClusterStateService] [node_s1] writing cluster state took [11188ms] which is above the warn threshold of [10s]; wrote global metadata [false] and metadata for [1] indices and skipped [0] unchanged indices 1> [2025-05-08T13:50:56,399][WARN ][o.o.g.PersistedClusterStateService] [node_s4] writing cluster state took [11204ms] which is above the warn threshold of [10s]; wrote global metadata [false] and metadata for [1] indices and skipped [0] unchanged indices 1> [2025-05-08T13:50:56,730][INFO ][o.o.c.c.C.CoordinatorPublication] [node_s1] after [11.9s] publication of cluster state version [45] is still waiting for {node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true} [SENT_PUBLISH_REQUEST], {node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true} [SENT_PUBLISH_REQUEST], {node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true} [SENT_PUBLISH_REQUEST], {node_s5}{7Bx5ewP5Q42abT6xeM4uBA}{OVcy3CfWRJeYN8aHI5nZRg}{127.0.0.1}{127.0.0.1:41581}{shard_indexing_pressure_enabled=true} [SENT_PUBLISH_REQUEST], {node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true} [SENT_PUBLISH_REQUEST], {node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true} [SENT_PUBLISH_REQUEST] 1> [2025-05-08T13:51:06,154][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:51:08,301][INFO ][o.o.i.m.ICUCollationKeywordFieldMapperIT] [testNormalization] Index [2] docs async: [true] bulk: [false] 1> [2025-05-08T13:51:18,181][INFO ][o.o.c.r.a.AllocationService] [node_s1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[foo][8]]]). 1> [2025-05-08T13:51:22,316][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:51:28,892][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][837] overhead, spent [684ms] collecting in the last [1.8s] 1> [2025-05-08T13:51:28,909][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][835] overhead, spent [684ms] collecting in the last [2s] 1> [2025-05-08T13:51:28,887][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][836] overhead, spent [684ms] collecting in the last [2s] 1> [2025-05-08T13:51:28,898][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][837] overhead, spent [684ms] collecting in the last [1.8s] 1> [2025-05-08T13:51:28,894][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][833] overhead, spent [684ms] collecting in the last [1.9s] 1> [2025-05-08T13:51:28,876][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [800 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:51:29,609][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:51:29,602][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:51:30,325][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:51:32,421][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [1200 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:51:32,121][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:51:32,423][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:51:33,779][WARN ][o.o.c.r.a.AllocationService] [node_s1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set 1> [2025-05-08T13:51:43,960][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [2801 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:51:43,087][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:51:43,574][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:51:42,950][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [324 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:51:43,515][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Potentially blocked execution on network thread [opensearch[node_s3][transport_worker][T#1]] [RUNNABLE] [400 milliseconds]: 1> java.base/java.lang.String.split(String.java:3371) 1> java.base/java.lang.String.split(String.java:3354) 1> java.base/java.lang.String.split(String.java:3447) 1> org.opensearch.test.telemetry.tracing.MockTracingContextPropagator.extract(MockTracingContextPropagator.java:45) 1> org.opensearch.test.telemetry.tracing.MockTracingContextPropagator.extractFromHeaders(MockTracingContextPropagator.java:60) 1> org.opensearch.telemetry.tracing.DefaultTracer.startSpan(DefaultTracer.java:106) 1> org.opensearch.telemetry.tracing.WrappedTracer.startSpan(WrappedTracer.java:79) 1> org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:211) 1> org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) 1> org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 1> org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 1> org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:52:11,943][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][872] overhead, spent [578ms] collecting in the last [1.9s] 1> [2025-05-08T13:52:11,930][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][869] overhead, spent [578ms] collecting in the last [1.9s] 1> [2025-05-08T13:52:11,935][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][871] overhead, spent [578ms] collecting in the last [2s] 1> [2025-05-08T13:52:11,931][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][873] overhead, spent [578ms] collecting in the last [1.9s] 1> [2025-05-08T13:52:11,940][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][873] overhead, spent [578ms] collecting in the last [1.9s] 1> [2025-05-08T13:52:14,547][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [381 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:52:14,608][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [784 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:52:13,204][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [600 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:52:14,568][WARN ][o.o.t.n.MockNioTransport ] [node_s3] Slow execution on network thread [589 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:52:14,528][WARN ][o.o.t.n.MockNioTransport ] [node_s0] Slow execution on network thread [336 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:194) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:52:13,211][WARN ][o.o.t.n.MockNioTransport ] [node_s2] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:52:15,701][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [400 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:52:22,966][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Slow execution on network thread [612 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:52:22,937][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [616 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:52:27,480][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Slow execution on network thread [2543 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:52:27,621][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Slow execution on network thread [3144 milliseconds] 1> java.lang.RuntimeException: Slow exception on network thread 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.maybeLogElapsedTime(MockNioTransport.java:455) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.unregister(MockNioTransport.java:446) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:170) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:52:27,459][WARN ][o.o.t.n.MockNioTransport ] [node_s4] Potentially blocked execution on network thread [opensearch[node_s4][transport_worker][T#1]] [RUNNABLE] [2744 milliseconds]: 1> org.opensearch.core.common.io.stream.InputStreamStreamInput.readByte(InputStreamStreamInput.java:73) 1> org.opensearch.core.common.io.stream.FilterStreamInput.readByte(FilterStreamInput.java:55) 1> org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:586) 1> org.opensearch.core.common.io.stream.StreamInput.readOptionalString(StreamInput.java:376) 1> org.opensearch.action.index.IndexRequest.(IndexRequest.java:163) 1> org.opensearch.action.DocWriteRequest.readDocumentRequest(DocWriteRequest.java:247) 1> org.opensearch.action.bulk.BulkItemRequest.(BulkItemRequest.java:68) 1> org.opensearch.action.bulk.BulkShardRequest.lambda$new$0(BulkShardRequest.java:65) 1> org.opensearch.core.common.io.stream.StreamInput.readOptionalWriteable(StreamInput.java:1013) 1> org.opensearch.action.bulk.BulkShardRequest.lambda$new$1(BulkShardRequest.java:65) 1> org.opensearch.core.common.io.stream.StreamInput.readArray(StreamInput.java:1001) 1> org.opensearch.action.bulk.BulkShardRequest.(BulkShardRequest.java:65) 1> org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:87) 1> org.opensearch.transport.NativeMessageHandler.newRequest(NativeMessageHandler.java:316) 1> org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:271) 1> org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) 1> org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 1> org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 1> org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:52:27,459][WARN ][o.o.t.n.MockNioTransport ] [node_s5] Potentially blocked execution on network thread [opensearch[node_s5][transport_worker][T#2]] [RUNNABLE] [2543 milliseconds]: 1> org.opensearch.nio.SocketChannelContext.readFromChannel(SocketChannelContext.java:297) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:53) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 2> मई ०८, २०२५ १:५२:३१ अपराह्न com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate 2> WARNING: Suite execution timed out: org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT 2> ==== jstack at approximately timeout time ==== 2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@636d0c61 2> at java.base@21.0.6/java.lang.Object.wait0(Native Method) 2> - waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@636d0c61 2> at java.base@21.0.6/java.lang.Object.wait(Object.java:366) 2> at java.base@21.0.6/java.lang.Thread.join(Thread.java:2079) 2> at java.base@21.0.6/java.lang.Thread.join(Thread.java:2155) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:639) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:496) 2> at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:112) 2> at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) 2> at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40) 2> at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:54) 2> at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:53) 2> at java.base@21.0.6/java.lang.invoke.LambdaForm$DMH/0x00007b49d8020000.invokeInterface(LambdaForm$DMH) 2> at java.base@21.0.6/java.lang.invoke.LambdaForm$MH/0x00007b49d8032800.invoke(LambdaForm$MH) 2> at java.base@21.0.6/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder) 2> at java.base@21.0.6/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:154) 2> at java.base@21.0.6/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) 2> at java.base@21.0.6/java.lang.reflect.Method.invoke(Method.java:580) 2> at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) 2> at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) 2> at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) 2> at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:92) 2> at jdk.proxy2/jdk.proxy2.$Proxy6.processTestClass(Unknown Source) 2> at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:183) 2> at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:132) 2> at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:103) 2> at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:63) 2> at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56) 2> at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:121) 2> at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71) 2> at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69) 2> at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74) 2> "Reference Handler" ID=9 RUNNABLE 2> at java.base@21.0.6/java.lang.ref.Reference.waitForReferencePendingList(Native Method) 2> at java.base@21.0.6/java.lang.ref.Reference.processPendingReferences(Reference.java:246) 2> at java.base@21.0.6/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:208) 2> "Finalizer" ID=10 WAITING on java.lang.ref.NativeReferenceQueue$Lock@47fdc353 2> at java.base@21.0.6/java.lang.Object.wait0(Native Method) 2> - waiting on java.lang.ref.NativeReferenceQueue$Lock@47fdc353 2> at java.base@21.0.6/java.lang.Object.wait(Object.java:366) 2> at java.base@21.0.6/java.lang.Object.wait(Object.java:339) 2> at java.base@21.0.6/java.lang.ref.NativeReferenceQueue.await(NativeReferenceQueue.java:48) 2> at java.base@21.0.6/java.lang.ref.ReferenceQueue.remove0(ReferenceQueue.java:158) 2> at java.base@21.0.6/java.lang.ref.NativeReferenceQueue.remove(NativeReferenceQueue.java:89) 2> at java.base@21.0.6/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:173) 2> "Signal Dispatcher" ID=11 RUNNABLE 2> "Common-Cleaner" ID=18 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3c145bc6 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3c145bc6 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1852) 2> at java.base@21.0.6/java.lang.ref.ReferenceQueue.await(ReferenceQueue.java:71) 2> at java.base@21.0.6/java.lang.ref.ReferenceQueue.remove0(ReferenceQueue.java:143) 2> at java.base@21.0.6/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:218) 2> at java.base@21.0.6/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:140) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> at java.base@21.0.6/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:186) 2> "Notification Thread" ID=20 RUNNABLE 2> "/127.0.0.1:43552 to /127.0.0.1:41323 workers" ID=31 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@12679463 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@12679463 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49) 2> at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:403) 2> at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) 2> at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:48) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> Locked synchronizers: 2> - java.util.concurrent.ThreadPoolExecutor$Worker@10a9d961 2> "/127.0.0.1:43552 to /127.0.0.1:41323 workers Thread 2" ID=32 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f4d1db5 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f4d1db5 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49) 2> at org.gradle.internal.remote.internal.hub.MessageHub$Connection 2> Dispatch.run(MessageHub.java:322) 2> at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) 2> at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:48) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> Locked synchronizers: 2> - java.util.concurrent.ThreadPoolExecutor$Worker@dbd8e44 2> "/127.0.0.1:43552 to /127.0.0.1:41323 workers Thread 3" ID=33 RUNNABLE (in native code) 2> at java.base@21.0.6/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base@21.0.6/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> - locked sun.nio.ch.Util$2@3683b254 2> - locked sun.nio.ch.EPollSelectorImpl@6bd58cc 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:147) 2> at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:187) 2> at com.esotericsoftware.kryo.io.Input.fill(Input.java:146) 2> at com.esotericsoftware.kryo.io.Input.require(Input.java:178) 2> at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295) 2> at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88) 2> at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64) 2> at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52) 2> at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:83) 2> at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270) 2> at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) 2> at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:48) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> Locked synchronizers: 2> - java.util.concurrent.ThreadPoolExecutor$Worker@3e7dd664 2> "SUITE-ICUCollationKeywordFieldMapperIT-seed#[298D86C07ECB082C]" ID=34 RUNNABLE 2> at java.management@21.0.6/sun.management.ThreadImpl.dumpThreads0(Native Method) 2> at java.management@21.0.6/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:518) 2> at java.management@21.0.6/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:506) 2> at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:681) 2> at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65) 2> at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415) 2> - locked java.lang.Object@71ea5d1b 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:708) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:629) 2> "JNA Cleaner" ID=37 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5a505060 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5a505060 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base@21.0.6/java.lang.ref.ReferenceQueue.await(ReferenceQueue.java:67) 2> at java.base@21.0.6/java.lang.ref.ReferenceQueue.remove0(ReferenceQueue.java:158) 2> at java.base@21.0.6/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:234) 2> at app//com.sun.jna.internal.Cleaner$1.run(Cleaner.java:58) 2> "TEST-ICUCollationKeywordFieldMapperIT.testNormalization-seed#[298D86C07ECB082C]" ID=38 RUNNABLE 2> at app//com.carrotsearch.randomizedtesting.AssertingRandom.nextInt(AssertingRandom.java) 2> at app//org.opensearch.test.client.RandomizingClient.(RandomizingClient.java:65) 2> at app//org.opensearch.test.OpenSearchTestClusterRule.clientForNode(OpenSearchTestClusterRule.java:133) 2> at app//org.opensearch.test.OpenSearchIntegTestCase.client(OpenSearchIntegTestCase.java:594) 2> at app//org.opensearch.test.OpenSearchIntegTestCase.client(OpenSearchIntegTestCase.java:590) 2> at app//org.opensearch.test.OpenSearchIntegTestCase.indexRandomForMultipleSlices(OpenSearchIntegTestCase.java:1693) 2> at app//org.opensearch.test.OpenSearchIntegTestCase.indexRandom(OpenSearchIntegTestCase.java:1670) 2> at app//org.opensearch.test.OpenSearchIntegTestCase.indexRandom(OpenSearchIntegTestCase.java:1565) 2> at app//org.opensearch.test.OpenSearchIntegTestCase.indexRandom(OpenSearchIntegTestCase.java:1549) 2> at app//org.opensearch.test.OpenSearchIntegTestCase.indexRandom(OpenSearchIntegTestCase.java:1526) 2> at app//org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT.testNormalization(ICUCollationKeywordFieldMapperIT.java:202) 2> at java.base@21.0.6/java.lang.invoke.LambdaForm$DMH/0x00007b49d8141400.invokeVirtual(LambdaForm$DMH) 2> at java.base@21.0.6/java.lang.invoke.LambdaForm$MH/0x00007b49d80f4800.invoke(LambdaForm$MH) 2> at java.base@21.0.6/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder) 2> at java.base@21.0.6/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:153) 2> at java.base@21.0.6/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) 2> at java.base@21.0.6/java.lang.reflect.Method.invoke(Method.java:580) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) 2> at app//org.opensearch.test.OpenSearchTestClusterRule$1.evaluate(OpenSearchTestClusterRule.java:369) 2> at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at app//org.junit.rules.RunRules.evaluate(RunRules.java:20) 2> at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) 2> at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) 2> at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) 2> at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) 2> at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) 2> at app//org.junit.rules.RunRules.evaluate(RunRules.java:20) 2> at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) 2> at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) 2> at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) 2> at app//com.carrotsearch.randomize 2> dtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) 2> at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) 2> at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) 2> at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) 2> at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) 2> at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) 2> at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) 2> at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) 2> at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) 2> at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) 2> at app//org.junit.rules.RunRules.evaluate(RunRules.java:20) 2> at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s0][[timer]]" ID=41 TIMED_WAITING 2> at java.base@21.0.6/java.lang.Thread.sleep0(Native Method) 2> at java.base@21.0.6/java.lang.Thread.sleep(Thread.java:509) 2> at app//org.opensearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:852) 2> "opensearch[node_s0][scheduler][T#1]" ID=42 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4a3e784e 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4a3e784e 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1763) 2> at java.base@21.0.6/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) 2> at java.base@21.0.6/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s0][generic][T#1]" ID=43 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s0][generic][T#2]" ID=44 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s0][generic][T#3]" ID=45 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s0][generic][T#4]" ID=46 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s1][[timer]]" ID=47 2> TIMED_WAITING 2> at java.base@21.0.6/java.lang.Thread.sleep0(Native Method) 2> at java.base@21.0.6/java.lang.Thread.sleep(Thread.java:509) 2> at app//org.opensearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:852) 2> "opensearch[node_s1][scheduler][T#1]" ID=48 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@508b1a6 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@508b1a6 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1763) 2> at java.base@21.0.6/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) 2> at java.base@21.0.6/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s1][generic][T#1]" ID=49 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s2][[timer]]" ID=50 TIMED_WAITING 2> at java.base@21.0.6/java.lang.Thread.sleep0(Native Method) 2> at java.base@21.0.6/java.lang.Thread.sleep(Thread.java:509) 2> at app//org.opensearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:852) 2> "opensearch[node_s2][scheduler][T#1]" ID=51 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6446cdc1 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6446cdc1 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1763) 2> at java.base@21.0.6/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) 2> at java.base@21.0.6/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s1][generic][T#2]" ID=52 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s2][generic][T#1]" ID=53 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@b97257e 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@b97257e 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s1][generic][T#3]" ID=54 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][[timer]]" ID=55 TIMED_WAITING 2> at java.base@21.0.6/java.lang.Thread.sleep0(Native Method) 2> at java.base@21.0.6/java.lang.Thread.sleep(Thread.java:509) 2> at app//org.opensearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:852) 2> "opensearch[node_s3][scheduler][T#1]" ID=56 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4bd2202b 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4bd2202b 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269) 2> at java.base@21.0.6/java.util.concurrent 2> .locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1763) 2> at java.base@21.0.6/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) 2> at java.base@21.0.6/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s2][generic][T#2]" ID=57 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@b97257e 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@b97257e 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s1][generic][T#4]" ID=58 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][generic][T#1]" ID=59 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][[timer]]" ID=60 TIMED_WAITING 2> at java.base@21.0.6/java.lang.Thread.sleep0(Native Method) 2> at java.base@21.0.6/java.lang.Thread.sleep(Thread.java:509) 2> at app//org.opensearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:852) 2> "opensearch[node_s4][scheduler][T#1]" ID=61 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d7edd08 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d7edd08 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1763) 2> at java.base@21.0.6/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) 2> at java.base@21.0.6/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s2][generic][T#3]" ID=62 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@b97257e 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@b97257e 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][generic][T#2]" ID=63 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s2][generic][T#4]" ID=64 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@b97257e 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed wai 2> ting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@b97257e 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s5][[timer]]" ID=65 TIMED_WAITING 2> at java.base@21.0.6/java.lang.Thread.sleep0(Native Method) 2> at java.base@21.0.6/java.lang.Thread.sleep(Thread.java:509) 2> at app//org.opensearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:852) 2> "opensearch[node_s5][scheduler][T#1]" ID=66 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6f70af0c 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6f70af0c 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1763) 2> at java.base@21.0.6/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) 2> at java.base@21.0.6/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][generic][T#1]" ID=67 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][generic][T#3]" ID=68 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][generic][T#2]" ID=69 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s5][generic][T#1]" ID=76 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@1bdcd523 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@1bdcd523 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][generic][T#4]" ID=77 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s5][generic][T#2]" ID=78 TIMED_WAITING on org.opensearch.common.util.concu 2> rrent.OpenSearchExecutors$ExecutorScalingQueue@1bdcd523 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@1bdcd523 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][generic][T#3]" ID=79 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s5][generic][T#3]" ID=80 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@1bdcd523 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@1bdcd523 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][generic][T#4]" ID=81 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s5][generic][T#4]" ID=82 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@1bdcd523 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@1bdcd523 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s1][transport_worker][T#1]" ID=84 RUNNABLE 2> at java.base@21.0.6/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base@21.0.6/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> - locked sun.nio.ch.Util$2@3061ddcd 2> - locked sun.nio.ch.EPollSelectorImpl@5a831c02 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at app//org.opensearch.nio.NioSelectorGroup$$Lambda/0x00007b49d8853b20.run(Unknown Source) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> Locked synchronizers: 2> - java.util.concurrent.locks.ReentrantLock$NonfairSync@27ca3262 2> "opensearch[node_s2][transport_worker][T#1]" ID=85 RUNNABLE 2> at java.base@21.0.6/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base@21.0.6/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> - locked sun.nio.ch.Util$2@5a428e75 2> - locked sun.nio.ch.EPollSelectorImpl@64f3af45 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at app//org.opensearch.nio.NioSelectorGroup$$Lambda/0x00007b49d8853b20.run(Unknown Source) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> Locked synchronizers: 2> - java.util.concurrent.locks.ReentrantLock$NonfairSync@2652dcd5 2> "opensearch[node_s4][transport_worker][T#1]" ID=83 RUNNABLE 2> at app//org.opensearch.core.common.io.stream.InputStreamStreamInput.readByte(InputStreamStreamInput.java:73) 2> at app//org.opensearch.core.common.io.stream.FilterStreamInput.readByte(FilterStreamInput.java:55) 2> at app//org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:586) 2> at app//org.opensearch.core.common.io.stream.StreamInput.readOptionalString(StreamInput.java:376) 2> at app//org.opensearch.action.index.IndexRequest.(IndexRequest.java:152) 2> at app//org.opensearch.action.DocWriteRequest.readDocumentRequest(DocWriteRequest.java:247) 2> at app//org.opensearch.action.bulk.BulkItemRequest.(BulkItemRequest.java:68) 2> at app//org.opensearch.action.bulk.BulkShardRequest.lambda$new$0( 2> BulkShardRequest.java:65) 2> at app//org.opensearch.action.bulk.BulkShardRequest$$Lambda/0x00007b49d8b03968.read(Unknown Source) 2> at app//org.opensearch.core.common.io.stream.StreamInput.readOptionalWriteable(StreamInput.java:1013) 2> at app//org.opensearch.action.bulk.BulkShardRequest.lambda$new$1(BulkShardRequest.java:65) 2> at app//org.opensearch.action.bulk.BulkShardRequest$$Lambda/0x00007b49d8b02f08.read(Unknown Source) 2> at app//org.opensearch.core.common.io.stream.StreamInput.readArray(StreamInput.java:1001) 2> at app//org.opensearch.action.bulk.BulkShardRequest.(BulkShardRequest.java:65) 2> at app//org.opensearch.action.bulk.TransportShardBulkAction$$Lambda/0x00007b49d87e7230.read(Unknown Source) 2> at app//org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:87) 2> at app//org.opensearch.transport.NativeMessageHandler.newRequest(NativeMessageHandler.java:316) 2> at app//org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:271) 2> at app//org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) 2> at app//org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 2> at app//org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 2> at app//org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 2> at app//org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 2> at app//org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler$$Lambda/0x00007b49d8921868.accept(Unknown Source) 2> at app//org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 2> at app//org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 2> at app//org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 2> at app//org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 2> at app//org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 2> at app//org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 2> at app//org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 2> at app//org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 2> at app//org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 2> at app//org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 2> at app//org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 2> at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 2> at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at app//org.opensearch.nio.NioSelectorGroup$$Lambda/0x00007b49d8853b20.run(Unknown Source) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> Locked synchronizers: 2> - java.util.concurrent.locks.ReentrantLock$NonfairSync@507da35f 2> "opensearch[node_s3][transport_worker][T#1]" ID=86 RUNNABLE 2> at java.base@21.0.6/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base@21.0.6/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> - locked sun.nio.ch.Util$2@1e82be03 2> - locked sun.nio.ch.EPollSelectorImpl@6772a311 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at app//org.opensearch.nio.NioSelectorGroup$$Lambda/0x00007b49d8853b20.run(Unknown Source) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> Locked synchronizers: 2> - java.util.concurrent.locks.ReentrantLock$NonfairSync@57767f5e 2> "opensearch[node_s0][transport_worker][T#1]" ID=87 RUNNABLE 2> at java.base@21.0.6/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base@21.0.6/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> - locked sun.nio.ch.Util$2@6aac49b4 2> - locked sun.nio.ch.EPollSelectorImpl@61558044 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at app//org.opensearch.nio.NioSelectorGroup$$Lambda/0x00007b49d8853b20.run(Unknown Source) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> Locked synchronizers: 2> - java.util.concurrent.locks.ReentrantLock$NonfairSync@134dcd54 2> "opensearch[node_s5][transport_worker][T#1]" ID=88 RUNNABLE 2> at java.base@21.0.6/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base@21.0.6/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> - locked sun.nio.ch.Util$2@7725ccec 2> - locked sun.nio.ch.EPollSelectorImpl@7f4cda55 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at app//org.opensearch.nio.NioSelectorGroup$$Lambda/0x00007b49d8853b20.run(Unknown Source) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> Locked synchronizers: 2> - java.util.concurrent.locks.ReentrantLock$NonfairSync@32d62258 2> "opensearch[node_s3][transport_worker][T#2]" ID=89 RUNNABLE 2> at java.base@21.0.6/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base@21.0.6/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> - locked sun.nio.ch.Util$2@4f8c7e3b 2> - locked sun.nio.ch.EPollSelectorImpl@1617898a 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at app//org.opensearch.nio.NioSelectorGroup$$Lambda/0x00007b49d8853b20.run(Unknown Source) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> Locked synchronizers: 2> - java.util.concurrent.locks.ReentrantLock$NonfairSync@f90676f 2> "opensearch[node_s1][transport_worker][T#2]" ID=91 RUNNABLE 2> at java.base@21.0.6/sun.nio.ch.SocketDispatcher.write0(Native Method) 2> at java.base@21.0.6/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:62) 2> at java.base@21.0.6/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:137) 2> at java.base@21.0.6/sun.nio.ch.IOUtil.write(IOUtil.java:81) 2> at java.base@21.0.6/sun.nio.ch.IOUtil.write(IOUtil.java:58) 2> at java.base@21.0.6/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:542) 2> at app//org.opensearch.nio.SocketChannelContext.flushToChannel(SocketChannelContext.java:327) 2> at app//org.opensearch.nio.BytesChannelContext.singleFlush(BytesChannelContext.java:99) 2> at app//org.opensearch.nio.BytesChannelContext.flushChannel(BytesChannelContext.java:71) 2> at app//org.opensearch.nio.EventHandler.handleWrite(EventHandler.java:172) 2> at app//org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:191) 2> at app//org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) 2> at app//org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) 2> at app//org.opensearch.nio.NioSelector.queueWrite(NioSelector.java:307) 2> at app//org.opensearch.nio.SocketChannelContext.sendMessage(SocketChannelContext.java:179) 2> at app//org.opensearch.transport.nio.MockNioTransport$MockSocketChannel.sendMessage(MockNioTransport.java:416) 2> at app//org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:83) 2> at app//org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:71) 2> at app//org.opense 2> arch.transport.TcpTransport$$Lambda/0x00007b49d86c81f0.apply(Unknown Source) 2> at app//org.opensearch.transport.TransportKeepAlive.sendPing(TransportKeepAlive.java:133) 2> at app//org.opensearch.transport.TransportKeepAlive.receiveKeepAlive(TransportKeepAlive.java:120) 2> at app//org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:124) 2> at app//org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 2> at app//org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 2> at app//org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 2> at app//org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler$$Lambda/0x00007b49d8921868.accept(Unknown Source) 2> at app//org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:132) 2> at app//org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 2> at app//org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 2> at app//org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 2> at app//org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 2> at app//org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 2> at app//org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 2> at app//org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 2> at app//org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 2> at app//org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 2> at app//org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 2> at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 2> at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at app//org.opensearch.nio.NioSelectorGroup$$Lambda/0x00007b49d8853b20.run(Unknown Source) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> Locked synchronizers: 2> - java.util.concurrent.locks.ReentrantLock$NonfairSync@c574027 2> - java.util.concurrent.locks.ReentrantLock$NonfairSync@20869a30 2> "opensearch[node_s4][transport_worker][T#2]" ID=92 RUNNABLE 2> at java.base@21.0.6/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base@21.0.6/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> - locked sun.nio.ch.Util$2@a770c73 2> - locked sun.nio.ch.EPollSelectorImpl@650b94d 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at app//org.opensearch.nio.NioSelectorGroup$$Lambda/0x00007b49d8853b20.run(Unknown Source) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> Locked synchronizers: 2> - java.util.concurrent.locks.ReentrantLock$NonfairSync@30b087d8 2> "opensearch[node_s0][transport_worker][T#2]" ID=90 RUNNABLE 2> at java.base@21.0.6/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base@21.0.6/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> - locked sun.nio.ch.Util$2@2473cce4 2> - locked sun.nio.ch.EPollSelectorImpl@16d5588d 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at app//org.opensearch.nio.NioSelectorGroup$$Lambda/0x00007b49d8853b20.run(Unknown Source) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> Locked synchronizers: 2> - java.util.concurrent.locks.ReentrantLock$NonfairSync@154c11cb 2> "opensearch[node_s2][transport_worker][T#2]" ID=93 RUNNABLE 2> at java.base@21.0.6/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base@21.0.6/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> - locked sun.nio.ch.Util$2@2176dafd 2> - locked sun.nio.ch.EPollSelectorImpl@1b5836db 2> at java.base@21.0.6/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at app//org.opensearch.nio.NioSelectorGroup$$Lambda/0x00007b49d8853b20.run(Unknown Source) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> Locked synchronizers: 2> - java.util.concurrent.locks.ReentrantLock$NonfairSync@27f5040d 2> "opensearch[node_s5][transport_worker][T#2]" ID=94 RUNNABLE 2> at java.base@21.0.6/java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2331) 2> at java.base@21.0.6/java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1075) 2> at java.base@21.0.6/java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006) 2> at app//org.opensearch.transport.nio.MockNioTransport$TransportThreadWatchdog.register(MockNioTransport.java:439) 2> at app//org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:165) 2> at app//org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 2> at app//org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 2> at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 2> at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at app//org.opensearch.nio.NioSelectorGroup$$Lambda/0x00007b49d8853b20.run(Unknown Source) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> Locked synchronizers: 2> - java.util.concurrent.locks.ReentrantLock$NonfairSync@52428b71 2> "opensearch[node_s4][generic][T#5]" ID=95 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s1][generic][T#5]" ID=98 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java 2> .util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][generic][T#5]" ID=96 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s2][generic][T#5]" ID=97 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@b97257e 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@b97257e 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s0][generic][T#5]" ID=99 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s2][generic][T#6]" ID=102 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@b97257e 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@b97257e 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s5][generic][T#5]" ID=101 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@1bdcd523 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@1bdcd523 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][generic][T#6]" ID=104 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s0][generic][T#6]" ID=103 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensea 2> rch[node_s1][generic][T#6]" ID=100 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s0][clusterApplierService#updateTask][T#1]" ID=123 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@45f412f6 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@45f412f6 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base@21.0.6/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][clusterApplierService#updateTask][T#1]" ID=125 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f401a7d 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f401a7d 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base@21.0.6/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s2][clusterApplierService#updateTask][T#1]" ID=124 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@101aa0f4 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@101aa0f4 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base@21.0.6/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s5][clusterApplierService#updateTask][T#1]" ID=127 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5645a885 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5645a885 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base@21.0.6/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][clusterApplierService#updateTask][T#1]" ID=126 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@33c37319 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@33c37319 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base@21.0.6/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s1][clusterApplierService#updateTask][T#1]" ID=128 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$Condi 2> tionObject@2c4f14ee 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2c4f14ee 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base@21.0.6/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s5][generic][T#6]" ID=129 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@1bdcd523 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@1bdcd523 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][generic][T#6]" ID=130 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][generic][T#7]" ID=131 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][generic][T#7]" ID=133 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s0][generic][T#7]" ID=132 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s5][generic][T#7]" ID=134 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@1bdcd523 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@1bdcd523 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s2][generic][T#7]" ID=135 TIMED_WAITING on org.opensearch.common.util.concurren 2> t.OpenSearchExecutors$ExecutorScalingQueue@b97257e 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@b97257e 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s5][generic][T#8]" ID=136 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@1bdcd523 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@1bdcd523 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s2][generic][T#8]" ID=137 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@b97257e 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@b97257e 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s0][generic][T#8]" ID=138 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][generic][T#8]" ID=139 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s1][generic][T#7]" ID=140 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s1][generic][T#8]" ID=141 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s1][generic][T#9]" ID=142 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$Execut 2> orScalingQueue@5d832237 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s1][clusterManagerService#updateTask][T#1]" ID=143 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@69170c93 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@69170c93 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base@21.0.6/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base@21.0.6/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][AsyncLucenePersistedState#updateTask][T#1]" ID=144 WAITING on java.util.concurrent.LinkedTransferQueue@12ed80b4 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@12ed80b4 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][AsyncLucenePersistedState#updateTask][T#1]" ID=145 WAITING on java.util.concurrent.LinkedTransferQueue@3a9e724b 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@3a9e724b 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s5][generic][T#9]" ID=149 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@1bdcd523 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@1bdcd523 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][generic][T#9]" ID=146 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][generic][T#8]" ID=148 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s0][generic][T#9]" ID=147 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/jdk.internal.misc.U 2> nsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s1][management][T#1]" ID=150 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@6e22e992 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@6e22e992 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][management][T#1]" ID=152 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@3f801351 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@3f801351 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][management][T#1]" ID=151 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4196dc35 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4196dc35 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s1][refresh][T#1]" ID=153 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@8d3e266 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@8d3e266 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][DanglingIndices#updateTask][T#1]" ID=154 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@7144e497 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@7144e497 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][DanglingIndices#updateTask][T#1]" ID=155 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4e616b75 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4e616b75 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][refresh][T#1]" ID=156 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@396161f8 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@396161f8 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.bas 2> e@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][refresh][T#1]" ID=157 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@2eb67460 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@2eb67460 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][generic][T#9]" ID=158 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][generic][T#10]" ID=159 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@35f909b1 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][management][T#2]" ID=160 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4196dc35 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4196dc35 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][management][T#2]" ID=161 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@3f801351 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@3f801351 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][write][T#1]" ID=163 WAITING on java.util.concurrent.LinkedTransferQueue@78804d7d 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@78804d7d 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][write][T#1]" ID=162 WAITING on java.util.concurrent.LinkedTransferQueue@3dffbbdb 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@3dffbbdb 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org 2> .opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][management][T#3]" ID=165 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@3f801351 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@3f801351 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][management][T#3]" ID=164 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4196dc35 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4196dc35 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][management][T#4]" ID=166 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@3f801351 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@3f801351 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][management][T#4]" ID=167 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4196dc35 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4196dc35 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][management][T#5]" ID=168 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@3f801351 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@3f801351 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][management][T#5]" ID=169 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4196dc35 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4196dc35 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][search][T#1]" ID=170 WAITING on java.util.concurrent.LinkedTransferQueue@fa3733d 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@fa3733d 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java 2> :1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][search][T#2]" ID=171 WAITING on java.util.concurrent.LinkedTransferQueue@122b13dd 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@122b13dd 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][search][T#1]" ID=172 WAITING on java.util.concurrent.LinkedTransferQueue@122b13dd 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@122b13dd 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s4][search][T#2]" ID=173 WAITING on java.util.concurrent.LinkedTransferQueue@fa3733d 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@fa3733d 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s5][search][T#1]" ID=174 WAITING on java.util.concurrent.LinkedTransferQueue@68ba4f40 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@68ba4f40 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s5][search][T#2]" ID=175 WAITING on java.util.concurrent.LinkedTransferQueue@68ba4f40 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@68ba4f40 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s5][search][T#3]" ID=176 WAITING on java.util.concurrent.LinkedTransferQueue@68ba4f40 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@68ba4f40 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s5][search][T#4]" ID=177 WAITING on java.util.concurrent.LinkedTransferQueue@68ba4f40 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@68ba4f40 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Wor 2> ker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s5][search][T#5]" ID=178 WAITING on java.util.concurrent.LinkedTransferQueue@68ba4f40 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@68ba4f40 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s3][generic][T#10]" ID=179 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@f8431fc 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s5][search][T#6]" ID=180 WAITING on java.util.concurrent.LinkedTransferQueue@68ba4f40 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@68ba4f40 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s5][search][T#7]" ID=181 WAITING on java.util.concurrent.LinkedTransferQueue@68ba4f40 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@68ba4f40 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s0][generic][T#10]" ID=182 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@262922fd 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s2][search][T#1]" ID=183 WAITING on java.util.concurrent.LinkedTransferQueue@285b10ef 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@285b10ef 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s2][search][T#2]" ID=184 WAITING on java.util.concurrent.LinkedTransferQueue@285b10ef 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@285b10ef 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node 2> _s2][search][T#3]" ID=185 WAITING on java.util.concurrent.LinkedTransferQueue@285b10ef 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@285b10ef 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s2][search][T#4]" ID=186 WAITING on java.util.concurrent.LinkedTransferQueue@285b10ef 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@285b10ef 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s2][search][T#5]" ID=187 WAITING on java.util.concurrent.LinkedTransferQueue@285b10ef 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@285b10ef 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s2][search][T#6]" ID=188 WAITING on java.util.concurrent.LinkedTransferQueue@285b10ef 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@285b10ef 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s2][search][T#7]" ID=189 WAITING on java.util.concurrent.LinkedTransferQueue@285b10ef 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.LinkedTransferQueue@285b10ef 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at app//org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s1][generic][T#10]" ID=190 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@5d832237 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> "opensearch[node_s2][generic][T#9]" ID=191 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@b97257e 2> at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method) 2> - timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@b97257e 2> at java.base@21.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base@21.0.6/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596) 2> at java.base@21.0.6/java.lang.Thread.run(Thread.java:1583) 2> ^^============================================== 1> [2025-05-08T13:52:36,602][WARN ][o.o.t.n.MockNioTransport ] [node_s1] Potentially blocked execution on network thread [opensearch[node_s1][transport_worker][T#1]] [RUNNABLE] [400 milliseconds]: 1> java.base/jdk.internal.util.ArraysSupport.signedHashCode(ArraysSupport.java:216) 1> java.base/jdk.internal.util.ArraysSupport.vectorizedHashCode(ArraysSupport.java:203) 1> java.base/java.lang.StringLatin1.hashCode(StringLatin1.java:191) 1> java.base/java.lang.String.hashCode(String.java:2369) 1> java.base/java.util.HashMap.hash(HashMap.java:338) 1> java.base/java.util.HashMap.getNode(HashMap.java:576) 1> java.base/java.util.HashMap.get(HashMap.java:564) 1> java.base/java.util.Collections$UnmodifiableMap.get(Collections.java:1656) 1> org.opensearch.transport.Transport$RequestHandlers.getHandler(Transport.java:358) 1> org.opensearch.transport.InboundAggregator.lambda$new$0(InboundAggregator.java:73) 1> org.opensearch.transport.InboundAggregator.initializeRequestState(InboundAggregator.java:217) 1> org.opensearch.transport.InboundAggregator.headerReceived(InboundAggregator.java:94) 1> org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:129) 1> org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 1> org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 1> org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> java.base/java.lang.Thread.run(Thread.java:1583) 2> REPRODUCE WITH: ./gradlew ':plugins:analysis-icu:internalClusterTest' --tests "org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT.testNormalization" -Dtests.seed=298D86C07ECB082C -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=hi-IN -Dtests.timezone=SystemV/PST8 -Druntime.java=21 2> java.lang.Exception: Test abandoned because suite timeout was reached. at __randomizedtesting.SeedInfo.seed([298D86C07ECB082C]:0) 1> [2025-05-08T13:52:39,813][INFO ][o.o.m.j.JvmGcMonitorService] [node_s5] [gc][893] overhead, spent [300ms] collecting in the last [1.1s] 2> मई ०८, २०२५ १:५२:४० अपराह्न com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks 2> WARNING: Will linger awaiting termination of 128 leaked thread(s). 1> [2025-05-08T13:52:40,251][INFO ][o.o.m.j.JvmGcMonitorService] [node_s0] [gc][892] overhead, spent [300ms] collecting in the last [1s] 1> [2025-05-08T13:52:40,252][INFO ][o.o.m.j.JvmGcMonitorService] [node_s3] [gc][896] overhead, spent [300ms] collecting in the last [1s] 1> [2025-05-08T13:52:40,266][INFO ][o.o.m.j.JvmGcMonitorService] [node_s4] [gc][896] overhead, spent [300ms] collecting in the last [1s] 1> [2025-05-08T13:52:40,253][INFO ][o.o.m.j.JvmGcMonitorService] [node_s2] [gc][895] overhead, spent [300ms] collecting in the last [1s] 2> मई ०८, २०२५ १:५२:४५ अपराह्न com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks 2> SEVERE: 128 threads leaked from SUITE scope at org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT: 2> 1) Thread[id=166, name=opensearch[node_s3][management][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 2) Thread[id=146, name=opensearch[node_s4][generic][T#9], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 3) Thread[id=55, name=opensearch[node_s3][[timer]], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/java.lang.Thread.sleep0(Native Method) 2> at java.base/java.lang.Thread.sleep(Thread.java:509) 2> at org.opensearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:852) 2> 4) Thread[id=191, name=opensearch[node_s2][generic][T#9], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 5) Thread[id=97, name=opensearch[node_s2][generic][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 6) Thread[id=51, name=opensearch[node_s2][scheduler][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1763) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 7) Thread[id=69, name=opensearch[node_s4][generic][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 8) Thread[id=43, name=opensearch[node_s0][generic][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 9) Thread[id=141, name=opensearch[node_s1][generic][T#8], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto 2> r.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 10) Thread[id=78, name=opensearch[node_s5][generic][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 11) Thread[id=68, name=opensearch[node_s3][generic][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 12) Thread[id=149, name=opensearch[node_s5][generic][T#9], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 13) Thread[id=147, name=opensearch[node_s0][generic][T#9], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 14) Thread[id=85, name=opensearch[node_s2][transport_worker][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> at java.base/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 15) Thread[id=182, name=opensearch[node_s0][generic][T#10], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 16) Thread[id=63, name=opensearch[node_s3][generic][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 17) Thread[id=81, name=opensearch[node_s4][generic][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 18) Thread[id=185, name=opensearch[node_s2][search][T#3], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(T 2> hreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 19) Thread[id=58, name=opensearch[node_s1][generic][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 20) Thread[id=89, name=opensearch[node_s3][transport_worker][T#2], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> at java.base/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 21) Thread[id=65, name=opensearch[node_s5][[timer]], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/java.lang.Thread.sleep0(Native Method) 2> at java.base/java.lang.Thread.sleep(Thread.java:509) 2> at org.opensearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:852) 2> 22) Thread[id=140, name=opensearch[node_s1][generic][T#7], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 23) Thread[id=54, name=opensearch[node_s1][generic][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 24) Thread[id=123, name=opensearch[node_s0][clusterApplierService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 25) Thread[id=102, name=opensearch[node_s2][generic][T#6], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 26) Thread[id=173, name=opensearch[node_s4][search][T#2], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 27) Thread[id=124, name=opensearch[node_s2][clusterApplierService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concur 2> rent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 28) Thread[id=165, name=opensearch[node_s3][management][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 29) Thread[id=151, name=opensearch[node_s4][management][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 30) Thread[id=161, name=opensearch[node_s3][management][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 31) Thread[id=93, name=opensearch[node_s2][transport_worker][T#2], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> at java.base/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 32) Thread[id=187, name=opensearch[node_s2][search][T#5], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 33) Thread[id=168, name=opensearch[node_s3][management][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 34) Thread[id=126, name=opensearch[node_s4][clusterApplierService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 35) Thread[id=88, name=opensearch[node_s5][transport_worker][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> at java.base/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 36) Thread[id=45, name=opensearch[node_s0][generic][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.Thr 2> eadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 37) Thread[id=177, name=opensearch[node_s5][search][T#4], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 38) Thread[id=169, name=opensearch[node_s4][management][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 39) Thread[id=87, name=opensearch[node_s0][transport_worker][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> at java.base/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 40) Thread[id=104, name=opensearch[node_s3][generic][T#6], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 41) Thread[id=135, name=opensearch[node_s2][generic][T#7], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 42) Thread[id=139, name=opensearch[node_s4][generic][T#8], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 43) Thread[id=47, name=opensearch[node_s1][[timer]], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/java.lang.Thread.sleep0(Native Method) 2> at java.base/java.lang.Thread.sleep(Thread.java:509) 2> at org.opensearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:852) 2> 44) Thread[id=59, name=opensearch[node_s3][generic][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 45) Thread[id=150, name=opensearch[node_s1][management][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 46) Thread[id=176, name=opensearch[node_s5][search][T#3], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 47) Thread[id=86, name=opensearch[node_s3][transport_worker][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> at java.base/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 48) Thread[id=130, name=opensearch[node_s4][generic][T#6], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 49) Thread[id=162, name=opensearch[node_s3][write][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 50) Thread[id=80, name=opensearch[node_s5][generic][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 51) Thread[id=175, name=opensearch[node_s5][search][T#2], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 52) Thread[id=183, name=opensearch[node_s2][search][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 53) Thread[id=188, name=opensearch[node_s2][search][T#6], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 54) Thread[id=125, name=opensearch[node_s3][clusterApplierService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base/java.util.concurrent.loc 2> ks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 55) Thread[id=128, name=opensearch[node_s1][clusterApplierService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 56) Thread[id=167, name=opensearch[node_s4][management][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 57) Thread[id=164, name=opensearch[node_s4][management][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 58) Thread[id=90, name=opensearch[node_s0][transport_worker][T#2], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> at java.base/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 59) Thread[id=50, name=opensearch[node_s2][[timer]], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/java.lang.Thread.sleep0(Native Method) 2> at java.base/java.lang.Thread.sleep(Thread.java:509) 2> at org.opensearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:852) 2> 60) Thread[id=134, name=opensearch[node_s5][generic][T#7], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 61) Thread[id=178, name=opensearch[node_s5][search][T#5], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 62) Thread[id=91, name=opensearch[node_s1][transport_worker][T#2], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> at java.base/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 63) Thread[id=142, name=opensearch[node_s1][generic][T#9], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread. 2> run(Thread.java:1583) 2> 64) Thread[id=155, name=opensearch[node_s4][DanglingIndices#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 65) Thread[id=62, name=opensearch[node_s2][generic][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 66) Thread[id=131, name=opensearch[node_s3][generic][T#7], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 67) Thread[id=76, name=opensearch[node_s5][generic][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 68) Thread[id=77, name=opensearch[node_s3][generic][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 69) Thread[id=49, name=opensearch[node_s1][generic][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 70) Thread[id=95, name=opensearch[node_s4][generic][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 71) Thread[id=96, name=opensearch[node_s3][generic][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 72) Thread[id=163, name=opensearch[node_s4][write][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at 2> org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 73) Thread[id=158, name=opensearch[node_s3][generic][T#9], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 74) Thread[id=100, name=opensearch[node_s1][generic][T#6], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 75) Thread[id=190, name=opensearch[node_s1][generic][T#10], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 76) Thread[id=38, name=TEST-ICUCollationKeywordFieldMapperIT.testNormalization-seed#[298D86C07ECB082C], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:754) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1099) 2> at org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:272) 2> at org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:104) 2> at org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:74) 2> at org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:55) 2> at org.opensearch.test.OpenSearchIntegTestCase.refresh(OpenSearchIntegTestCase.java:1431) 2> at org.opensearch.test.OpenSearchIntegTestCase.indexRandomForMultipleSlices(OpenSearchIntegTestCase.java:1717) 2> at org.opensearch.test.OpenSearchIntegTestCase.indexRandom(OpenSearchIntegTestCase.java:1670) 2> at org.opensearch.test.OpenSearchIntegTestCase.indexRandom(OpenSearchIntegTestCase.java:1565) 2> at org.opensearch.test.OpenSearchIntegTestCase.indexRandom(OpenSearchIntegTestCase.java:1549) 2> at org.opensearch.test.OpenSearchIntegTestCase.indexRandom(OpenSearchIntegTestCase.java:1526) 2> at org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT.testNormalization(ICUCollationKeywordFieldMapperIT.java:202) 2> at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) 2> at java.base/java.lang.reflect.Method.invoke(Method.java:580) 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) 2> at org.opensearch.test.OpenSearchTestClusterRule$1.evaluate(OpenSearchTestClusterRule.java:369) 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at org.junit.rules.RunRules.evaluate(RunRules.java:20) 2> at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) 2> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) 2> at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) 2> at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) 2> at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) 2> at org.junit.rules.RunRules.evaluate(RunRules.java:20) 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) 2> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1 2> .evaluate(TestRuleAssertionsRequired.java:53) 2> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) 2> at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) 2> at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) 2> at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) 2> at org.junit.rules.RunRules.evaluate(RunRules.java:20) 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 77) Thread[id=66, name=opensearch[node_s5][scheduler][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1763) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 78) Thread[id=56, name=opensearch[node_s3][scheduler][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1763) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 79) Thread[id=143, name=opensearch[node_s1][clusterManagerService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 80) Thread[id=184, name=opensearch[node_s2][search][T#2], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 81) Thread[id=53, name=opensearch[node_s2][generic][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 82) Thread[id=52, name=opensearch[node_s1][generic][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 83) Thread[id=92, name=opensearch[node_s4][transport_worker][T#2], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> at java.base/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 84) Thread[id=180, name=opensearch[node_s5][search][T#6], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> a 2> t java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 85) Thread[id=42, name=opensearch[node_s0][scheduler][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1763) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 86) Thread[id=67, name=opensearch[node_s4][generic][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 87) Thread[id=170, name=opensearch[node_s4][search][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 88) Thread[id=171, name=opensearch[node_s3][search][T#2], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 89) Thread[id=60, name=opensearch[node_s4][[timer]], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/java.lang.Thread.sleep0(Native Method) 2> at java.base/java.lang.Thread.sleep(Thread.java:509) 2> at org.opensearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:852) 2> 90) Thread[id=172, name=opensearch[node_s3][search][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 91) Thread[id=154, name=opensearch[node_s3][DanglingIndices#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 92) Thread[id=133, name=opensearch[node_s4][generic][T#7], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 93) Thread[id=160, name=opensearch[node_s4][management][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java 2> .base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 94) Thread[id=144, name=opensearch[node_s4][AsyncLucenePersistedState#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 95) Thread[id=152, name=opensearch[node_s3][management][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 96) Thread[id=156, name=opensearch[node_s3][refresh][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/sun.nio.fs.UnixPath.getName(UnixPath.java:51) 2> at java.base/java.io.FilePermission.containsPath(FilePermission.java:744) 2> at java.base/java.io.FilePermission.impliesIgnoreMask(FilePermission.java:611) 2> at java.base/java.io.FilePermissionCollection.implies(FilePermission.java:1196) 2> at java.base/java.security.Permissions.implies(Permissions.java:179) 2> at org.opensearch.bootstrap.OpenSearchPolicy.implies(OpenSearchPolicy.java:134) 2> at org.opensearch.bootstrap.BootstrapForTesting$1.implies(BootstrapForTesting.java:174) 2> at java.base/java.security.ProtectionDomain.implies(ProtectionDomain.java:330) 2> at java.base/java.security.ProtectionDomain.impliesWithAltFilePerm(ProtectionDomain.java:362) 2> at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:466) 2> at java.base/java.security.AccessController.checkPermission(AccessController.java:1071) 2> at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411) 2> at java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:742) 2> at java.base/sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:246) 2> at java.base/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:133) 2> at java.base/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:146) 2> at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:222) 2> at java.base/java.nio.channels.FileChannel.open(FileChannel.java:309) 2> at java.base/java.nio.channels.FileChannel.open(FileChannel.java:369) 2> at org.apache.lucene.store.MemorySegmentIndexInputProvider.openInput(MemorySegmentIndexInputProvider.java:70) 2> at org.apache.lucene.store.MemorySegmentIndexInputProvider.openInput(MemorySegmentIndexInputProvider.java:33) 2> at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:394) 2> at org.apache.lucene.tests.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:825) 2> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101) 2> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101) 2> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101) 2> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101) 2> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101) 2> at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:156) 2> at org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat.writeCompoundFile(Lucene90CompoundFormat.java:144) 2> at org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat.write(Lucene90CompoundFormat.java:100) 2> at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5890) 2> at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:599) 2> at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:527) 2> at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502) 2> at org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456) 2> at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:649) 2> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:579) 2> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) 2> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) 2> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) 2> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) 2> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) 2> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:72) 2> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:52) 2> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) 2> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) 2> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:434) 2> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:414) 2> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) 2> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) 2> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1887) 2> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1864) 2> at org.opensearch.index.shard.IndexShard.refresh(IndexShard.java:1407) 2> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.lambda$shardOperationOnPrimary$0(TransportShardRefreshAction.java:101) 2> at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344) 2> at org.opensearch.action.admi 2> n.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:100) 2> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:57) 2> at org.opensearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1333) 2> at org.opensearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:150) 2> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:654) 2> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:547) 2> at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) 2> at org.opensearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$36(IndexShard.java:4194) 2> at org.opensearch.core.action.ActionListener$3.onResponse(ActionListener.java:132) 2> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:310) 2> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:255) 2> at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:4165) 2> at org.opensearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1262) 2> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:544) 2> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) 2> at org.opensearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:483) 2> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) 2> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) 2> at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1071) 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) 2> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 97) Thread[id=136, name=opensearch[node_s5][generic][T#8], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 98) Thread[id=101, name=opensearch[node_s5][generic][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 99) Thread[id=57, name=opensearch[node_s2][generic][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 100) Thread[id=48, name=opensearch[node_s1][scheduler][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1763) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 101) Thread[id=103, name=opensearch[node_s0][generic][T#6], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 102) Thread[id=181, name=opensearch[node_s5][search][T#7], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.ta 2> ke(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 103) Thread[id=157, name=opensearch[node_s4][refresh][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/sun.nio.fs.UnixPath.getNameCount(UnixPath.java:303) 2> at java.base/java.io.FilePermission.containsPath(FilePermission.java:739) 2> at java.base/java.io.FilePermission.impliesIgnoreMask(FilePermission.java:611) 2> at java.base/java.io.FilePermissionCollection.implies(FilePermission.java:1196) 2> at java.base/java.security.Permissions.implies(Permissions.java:179) 2> at java.base/sun.security.provider.PolicyFile.implies(PolicyFile.java:1006) 2> at java.base/sun.security.provider.PolicySpiFile.engineImplies(PolicySpiFile.java:76) 2> at java.base/java.security.Policy$PolicyDelegate.implies(Policy.java:820) 2> at org.opensearch.bootstrap.OpenSearchPolicy.implies(OpenSearchPolicy.java:134) 2> at org.opensearch.bootstrap.BootstrapForTesting$1.implies(BootstrapForTesting.java:174) 2> at java.base/java.security.ProtectionDomain.implies(ProtectionDomain.java:330) 2> at java.base/java.security.ProtectionDomain.impliesWithAltFilePerm(ProtectionDomain.java:362) 2> at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:466) 2> at java.base/java.security.AccessController.checkPermission(AccessController.java:1071) 2> at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411) 2> at java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:742) 2> at java.base/sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:246) 2> at java.base/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:133) 2> at java.base/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:146) 2> at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:222) 2> at java.base/java.nio.channels.FileChannel.open(FileChannel.java:309) 2> at java.base/java.nio.channels.FileChannel.open(FileChannel.java:369) 2> at org.apache.lucene.store.MemorySegmentIndexInputProvider.openInput(MemorySegmentIndexInputProvider.java:70) 2> at org.apache.lucene.store.MemorySegmentIndexInputProvider.openInput(MemorySegmentIndexInputProvider.java:33) 2> at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:394) 2> at org.apache.lucene.tests.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:3088) 2> at org.apache.lucene.tests.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:800) 2> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101) 2> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101) 2> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101) 2> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101) 2> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101) 2> at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:156) 2> at org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat.writeCompoundFile(Lucene90CompoundFormat.java:144) 2> at org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat.write(Lucene90CompoundFormat.java:100) 2> at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5890) 2> at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:599) 2> at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:527) 2> at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502) 2> at org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456) 2> at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:649) 2> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:579) 2> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) 2> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) 2> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) 2> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) 2> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) 2> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:72) 2> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:52) 2> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) 2> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) 2> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:434) 2> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:414) 2> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) 2> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) 2> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1887) 2> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1864) 2> at org.opensearch.index.shard.IndexShard.refresh(IndexShard.java:1407) 2> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.lambda$shardOperationOnPrimary$0(TransportShardRefreshAction.java:101) 2> at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344) 2> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:100) 2> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:57) 2> at org.opensearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1333) 2> at org.opensearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:150) 2> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:654) 2> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:547) 2> at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) 2> at org.opensearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$36(IndexShard.java:4194) 2> at org.opensearch.core.action.ActionListener$3.onResponse(ActionListener.java:132) 2> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:310) 2> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:255) 2> at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:4165) 2> at org.opensearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1262) 2> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:544) 2> at org.opensearch.common.util.concurrent.AbstractRunnable.run( 2> AbstractRunnable.java:52) 2> at org.opensearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:483) 2> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) 2> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) 2> at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1071) 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) 2> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 104) Thread[id=64, name=opensearch[node_s2][generic][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 105) Thread[id=99, name=opensearch[node_s0][generic][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 106) Thread[id=82, name=opensearch[node_s5][generic][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 107) Thread[id=186, name=opensearch[node_s2][search][T#4], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 108) Thread[id=148, name=opensearch[node_s3][generic][T#8], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 109) Thread[id=46, name=opensearch[node_s0][generic][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 110) Thread[id=179, name=opensearch[node_s3][generic][T#10], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 111) Thread[id=153, name=opensearch[node_s1][refresh][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransfe 2> rQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 112) Thread[id=174, name=opensearch[node_s5][search][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 113) Thread[id=129, name=opensearch[node_s5][generic][T#6], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 114) Thread[id=61, name=opensearch[node_s4][scheduler][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1763) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 115) Thread[id=127, name=opensearch[node_s5][clusterApplierService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 116) Thread[id=138, name=opensearch[node_s0][generic][T#8], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 117) Thread[id=94, name=opensearch[node_s5][transport_worker][T#2], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> at java.base/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 118) Thread[id=132, name=opensearch[node_s0][generic][T#7], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 119) Thread[id=137, name=opensearch[node_s2][generic][T#8], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread 2> .run(Thread.java:1583) 2> 120) Thread[id=79, name=opensearch[node_s4][generic][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 121) Thread[id=44, name=opensearch[node_s0][generic][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 122) Thread[id=84, name=opensearch[node_s1][transport_worker][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> at java.base/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 123) Thread[id=98, name=opensearch[node_s1][generic][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 124) Thread[id=41, name=opensearch[node_s0][[timer]], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/java.lang.Thread.sleep0(Native Method) 2> at java.base/java.lang.Thread.sleep(Thread.java:509) 2> at org.opensearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:852) 2> 125) Thread[id=83, name=opensearch[node_s4][transport_worker][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> at java.base/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 126) Thread[id=159, name=opensearch[node_s4][generic][T#10], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 127) Thread[id=189, name=opensearch[node_s2][search][T#7], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 128) Thread[id=145, name=opensearch[node_s3][AsyncLucenePersistedState#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> मई ०८, २०२५ १:५२:४६ अपराह्न com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll 2> INFO: Starting to interrupt leaked threads: 2> 1) Thread[id=166, name=opensearch[node_s3][management][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 2) Thread[id=146, name=opensearch[node_s4][generic][T#9], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 3) Thread[id=55, name=opensearch[node_s3][[timer]], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 4) Thread[id=191, name=opensearch[node_s2][generic][T#9], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 5) Thread[id=97, name=opensearch[node_s2][generic][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 6) Thread[id=51, name=opensearch[node_s2][scheduler][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 7) Thread[id=69, name=opensearch[node_s4][generic][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 8) Thread[id=43, name=opensearch[node_s0][generic][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 9) Thread[id=141, name=opensearch[node_s1][generic][T#8], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 10) Thread[id=78, name=opensearch[node_s5][generic][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 11) Thread[id=68, name=opensearch[node_s3][generic][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 12) Thread[id=149, name=opensearch[node_s5][generic][T#9], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 13) Thread[id=147, name=opensearch[node_s0][generic][T#9], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 14) Thread[id=85, name=opensearch[node_s2][transport_worker][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 15) Thread[id=182, name=opensearch[node_s0][generic][T#10], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 16) Thread[id=63, name=opensearch[node_s3][generic][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 17) Thread[id=81, name=opensearch[node_s4][generic][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 18) Thread[id=185, name=opensearch[node_s2][search][T#3], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 19) Thread[id=58, name=opensearch[node_s1][generic][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 20) Thread[id=89, name=opensearch[node_s3][transport_worker][T#2], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 21) Thread[id=65, name=opensearch[node_s5][[timer]], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 22) Thread[id=140, name=opensearch[node_s1][generic][T#7], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 23) Thread[id=54, name=opensearch[node_s1][generic][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 24) Thread[id=123, name=opensearch[node_s0][clusterApplierService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 25) Thread[id=102, name=opensearch[node_s2][generic][T#6], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 26) Thread[id=173, name=opensearch[node_s4][search][T#2], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 27) Thread[id=124, name=opensearch[node_s2][clusterApplierService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 28) Thread[id=165, name=opensearch[node_s3][management][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 29) Thread[id=151, name=opensearch[node_s4][management][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 30) Thread[id=161, name=opensearch[node_s3][management][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 31) Thread[id=93, name=opensearch[node_s2][transport_worker][T#2], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 32) Thread[id=187, name=opensearch[node_s2][search][T#5], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 33) Thread[id=168, name=opensearch[node_s3][management][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 34) Thread[id=126, name=opensearch[node_s4][clusterApplierService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 35) Thread[id=88, name=opensearch[node_s5][transport_worker][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 36) Thread[id=45, name=opensearch[node_s0][generic][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 37) Thread[id=177, name=opensearch[node_s5][search][T#4], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 38) Thread[id=169, name=opensearch[node_s4][management][T#5], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 39) Thread[id=87, name=opensearch[node_s0][transport_worker][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 40) Thread[id=104, name=opensearch[node_s3][generic][T#6], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 41) Thread[id=135, name=opensearch[node_s2][generic][T#7], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 42) Thread[id=139, name=opensearch[node_s4][generic][T#8], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 43) Thread[id=47, name=opensearch[node_s1][[timer]], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 44) Thread[id=59, name=opensearch[node_s3][generic][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 45) Thread[id=150, name=opensearch[node_s1][management][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 46) Thread[id=176, name=opensearch[node_s5][search][T#3], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 47) Thread[id=86, name=opensearch[node_s3][transport_worker][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 48) Thread[id=130, name=opensearch[node_s4][generic][T#6], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 49) Thread[id=162, name=opensearch[node_s3][write][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 50) Thread[id=80, name=opensearch[node_s5][generic][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 51) Thread[id=175, name=opensearch[node_s5][search][T#2], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 52) Thread[id=183, name=opensearch[node_s2][search][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 53) Thread[id=188, name=opensearch[node_s2][search][T#6], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 54) Thread[id=125, name=opensearch[node_s3][clusterApplierService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 55) Thread[id=128, name=opensearch[node_s1][clusterApplierService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 56) Thread[id=167, name=opensearch[node_s4][management][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 57) Thread[id=164, name=opensearch[node_s4][management][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 58) Thread[id=90, name=opensearch[node_s0][transport_worker][T#2], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 59) Thread[id=50, name=opensearch[node_s2][[timer]], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 60) Thread[id=134, name=opensearch[node_s5][generic][T#7], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 61) Thread[id=178, name=opensearch[node_s5][search][T#5], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 62) Thread[id=91, name=opensearch[node_s1][transport_worker][T#2], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 63) Thread[id=142, name=opensearch[node_s1][generic][T#9], state=TIMED_WAITING, group=TGRP-ICUCol 2> lationKeywordFieldMapperIT] 2> 64) Thread[id=155, name=opensearch[node_s4][DanglingIndices#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 65) Thread[id=62, name=opensearch[node_s2][generic][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 66) Thread[id=131, name=opensearch[node_s3][generic][T#7], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 67) Thread[id=76, name=opensearch[node_s5][generic][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 68) Thread[id=77, name=opensearch[node_s3][generic][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 69) Thread[id=49, name=opensearch[node_s1][generic][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 70) Thread[id=95, name=opensearch[node_s4][generic][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 71) Thread[id=96, name=opensearch[node_s3][generic][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 72) Thread[id=163, name=opensearch[node_s4][write][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 73) Thread[id=158, name=opensearch[node_s3][generic][T#9], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 74) Thread[id=100, name=opensearch[node_s1][generic][T#6], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 75) Thread[id=190, name=opensearch[node_s1][generic][T#10], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 76) Thread[id=38, name=TEST-ICUCollationKeywordFieldMapperIT.testNormalization-seed#[298D86C07ECB082C], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 77) Thread[id=66, name=opensearch[node_s5][scheduler][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 78) Thread[id=56, name=opensearch[node_s3][scheduler][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 79) Thread[id=143, name=opensearch[node_s1][clusterManagerService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 80) Thread[id=184, name=opensearch[node_s2][search][T#2], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 81) Thread[id=53, name=opensearch[node_s2][generic][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 82) Thread[id=52, name=opensearch[node_s1][generic][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 83) Thread[id=92, name=opensearch[node_s4][transport_worker][T#2], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 84) Thread[id=180, name=opensearch[node_s5][search][T#6], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 85) Thread[id=42, name=opensearch[node_s0][scheduler][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 86) Thread[id=67, name=opensearch[node_s4][generic][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 87) Thread[id=170, name=opensearch[node_s4][search][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 88) Thread[id=171, name=opensearch[node_s3][search][T#2], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 89) Thread[id=60, name=opensearch[node_s4][[timer]], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 90) Thread[id=172, name=opensearch[node_s3][search][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 91) Thread[id=154, name=opensearch[node_s3][DanglingIndices#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 92) Thread[id=133, name=opensearch[node_s4][generic][T#7], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 93) Thread[id=160, name=opensearch[node_s4][management][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 94) Thread[id=144, name=opensearch[node_s4][AsyncLucenePersistedState#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 95) Thread[id=152, name=opensearch[node_s3][management][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 96) Thread[id=156, name=opensearch[node_s3][refresh][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 97) Thread[id=136, name=opensearch[node_s5][generic][T#8], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 98) Thread[id=101, name=opensearch[node_s5][generic][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 99) Thread[id=57, name=opensearch[node_s2][generic][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 100) Thread[id=48, name=opensearch[node_s1][scheduler][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 101) Thread[id=103, name=opensearch[node_s0][generic][T#6], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 102) Thread[id=181, name=opensearch[node_s5][search][T#7], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 103) Thread[id=157, name=opensearch[node_s4][refresh][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 104) Thread[id=64, name=opensearch[node_s2][generic][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 105) Thread[id=99, name=opensearch[node_s0][generic][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 106) Thread[id=82, name=opensearch[node_s5][generic][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 107) Thread[id=186, name=opensearch[node_s2][search][T#4], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 108) Thread[id=148, name=opensearch[node_s3][generic][T#8], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 109) Thread[id=46, name=opensearch[node_s0][generic][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 110) Thread[id=179, name=opensearch[node_s3][generic][T#10], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 111) Thread[id=153, name=opensearch[node_s1][refresh][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 112) Thread[id=174, name=opensearch[node_s5][search][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 113) Thread[id=129, name=opensearch[node_s5][generic][T#6], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 114) Thread[id=61, name=opensearch[node_s4][scheduler][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 115) Thread[id=127, name=opensearch[node_s5][clusterApplierService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 116) Thread[id=138, name=opensearch[node_s0][generic][T#8], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 117) Thread[id=94, name=opensearch[node_s5][transport_worker][T#2], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 118) Thread[id=132, name=opensearch[node_s0][generic][T#7], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 119) Thread[id=137, name=opensearch[node_s2][generic][T#8], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 120) Thread[id=79, name=opensearch[node_s4][generic][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 121) Thread[id=44, name=opensearch[node_s0][generic][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 122) Thread[id=84, name=opensearch[node_s1][transport_worker][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 123) Thread[id=98, name=opensearch[node_s1][generic][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 124) Thread[id=41, name=opensearch[node_s0][[timer]], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 125) Thread[id=83, name=opensearch[node_s4][transport_worker][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 126) Thread[id=159, name=opensearch[node_s4][generic][T#10], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> 127) Thread[id=189, name=opensearch[node_s2][search][T#7], state=WAITING, group=TGRP-ICUCollatio 2> nKeywordFieldMapperIT] 2> 128) Thread[id=145, name=opensearch[node_s3][AsyncLucenePersistedState#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> [2025-05-08T13:52:48,917][INFO ][o.o.i.m.ICUCollationKeywordFieldMapperIT] [testNormalization] after test 1> [2025-05-08T13:52:50,484][INFO ][o.o.t.OpenSearchTestClusterRule] [testNormalization] [ICUCollationKeywordFieldMapperIT#testNormalization]: cleaning up after test 1> [2025-05-08T13:52:59,846][WARN ][o.o.e.NodeEnvironment ] [suite] lock assertion failed 1> java.nio.channels.ClosedByInterruptException: null 1> at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:200) ~[?:?] 1> at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:172) ~[?:?] 1> at java.base/sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:430) ~[?:?] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:169) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodeDataPaths(NodeEnvironment.java:911) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.InternalTestCluster$NodeAndClient.markNodeDataDirsAsPendingForWipe(InternalTestCluster.java:1147) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.InternalTestCluster$NodeAndClient.close(InternalTestCluster.java:1131) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.test.InternalTestCluster.close(InternalTestCluster.java:971) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchTestClusterRule.clearClusters(OpenSearchTestClusterRule.java:339) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchTestClusterRule.afterClass(OpenSearchTestClusterRule.java:86) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchIntegTestCase.afterClass(OpenSearchIntegTestCase.java:2365) ~[framework-2.19.1.jar:2.19.1] 1> at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?] 1> at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:901) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.junit.rules.RunRules.evaluate(RunRules.java:20) ~[junit-4.13.2.jar:4.13.2] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:01,034][WARN ][o.o.t.ThreadPool ] [node_s4] failed to run scheduled task [org.opensearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker@5622b19a] on thread pool [same] 1> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed 1> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:917) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:930) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.getFlushingBytes(IndexWriter.java:804) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine.getWritingBytes(InternalEngine.java:649) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.getWritingBytes(IndexShard.java:1418) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.IndexingMemoryController.getShardWritingBytes(IndexingMemoryController.java:213) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker.runUnlocked(IndexingMemoryController.java:346) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:326) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.nio.channels.ClosedByInterruptException 1> at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:200) ~[?:?] 1> at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:172) ~[?:?] 1> at java.base/sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:430) ~[?:?] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:169) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:42) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:41) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.codecs.lucene94.Lucene94FieldInfosFormat.write(Lucene94FieldInfosFormat.java:377) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:350) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:445) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:649) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:579) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:72) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:434) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:414) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1887) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1864) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.refresh(IndexShard.java:1407) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.lambda$shardOperationOnPrimary$0(TransportShardRefreshAction.java:101) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.sh 1> ardOperationOnPrimary(TransportShardRefreshAction.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:57) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1333) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:150) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:654) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:547) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$36(IndexShard.java:4194) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$3.onResponse(ActionListener.java:132) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:310) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:255) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:4165) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1262) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:544) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:483) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1071) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 3 more 1> [2025-05-08T13:53:01,042][WARN ][o.o.t.ThreadPool ] [node_s3] failed to run scheduled task [org.opensearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker@45d1c5b7] on thread pool [same] 1> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed 1> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:917) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:930) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.getFlushingBytes(IndexWriter.java:804) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine.getWritingBytes(InternalEngine.java:649) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.getWritingBytes(IndexShard.java:1418) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.IndexingMemoryController.getShardWritingBytes(IndexingMemoryController.java:213) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker.runUnlocked(IndexingMemoryController.java:346) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:326) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.nio.channels.ClosedByInterruptException 1> at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:200) ~[?:?] 1> at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:172) ~[?:?] 1> at java.base/sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:430) ~[?:?] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:169) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:42) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:41) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.codecs.lucene94.Lucene94FieldInfosFormat.write(Lucene94FieldInfosFormat.java:377) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:350) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:445) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:649) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:579) ~[lucene-core-9.12.1.jar:9 1> .12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:72) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:434) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:414) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1887) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1864) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.refresh(IndexShard.java:1407) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.lambda$shardOperationOnPrimary$0(TransportShardRefreshAction.java:101) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:57) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1333) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:150) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:654) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:547) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$36(IndexShard.java:4194) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$3.onResponse(ActionListener.java:132) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:310) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:255) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:4165) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1262) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:544) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:483) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1071) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 3 more 1> [2025-05-08T13:53:01,052][WARN ][o.o.i.e.Engine ] [node_s4] [foo][7] failed engine [refresh failed source[api]] 1> java.nio.channels.ClosedChannelException: null 1> at java.base/sun.nio.ch.FileLockImpl.release(FileLockImpl.java:58) ~[?:?] 1> at java.base/java.nio.channels.FileLock.close(FileLock.java:336) ~[?:?] 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.close(NativeFSLockFactory.java:195) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.util.IOUtils.close(IOUtils.java:85) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.util.IOUtils.close(IOUtils.java:72) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2561) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2466) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:5774) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:719) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:72) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:434) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:414) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1887) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1864) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.refresh(IndexShard.java:1407) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.lambda$shardOperationOnPrimary$0(TransportShardRefreshAction.java:101) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:57) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1333) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:150) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:654) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:547) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$36(IndexShard.java:4194) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$3.onResponse(ActionListener.java:132) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:310) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:255) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:4165) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1262) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:544) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:483) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1071) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:01,035][WARN ][o.o.i.e.Engine ] [node_s3] [foo][6] failed engine [refresh failed source[api]] 1> java.nio.channels.ClosedChannelException: null 1> at java.base/sun.nio.ch.FileLockImpl.release(FileLockImpl.java:58) ~[?:?] 1> at java.base/java.nio.channels.FileLock.close(FileLock.java:336) ~[?:?] 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.close(NativeFSLockFactory.java:195) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.util.IOUtils.close(IOUtils.java:85) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.util.IOUtils.close(IOUtils.java:72) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2561) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2466) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:5774) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:719) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:72) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:434) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:414) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1887) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1864) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.refresh(IndexShard.java:1407) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.lambda$shardOperationOnPrimary$0(TransportShardRefreshAction.java:101) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:57) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1333) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:150) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:654) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:547) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$36(IndexShard.java:4194) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$3.onResponse(ActionListener.java:132) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:310) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:255) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:4165) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1262) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:544) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:483) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1071) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:01,669][WARN ][o.o.e.NodeEnvironment ] [suite] lock assertion failed 1> java.nio.channels.ClosedByInterruptException: null 1> at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:200) ~[?:?] 1> at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:172) ~[?:?] 1> at java.base/sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:430) ~[?:?] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:169) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodeDataPaths(NodeEnvironment.java:911) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.InternalTestCluster$NodeAndClient.markNodeDataDirsAsPendingForWipe(InternalTestCluster.java:1147) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.InternalTestCluster$NodeAndClient.close(InternalTestCluster.java:1131) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.test.InternalTestCluster.close(InternalTestCluster.java:971) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchTestClusterRule.clearClusters(OpenSearchTestClusterRule.java:339) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchTestClusterRule.afterClass(OpenSearchTestClusterRule.java:86) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchIntegTestCase.afterClass(OpenSearchIntegTestCase.java:2365) ~[framework-2.19.1.jar:2.19.1] 1> at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?] 1> at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:901) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.junit.rules.RunRules.evaluate(RunRules.java:20) ~[junit-4.13.2.jar:4.13.2] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:02,426][WARN ][o.o.e.NodeEnvironment ] [suite] lock assertion failed 1> java.nio.channels.ClosedByInterruptException: null 1> at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:200) ~[?:?] 1> at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:172) ~[?:?] 1> at java.base/sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:430) ~[?:?] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:169) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodeDataPaths(NodeEnvironment.java:911) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.InternalTestCluster$NodeAndClient.markNodeDataDirsAsPendingForWipe(InternalTestCluster.java:1147) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.InternalTestCluster$NodeAndClient.close(InternalTestCluster.java:1131) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.test.InternalTestCluster.close(InternalTestCluster.java:971) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchTestClusterRule.clearClusters(OpenSearchTestClusterRule.java:339) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchTestClusterRule.afterClass(OpenSearchTestClusterRule.java:86) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchIntegTestCase.afterClass(OpenSearchIntegTestCase.java:2365) ~[framework-2.19.1.jar:2.19.1] 1> at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?] 1> at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:901) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.junit.rules.RunRules.evaluate(RunRules.java:20) ~[junit-4.13.2.jar:4.13.2] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:03,039][WARN ][o.o.e.NodeEnvironment ] [suite] lock assertion failed 1> java.nio.channels.ClosedByInterruptException: null 1> at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:200) ~[?:?] 1> at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:172) ~[?:?] 1> at java.base/sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:430) ~[?:?] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:169) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodeDataPaths(NodeEnvironment.java:911) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.InternalTestCluster$NodeAndClient.markNodeDataDirsAsPendingForWipe(InternalTestCluster.java:1147) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.InternalTestCluster$NodeAndClient.close(InternalTestCluster.java:1131) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.test.InternalTestCluster.close(InternalTestCluster.java:971) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchTestClusterRule.clearClusters(OpenSearchTestClusterRule.java:339) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchTestClusterRule.afterClass(OpenSearchTestClusterRule.java:86) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchIntegTestCase.afterClass(OpenSearchIntegTestCase.java:2365) ~[framework-2.19.1.jar:2.19.1] 1> at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?] 1> at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:901) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.junit.rules.RunRules.evaluate(RunRules.java:20) ~[junit-4.13.2.jar:4.13.2] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:03,669][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][7] start check index 1> [2025-05-08T13:53:03,772][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][6] start check index 1> [2025-05-08T13:53:03,687][WARN ][o.o.e.NodeEnvironment ] [suite] lock assertion failed 1> java.nio.channels.ClosedByInterruptException: null 1> at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:200) ~[?:?] 1> at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:172) ~[?:?] 1> at java.base/sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:430) ~[?:?] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:169) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodeDataPaths(NodeEnvironment.java:911) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.InternalTestCluster$NodeAndClient.markNodeDataDirsAsPendingForWipe(InternalTestCluster.java:1147) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.InternalTestCluster$NodeAndClient.close(InternalTestCluster.java:1131) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.test.InternalTestCluster.close(InternalTestCluster.java:971) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchTestClusterRule.clearClusters(OpenSearchTestClusterRule.java:339) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchTestClusterRule.afterClass(OpenSearchTestClusterRule.java:86) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchIntegTestCase.afterClass(OpenSearchIntegTestCase.java:2365) ~[framework-2.19.1.jar:2.19.1] 1> at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?] 1> at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:901) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.junit.rules.RunRules.evaluate(RunRules.java:20) ~[junit-4.13.2.jar:4.13.2] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:04,044][WARN ][o.o.e.NodeEnvironment ] [suite] lock assertion failed 1> java.nio.channels.ClosedByInterruptException: null 1> at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:200) ~[?:?] 1> at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:172) ~[?:?] 1> at java.base/sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:430) ~[?:?] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:169) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodeDataPaths(NodeEnvironment.java:911) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.InternalTestCluster$NodeAndClient.markNodeDataDirsAsPendingForWipe(InternalTestCluster.java:1147) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.InternalTestCluster$NodeAndClient.close(InternalTestCluster.java:1131) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.test.InternalTestCluster.close(InternalTestCluster.java:971) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchTestClusterRule.clearClusters(OpenSearchTestClusterRule.java:339) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchTestClusterRule.afterClass(OpenSearchTestClusterRule.java:86) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.OpenSearchIntegTestCase.afterClass(OpenSearchIntegTestCase.java:2365) ~[framework-2.19.1.jar:2.19.1] 1> at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?] 1> at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:901) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.junit.rules.RunRules.evaluate(RunRules.java:20) ~[junit-4.13.2.jar:4.13.2] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.7.1.jar:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:04,621][WARN ][o.o.t.ThreadPool ] [node_s1] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x00007b49d884b598@15554127] on thread pool [generic] 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s1/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:34:54.491447622Z) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:941) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:04,432][WARN ][o.o.t.ThreadPool ] [node_s2] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x00007b49d884b598@482791d3] on thread pool [generic] 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s2/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:34:59.176544704Z) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:941) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:04,456][WARN ][o.o.t.ThreadPool ] [node_s0] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x00007b49d884b598@2f4df60d] on thread pool [generic] 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s0/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:32:59.594066779Z) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:941) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2> NOTE: leaving temporary files on disk at: /build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001 1> [2025-05-08T13:53:05,477][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:43491, remoteAddress=/127.0.0.1:55722}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendResponse(NativeOutboundHandler.java:152) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:96) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.telemetry.tracing.channels.TraceableTcpTransportChannel.sendResponse(TraceableTcpTransportChannel.java:89) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:72) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:153) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:278) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:05,622][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:37340, remoteAddress=127.0.0.1/127.0.0.1:45155}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:05,956][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:55652, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:06,115][WARN ][o.o.t.s.M.Listener ] [node_s4] [foo][7] failed to check index 1> java.io.IOException: failed to check index for shard [foo][7];index files [[_0.cfe, _0.cfs, _0.si, _1.cfe, _1.cfs, _1.si, _2.fdm, _2.fdt, _2.fdx, _2.kdd, _2.kdi, _2.kdm, _2_Lucene90_0.dvd, _2_Lucene90_0.dvm, _2_Lucene912_0.doc, _2_Lucene912_0.psm, _2_Lucene912_0.tim, _2_Lucene912_0.tip, _2_Lucene912_0.tmd, segments_2, write.lock]] os [Checking index with threadCount: 12 1> ERROR: could not read latest commit point from segments file "segments_2" in directory 1> java.nio.channels.ClosedByInterruptException 1> at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:200) 1> at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:172) 1> at java.base/sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:430) 1> at org.apache.lucene.store.MemorySegmentIndexInputProvider.openInput(MemorySegmentIndexInputProvider.java:71) 1> at org.apache.lucene.store.MemorySegmentIndexInputProvider.openInput(MemorySegmentIndexInputProvider.java:33) 1> at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:394) 1> at org.apache.lucene.tests.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:3088) 1> at org.apache.lucene.tests.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:800) 1> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101) 1> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101) 1> at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:156) 1> at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:310) 1> at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:650) 1> at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:567) 1> at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:545) 1> at org.opensearch.index.store.Store.checkIndex(Store.java:489) 1> at org.opensearch.test.store.MockFSDirectoryFactory.checkIndex(MockFSDirectoryFactory.java:116) 1> at org.opensearch.test.store.MockFSIndexStore$Listener.afterIndexShardClosed(MockFSIndexStore.java:112) 1> at org.opensearch.index.CompositeIndexEventListener.afterIndexShardClosed(CompositeIndexEventListener.java:133) 1> at org.opensearch.index.IndexService.closeShard(IndexService.java:890) 1> at org.opensearch.index.IndexService.removeShard(IndexService.java:859) 1> at org.opensearch.indices.cluster.IndicesClusterStateService.failAndRemoveShard(IndicesClusterStateService.java:816) 1> at org.opensearch.indices.cluster.IndicesClusterStateService$FailedShardHandler.lambda$accept$0(IndicesClusterStateService.java:866) 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at java.base/java.lang.Thread.run(Thread.java:1583) 1> ] 1> at org.opensearch.test.store.MockFSDirectoryFactory.checkIndex(MockFSDirectoryFactory.java:125) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.store.MockFSIndexStore$Listener.afterIndexShardClosed(MockFSIndexStore.java:112) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.index.CompositeIndexEventListener.afterIndexShardClosed(CompositeIndexEventListener.java:133) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.IndexService.closeShard(IndexService.java:890) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.IndexService.removeShard(IndexService.java:859) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService.failAndRemoveShard(IndicesClusterStateService.java:816) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService$FailedShardHandler.lambda$accept$0(IndicesClusterStateService.java:866) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:05,849][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:55728, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:06,113][WARN ][o.o.t.s.M.Listener ] [node_s3] [foo][6] failed to check index 1> java.io.IOException: failed to check index for shard [foo][6];index files [[_0.cfe, _0.cfs, _0.si, _1.cfe, _1.cfs, _1.si, _2.fdm, _2.fdt, _2.fdx, _2.kdd, _2.kdi, _2.kdm, _2_Lucene90_0.dvd, _2_Lucene90_0.dvm, _2_Lucene912_0.doc, _2_Lucene912_0.psm, _2_Lucene912_0.tim, _2_Lucene912_0.tip, _2_Lucene912_0.tmd, segments_2, write.lock]] os [Checking index with threadCount: 12 1> ERROR: could not read latest commit point from segments file "segments_2" in directory 1> java.nio.channels.ClosedByInterruptException 1> at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:200) 1> at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:172) 1> at java.base/sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:430) 1> at org.apache.lucene.store.MemorySegmentIndexInputProvider.openInput(MemorySegmentIndexInputProvider.java:71) 1> at org.apache.lucene.store.MemorySegmentIndexInputProvider.openInput(MemorySegmentIndexInputProvider.java:33) 1> at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:394) 1> at org.apache.lucene.tests.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:3088) 1> at org.apache.lucene.tests.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:800) 1> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101) 1> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101) 1> at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:156) 1> at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:310) 1> at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:650) 1> at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:567) 1> at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:545) 1> at org.opensearch.index.store.Store.checkIndex(Store.java:489) 1> at org.opensearch.test.store.MockFSDirectoryFactory.checkIndex(MockFSDirectoryFactory.java:116) 1> at org.opensearch.test.store.MockFSIndexStore$Listener.afterIndexShardClosed(MockFSIndexStore.java:112) 1> at org.opensearch.index.CompositeIndexEventListener.afterIndexShardClosed(CompositeIndexEventListener.java:133) 1> at org.opensearch.index.IndexService.closeShard(IndexService.java:890) 1> at org.opensearch.index.IndexService.removeShard(IndexService.java:859) 1> at org.opensearch.indices.cluster.IndicesClusterStateService.failAndRemoveShard(IndicesClusterStateService.java:816) 1> at org.opensearch.indices.cluster.IndicesClusterStateService$FailedShardHandler.lambda$accept$0(IndicesClusterStateService.java:866) 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at java.base/java.lang.Thread.run(Thread.java:1583) 1> ] 1> at org.opensearch.test.store.MockFSDirectoryFactory.checkIndex(MockFSDirectoryFactory.java:125) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.test.store.MockFSIndexStore$Listener.afterIndexShardClosed(MockFSIndexStore.java:112) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.index.CompositeIndexEventListener.afterIndexShardClosed(CompositeIndexEventListener.java:133) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.IndexService.closeShard(IndexService.java:890) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.IndexService.removeShard(IndexService.java:859) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService.failAndRemoveShard(IndicesClusterStateService.java:816) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService$FailedShardHandler.lambda$accept$0(IndicesClusterStateService.java:866) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Th 1> readPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:05,710][WARN ][o.o.t.OutboundHandler ] [node_s5] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:55734, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:06,216][INFO ][o.o.t.s.M.Listener ] [node_s4] [foo][7] end check index 1> [2025-05-08T13:53:06,366][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][6] end check index 1> [2025-05-08T13:53:06,280][WARN ][o.o.t.TcpTransport ] [node_s0] exception caught on transport layer [NioSocketChannel{localAddress=/127.0.0.1:60292, remoteAddress=127.0.0.1/127.0.0.1:45155}], closing connection 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$MockTcpChannelFactory.lambda$createChannel$1(MockNioTransport.java:253) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.InboundChannelBuffer.ensureCapacity(InboundChannelBuffer.java:100) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.readFromChannel(SocketChannelContext.java:298) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:53) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:06,369][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:43491, remoteAddress=/127.0.0.1:55722}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendErrorResponse(NativeOutboundHandler.java:180) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:105) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.telemetry.tracing.channels.TraceableTcpTransportChannel.sendResponse(TraceableTcpTransportChannel.java:101) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.sendErrorResponse(NativeMessageHandler.java:390) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:280) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:06,560][WARN ][o.o.i.c.IndicesClusterStateService] [node_s4] [foo][7] marking and sending shard failed due to [shard failure, reason [refresh failed source[api]]] 1> java.nio.channels.ClosedChannelException: null 1> at java.base/sun.nio.ch.FileLockImpl.release(FileLockImpl.java:58) ~[?:?] 1> at java.base/java.nio.channels.FileLock.close(FileLock.java:336) ~[?:?] 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.close(NativeFSLockFactory.java:195) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.util.IOUtils.close(IOUtils.java:85) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.util.IOUtils.close(IOUtils.java:72) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2561) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2466) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:5774) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:719) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:72) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:434) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:414) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1887) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1864) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.refresh(IndexShard.java:1407) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.lambda$shardOperationOnPrimary$0(TransportShardRefreshAction.java:101) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:57) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1333) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:150) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:654) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:547) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$36(IndexShard.java:4194) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$3.onResponse(ActionListener.java:132) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:310) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:255) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:4165) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1262) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:544) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:483) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1071) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:06,553][WARN ][o.o.i.c.IndicesClusterStateService] [node_s3] [foo][6] marking and sending shard failed due to [shard failure, reason [refresh failed source[api]]] 1> java.nio.channels.ClosedChannelException: null 1> at java.base/sun.nio.ch.FileLockImpl.release(FileLockImpl.java:58) ~[?:?] 1> at java.base/java.nio.channels.FileLock.close(FileLock.java:336) ~[?:?] 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.close(NativeFSLockFactory.java:195) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.util.IOUtils.close(IOUtils.java:85) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.util.IOUtils.close(IOUtils.java:72) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2561) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2466) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:5774) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:719) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:72) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:434) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:414) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1887) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1864) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.refresh(IndexShard.java:1407) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.lambda$shardOperationOnPrimary$0(TransportShardRefreshAction.java:101) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:57) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1333) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:150) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:654) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:547) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$36(IndexShard.java:4194) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$3.onResponse(ActionListener.java:132) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:310) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:255) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:4165) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1262) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:544) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:483) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1071) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:08,946][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:55638, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:856) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.action.shard.ShardStateAction.sendShardAction(ShardStateAction.java:192) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.action.shard.ShardStateAction.localShardFailed(ShardStateAction.java:298) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService.sendFailShard(IndicesClusterStateService.java:845) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService.failAndRemoveShard(IndicesClusterStateService.java:834) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService$FailedShardHandler.lambda$accept$0(IndicesClusterStateService.java:866) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:10,099][WARN ][o.o.t.ThreadPool ] [node_s2] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x00007b49d884b598@482791d3] on thread pool [generic] 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s2/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:34:59.176544704Z) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:941) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:10,194][WARN ][o.o.t.ThreadPool ] [node_s4] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x00007b49d884b598@c30f50f] on thread pool [generic] 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s4/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:35:05.730680516Z) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:941) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:09,289][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:55730, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19. 1> 1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:856) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.action.shard.ShardStateAction.sendShardAction(ShardStateAction.java:192) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.action.shard.ShardStateAction.localShardFailed(ShardStateAction.java:298) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService.sendFailShard(IndicesClusterStateService.java:845) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService.failAndRemoveShard(IndicesClusterStateService.java:834) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService$FailedShardHandler.lambda$accept$0(IndicesClusterStateService.java:866) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:06,882][WARN ][o.o.t.NativeMessageHandler] [node_s1] Failed to send error message back to client for action [internal:coordination/fault_detection/leader_check] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendErrorResponse(NativeOutboundHandler.java:180) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:105) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.telemetry.tracing.channels.TraceableTcpTransportChannel.sendResponse(TraceableTcpTransportChannel.java:101) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.sendErrorResponse(NativeMessageHandler.java:390) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:280) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Suppressed: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendResponse(NativeOutboundHandler.java:152) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:96) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.telemetry.tracing.channels.TraceableTcpTransportChannel.sendResponse(TraceableTcpTransportChannel.java:89) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:72) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:153) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:278) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler 1> .java:126) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:10,172][WARN ][o.o.t.ThreadPool ] [node_s3] failed to run scheduled task [org.opensearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker@45d1c5b7] on thread pool [same] 1> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed 1> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:917) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:930) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.getFlushingBytes(IndexWriter.java:804) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine.getWritingBytes(InternalEngine.java:649) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.getWritingBytes(IndexShard.java:1418) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.IndexingMemoryController.getShardWritingBytes(IndexingMemoryController.java:213) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker.runUnlocked(IndexingMemoryController.java:346) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:326) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.nio.channels.ClosedByInterruptException 1> at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:200) ~[?:?] 1> at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:172) ~[?:?] 1> at java.base/sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:430) ~[?:?] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.mockfile.FilterFileChannel.size(FilterFileChannel.java:82) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:169) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:42) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:41) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.(Lucene90DocValuesConsumer.java:79) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.codecs.lucene90.Lucene90DocValuesFormat.fieldsConsumer(Lucene90DocValuesFormat.java:148) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:225) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:156) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:103) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.NumericDocValuesWriter.flush(NumericDocValuesWriter.java:111) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexingChain.writeDocValues(IndexingChain.java:424) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:282) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:445) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:649) ~[lucene-core-9.12.1.jar:9.12.1 7a97a 1> 05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:579) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:72) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:434) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:414) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1887) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1864) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.refresh(IndexShard.java:1407) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.lambda$shardOperationOnPrimary$0(TransportShardRefreshAction.java:101) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:57) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1333) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:150) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:654) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:547) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$36(IndexShard.java:4194) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$3.onResponse(ActionListener.java:132) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:310) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:255) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:4165) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1262) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:544) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:483) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1071) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 3 more 1> [2025-05-08T13:53:10,876][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:55644, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:10,907][WARN ][o.o.t.OutboundHandler ] [node_s5] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:55694, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:10,197][WARN ][o.o.i.e.Engine ] [node_s3] [foo][8] failed engine [refresh failed source[api]] 1> java.nio.channels.ClosedChannelException: null 1> at java.base/sun.nio.ch.FileLockImpl.release(FileLockImpl.java:58) ~[?:?] 1> at java.base/java.nio.channels.FileLock.close(FileLoc 1> k.java:336) ~[?:?] 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.close(NativeFSLockFactory.java:195) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.util.IOUtils.close(IOUtils.java:85) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.util.IOUtils.close(IOUtils.java:72) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2561) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2466) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:5774) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:719) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:72) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:434) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:414) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1887) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1864) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.refresh(IndexShard.java:1407) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.lambda$shardOperationOnPrimary$0(TransportShardRefreshAction.java:101) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:57) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1333) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:150) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:654) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:547) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$36(IndexShard.java:4194) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$3.onResponse(ActionListener.java:132) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:310) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:255) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:4165) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1262) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:544) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:483) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1071) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T21:53:10,219][WARN ][o.o.t.ThreadPool ] [node_s5] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x00007b49d884b598@27cc7d94] on thread pool [generic] 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s5/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:35:08.444736755Z) 1> at org.apa 1> che.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:941) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.19.1.jar:2.19.1] 2> May 08, 2025 9:53:11 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll 2> SEVERE: There are still zombie threads that couldn't be terminated: 2> 1) Thread[id=166, name=opensearch[node_s3][management][T#4], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.19.1.jar:2.19.1] 2> 2) Thread[id=146, name=opensearch[node_s4][generic][T#9], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 3) Thread[id=191, name=opensearch[node_s2][generic][T#9], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 4) Thread[id=97, name=opensearch[node_s2][generic][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 1> [2025-05-08T13:53:10,146][WARN ][o.o.t.ThreadPool ] [node_s0] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x00007b49d884b598@2f4df60d] on thread pool [generic] 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s0/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:32:59.594066779Z) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 5) Thread[id=51, name=opensearch[node_s2][scheduler][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 1> at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:941) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269) 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1763) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 6) Thread[id=69, name=opensearch[node_s4][generic][T#2], state=BLOCKED, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 2> 7) Thread[id=43, name=opensearch[node_s0][generic][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 8) Thread[id=141, name=opensearch[node_s1][generic][T#8], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 9) Thread[id=78, name=opensearch[node_s5][generic][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.a 2> wait(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 10) Thread[id=68, name=opensearch[node_s3][generic][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 11) Thread[id=149, name=opensearch[node_s5][generic][T#9], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 1> [2025-05-08T13:53:09,901][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:35194, remoteAddress=127.0.0.1/127.0.0.1:40859}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 12) Thread[id=147, name=opensearch[node_s0][generic][T#9], state=BLOCKED, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 13) Thread[id=85, name=opensearch[node_s2][transport_worker][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:282) 2> at java.base/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:140) 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> at java.base/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 2> 14) Thread[id=182, name=opensearch[node_s0][generic][T#10], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.locks.ReentrantLock$Sync.lock(ReentrantLock.java:152) 2> at java.base/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:322) 2> at java.base/jdk.internal.misc.InternalLock.lock(InternalLock.java:74) 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 2> at java.base/sun.nio.cs.StreamDecoder.close(StreamDecoder.java:249) 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.io.InputStreamReader.close(InputStreamReader.java:203) 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 2> at java.base/java.io.BufferedReader.implClose(BufferedReader.java:636) 2> at java.base/java.io.BufferedReader.close(BufferedReader.java:621) 2> at java.base/jdk.internal.platform.CgroupUtil.readStringValue(CgroupUtil.java:72) 2> at java.base/jdk.internal.platform.CgroupSubsystemController.getStringValue(CgroupSubsystemController.java:65) 2> at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.getFromCpuMax(CgroupV2Subsystem.java:140) 2> at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.getCpuQuota(CgroupV2Subsystem.java:136) 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 2> at java.base/jdk.internal.platform.CgroupMetrics.getCpuQuota(CgroupMetrics.java:70) 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 2> at jdk.management/com.sun.management.internal.OperatingSystemImpl$ContainerCpuTicks.getContainerCpuLoad(OperatingSystemImpl.java:85) 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 2> at jdk.management/com.sun.management.internal.OperatingSystemImpl.getProcessCpuLoad(OperatingSystemImpl.java:271) 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 2> at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) 2> at java.base/java.lang.reflect.Method.invoke(Method.java:580) 2> at org.opensearch.monitor.Probes.getLoadAndScaleToPercent(Probes.java:47) 2> at org.opensearch.monitor.process.ProcessProbe.getProcessCpuPercent(ProcessProbe.java:109) 2> at org.opensearch.node.resource.tracker.AverageCpuUsageTracker.getUsage(AverageCpuUsageTracker.java:33) 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2> at org.opensearch.node.resource.tracker.AbstractAverageUsageTracker.lambda$doStart$0(AbstractAverageUsageTracker.java:86) 2> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) 2> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 15) Thread[id=63, name=opensearch[node_s3][generic][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 1> [2025-05-08T13:53:12,588][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:55732, remoteAddress=127.0.0.1/127.0.0.1:43491}] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 16) Thread[id=81, name=opensearch[node_s4][generic][T#4], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransfe 2> rQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 17) Thread[id=185, name=opensearch[node_s2][search][T#3], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 2> 18) Thread[id=58, name=opensearch[node_s1][generic][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 19) Thread[id=89, name=opensearch[node_s3][transport_worker][T#2], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/sun.nio.ch.EPoll.wait(Native Method) 2> at java.base/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:121) 2> at java.base/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> at java.base/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 20) Thread[id=140, name=opensearch[node_s1][generic][T#7], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 2> 21) Thread[id=54, name=opensearch[node_s1][generic][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 22) Thread[id=123, name=opensearch[node_s0][clusterApplierService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/java.lang.Throwable.fillInStackTrace(Native Method) 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.lang.Throwable.fillInStackTrace(Throwable.java:820) 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.lang.Throwable.(Throwable.java:258) 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.lang.Exception.(Exception.java:55) 2> at java.base/java.lang.Thread.getStackTrace(Thread.java:2451) 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 2> at org.opensearch.cluster.service.ClusterApplierService.assertNotCalledFromClusterStateApplier(ClusterApplierService.java:446) 2> at org.opensearch.cluster.service.ClusterApplierService.state(ClusterApplierService.java:230) 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 2> at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.postAdded(ClusterStateObserver.java:250) 2> at org.opensearch.cluster.service.ClusterApplierService$1.run(ClusterApplierService.java:346) 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) 1> [2025-05-08T13:53:12,393][WARN ][o.o.c.a.s.ShardStateAction] [node_s3] unexpected failure while sending request [internal:cluster/shard/failure] to [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}] for shard entry [shard id [[foo][6]], allocation id [9R9VaCUsRfyImBXANX4Rhg], primary term [0], message [shard failure, reason [refresh failed source[api]]], failure [ClosedChannelException[null]], markAsStale [true]] 2> at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:283) 2> at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:246) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> org.opensearch.transport.SendRequestTransportException: [node_s1][127.0.0.1:43491][internal:cluster/shard/failure] 2> 23) Thread[id=102, name=opensearch[node_s2][generic][T#6], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/jdk.internal.misc.Unsafe.allocateInstance(Native Method) 2> at java.base/java.lang.invoke.DirectMethodHandle.allocateInstance(DirectMethodHandle.java:501) 2> at java.base/jdk.internal.platform.CgroupUtil.readStringValue(CgroupUtil.java:66) 2> at java.base/jdk.internal.platform.CgroupSubsystemController.getStringValue(CgroupSubsystemController.java:65) 2> at java.base/jdk.internal.platform.CgroupSubsystemController.getLongValue(CgroupSubsystemController.java:124) 2> at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.getLongVal(CgroupV2Subsystem.java:56) 2> at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.getLongVal(CgroupV2Subsystem.java:63) 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:856) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.getCpuShares(CgroupV2Subsystem.java:156) 2> at java.base/jdk.internal.platform.CgroupMetrics.getCpuShares(CgroupMetrics.java:75) 1> at org.opensearch.cluster.action.shard.ShardStateAction.sendShardAction(ShardStateAction.java:192) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.action.shard.ShardStateAction.localShardFailed(ShardStateAction.java:298) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService.sendFailShard(IndicesClusterStateService.java:845) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService.failAndRemoveShard(IndicesClusterStateService.java:834) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService$FailedShardHandler.lambda$accept$0(IndicesClusterStateService.java:866) ~[opensearch-2.19.1.jar:2.19.1] 2> at jdk.management/com.sun.management.internal.OperatingSystemImpl$ContainerCpuTicks.getContainerCpuLoad(OperatingSystemImpl.java:86) 2> at jdk.management/com.sun.management.internal.OperatingSystemImpl.getProcessCpuLoad(OperatingSystemImpl.java:271) 2> at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) 2> at java.base/java.lang.reflect.Method.invoke(Method.java: 2> 580) 2> at org.opensearch.monitor.Probes.getLoadAndScaleToPercent(Probes.java:47) 2> at org.opensearch.monitor.process.ProcessProbe.getProcessCpuPercent(ProcessProbe.java:109) 2> at org.opensearch.wlm.QueryGroupService.lambda$new$0(QueryGroupService.java:81) 2> at org.opensearch.search.backpressure.trackers.NodeDuressTrackers$NodeDuressTracker.test(NodeDuressTrackers.java:80) 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> at org.opensearch.search.backpressure.trackers.NodeDuressTrackers.isResourceInDuress(NodeDuressTrackers.java:35) 2> at org.opensearch.search.backpressure.trackers.NodeDuressTrackers.isNodeInDuress(NodeDuressTrackers.java:44) 2> at org.opensearch.wlm.cancellation.QueryGroupTaskCancellationService.handleNodeDuress(QueryGroupTaskCancellationService.java:120) 2> at org.opensearch.wlm.cancellation.QueryGroupTaskCancellationService.cancelTasks(QueryGroupTaskCancellationService.java:89) 2> at org.opensearch.wlm.QueryGroupService.doRun(QueryGroupService.java:129) 2> at org.opensearch.wlm.QueryGroupService.lambda$doStart$3(QueryGroupService.java:140) 2> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) 2> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 2> 24) Thread[id=173, name=opensearch[node_s4][search][T#2], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 25) Thread[id=124, name=opensearch[node_s2][clusterApplierService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 26) Thread[id=165, name=opensearch[node_s3][management][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opense 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 27) Thread[id=151, name=opensearch[node_s4][management][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> arch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 12 more 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 28) Thread[id=161, name=opensearch[node_s3][management][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 1> [2025-05-08T13:53:12,410][WARN ][o.o.c.a.s.ShardStateAction] [node_s4] unexpected failure while sending request [internal:cluster/shard/failure] to [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}] for shard entry [shard id [[foo][7]], allocation id [qC7D-4A8Ria7BQr7kT4z0Q], primary term [0], message [shard failure, reason [refresh failed source[api]]], failure [ClosedChannelException[null]], markAsStale [true]] 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 1> org.opensearch.transport.SendRequestTransportException: [node_s1][127.0.0.1:43491][internal:cluster/shard/failure] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:856) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.action.shard.ShardStateAction.sendShardAction(ShardStateAction.java:192) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.action.shard.ShardStateAction.localShardFailed(ShardStateAction.java:298) ~[opensearch-2.19.1.jar:2.19.1] 2> 29) Thread[id=93, name=opensearch[node_s2][transport_worker][T#2], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/java.lang.Thread.holdsLock(Native Method) 1> at org.opensearch.indices.cluster.IndicesClusterStateService.sendFailShard(IndicesClusterStateService.java:845) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:263) 2> at java.base/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:113) 2> at java.base/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> at java.base/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 30) Thread[id=187, name=opensearch[node_s2][search][T#5], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 1> at org.opensearch.indices.cluster.IndicesClusterStateService.failAndRemoveShard(IndicesClusterStateService.java:834) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 1> at org.opensearch.indices.cluster.IndicesClusterStateService$FailedShardHandler.lambda$accept$0(IndicesClusterStateService.java:866) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2> .base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 31) Thread[id=168, name=opensearch[node_s3][management][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 32) Thread[id=126, name=opensearch[node_s4][clusterApplierService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 33) Thread[id=88, name=opensearch[node_s5][transport_worker][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/java.lang.Thread.holdsLock(Native Method) 2> at java.base/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:104) 2> at java.base/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> at java.base/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 34) Thread[id=45, name=opensearch[node_s0][generic][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 35) Thread[id=177, name=opensearch[node_s5][search][T#4], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 36) Thread[id=169, name=opensearch[node_s4][management][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 37) Thread[id=87, name=opensearch[node_s0][transport_worker][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/java.util.ArrayList.clear(ArrayList.java:734) 2> at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:84) 2> at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 2> at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 2> at org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 2> at org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 2> at org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 2> at org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 2> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 38) Thread[id=104, name=opensearch[node_s3][generic][T#6], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 2> 39) Thread[id=135, name=opensearch[node_s2][generic][T#7], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 12 more 1> [2025-05-08T13:53:12,412][WARN ][o.o.t.ThreadPool ] [node_s3] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x00007b49d884b598@44e26291] on thread pool [generic] 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s3/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:35:02.283609087Z) 2> at jdk.management/com.sun.management.internal.OperatingSystemImpl.getProcessCpuLoad0(Native Method) 2> at jdk.management/com.sun.management.internal.OperatingSystemImpl$ProcessCpuTicks.lambda$cpuSetCalc$1(OperatingSystemImpl.java:133) 2> at jdk.management/com.sun.management.internal.OperatingSystemImpl$ContainerCpuTicks.getContainerCpuLoad(OperatingSystemImpl.java:109) 2> at jdk.management/com.sun.management.internal.OperatingSystemImpl.getProcessCpuLoad(OperatingSystemImpl.java:271) 2> at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) 2> at java.base/java.lang.reflect.Method.invoke(Method.java:580) 2> at org.opensearch.monitor.Probes.getLoadAndScaleToPercent(Probes.java:47) 2> at org.opensearch.monitor.process.ProcessProbe.getProcessCpuPercent(ProcessProbe.java:109) 2> at org.opensearch.search.backpressure.SearchBackpressureService$1.lambda$new$0(SearchBackpressureService.java:104) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:941) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.search.backpressure.trackers.NodeDuressTrackers$NodeDuressTracker.test(NodeDuressTrackers.java:80) 2> at org.opensearch.search.backpressure.trackers.NodeDuressTrackers.isResourceInDuress(NodeDuressTrackers.java:35) 2> at org.opensearch.search.backpressure.trackers.NodeDuressTrackers.isNodeInDuress(NodeDuressTrackers.java:44) 2> at org.opensearch.search.backpressure.SearchBackpressureService.doRun(SearchBackpressureService.java:191) 2> at org.opensearch.search.backpressure.SearchBackpressureService.lambda$doStart$5(SearchBackpressureService.java:435) 2> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) 2> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 40) Thread[id=139, name=opensearch[node_s4][generic][T#8], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:13,026][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:38084, remoteAddress=127.0.0.1/127.0.0.1:45869}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> 41) Thread[id=59, name=opensearch[node_s3][generic][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 42) Thread[id=150, name=opensearch[node_s1][management][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 2> 43) Thread[id=176, name=opensearch[node_s5][search][T#3], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 44) Thread[id=86, name=opensearch[node_s3][transport_worker][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 45) Thread[id=130, name=opensearch[node_s4][generic][T#6], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 46) Thread[id=162, name=opensearch[node_s3][write][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base 2> /java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 47) Thread[id=80, name=opensearch[node_s5][generic][T#3], state=BLOCKED, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 48) Thread[id=175, name=opensearch[node_s5][search][T#2], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 49) Thread[id=183, name=opensearch[node_s2][search][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 50) Thread[id=188, name=opensearch[node_s2][search][T#6], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 51) Thread[id=125, name=opensearch[node_s3][clusterApplierService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 52) Thread[id=128, name=opensearch[node_s1][clusterApplierService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 53) Thread[id=167, name=opensearch[node_s4][management][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 54) Thread[id=164, name=opensearch[node_s4][management][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.uti 2> l.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 55) Thread[id=90, name=opensearch[node_s0][transport_worker][T#2], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1551) 2> at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:285) 2> at java.base/java.nio.Buffer.(Buffer.java:257) 2> at java.base/java.nio.ByteBuffer.(ByteBuffer.java:316) 2> at java.base/java.nio.HeapByteBuffer.(HeapByteBuffer.java:102) 2> at java.base/java.nio.HeapByteBuffer.duplicate(HeapByteBuffer.java:142) 2> at org.opensearch.nio.FlushOperation.getBuffersToWrite(FlushOperation.java:99) 2> at org.opensearch.nio.SocketChannelContext.flushToChannel(SocketChannelContext.java:322) 2> at org.opensearch.nio.BytesChannelContext.singleFlush(BytesChannelContext.java:99) 2> at org.opensearch.nio.BytesChannelContext.flushChannel(BytesChannelContext.java:71) 2> at org.opensearch.nio.EventHandler.handleWrite(EventHandler.java:172) 2> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:191) 2> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) 2> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) 2> at org.opensearch.nio.NioSelector.handleQueuedWrites(NioSelector.java:507) 2> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:280) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 56) Thread[id=134, name=opensearch[node_s5][generic][T#7], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at org.opensearch.common.util.concurrent.ThreadContext$ThreadContextStruct.putPersistent(ThreadContext.java:779) 2> at org.opensearch.common.util.concurrent.ThreadContext.stashContext(ThreadContext.java:164) 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1012) 2> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 57) Thread[id=178, name=opensearch[node_s5][search][T#5], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 58) Thread[id=91, name=opensearch[node_s1][transport_worker][T#2], state=BLOCKED, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at org.apache.logging.log4j.core.layout.TextEncoderHelper.writeEncodedText(TextEncoderHelper.java:77) 2> at org.apache.logging.log4j.core.layout.TextEncoderHelper.encodeChunkedText(TextEncoderHelper.java:141) 2> at org.apache.logging.log4j.core.layout.TextEncoderHelper.encodeText(TextEncoderHelper.java:56) 2> at org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:67) 2> at org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:32) 2> at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:227) 2> at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:59) 2> at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:219) 2> at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:212) 2> at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:203) 2> at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161) 2> at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134) 2> at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125) 2> at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89) 2> at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:686) 2> at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:644) 2> at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:620) 2> at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:556) 2> at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:81) 2> at org.apache.logging.log4j.core.Logger.log(Logger.java:163) 2> at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2165) 2> at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2119) 2> at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2102) 2> at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1963) 2> at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1804) 2> at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2681) 2> at org.opensearch.transport.NativeMessageHandler.sendErrorResponse(NativeMessageHandler.java:393) 2> at org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:280) 2> at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) 2> at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) 2> at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 2> at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 2> at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 2> at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) 2> at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 2> at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 2> at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 2> at org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandle 2> r.consumeReads(MockNioTransport.java:343) 2> at org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 2> at org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 2> at org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 2> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 2> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 2> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 59) Thread[id=142, name=opensearch[node_s1][generic][T#9], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 60) Thread[id=155, name=opensearch[node_s4][DanglingIndices#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 61) Thread[id=62, name=opensearch[node_s2][generic][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 62) Thread[id=131, name=opensearch[node_s3][generic][T#7], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/java.lang.Object.clone(Native Method) 2> at org.opensearch.wlm.ResourceType.values(ResourceType.java:28) 2> at org.opensearch.search.backpressure.trackers.NodeDuressTrackers.isNodeInDuress(NodeDuressTrackers.java:43) 2> at org.opensearch.search.backpressure.SearchBackpressureService.doRun(SearchBackpressureService.java:191) 2> at org.opensearch.search.backpressure.SearchBackpressureService.lambda$doStart$5(SearchBackpressureService.java:435) 2> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) 2> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 63) Thread[id=76, name=opensearch[node_s5][generic][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at org.apache.logging.log4j.message.ParameterFormatter$MessagePatternAnalysis.access$000(ParameterFormatter.java) 2> at org.apache.logging.log4j.message.ParameterFormatter.analyzePattern(ParameterFormatter.java:138) 2> at org.apache.logging.log4j.message.ParameterFormatter.analyzePattern(ParameterFormatter.java:87) 2> at org.apache.logging.log4j.message.ParameterizedMessage.(ParameterizedMessage.java:132) 2> at org.apache.logging.log4j.message.ParameterizedMessage.(ParameterizedMessage.java:169) 2> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:340) 1> [2025-05-08T13:53:13,360][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:34648, remoteAddress=127.0.0.1/127.0.0.1:41581}] 2> at org.opensearch.telemetry.tracing.handler.TraceableTransportResponseHandler.handleException(TraceableTransportResponseHandler.java:81) 2> at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1527) 2> at org.opensearch.transport.TransportService$6.doRun(TransportService.java:1044) 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) 2> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 64) Thread[id=77, name=opensearch[node_s3][generic][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.io.BufferedReader.(BufferedReader.java:104) 2> at java.base/java.io.BufferedReader.(BufferedReader.java:115) 2> at java.base/java.nio.file.Files.newBufferedReader(Files.java:2911) 2> at java.base/java.nio.file.Files.newBufferedReader(Files.java:2943) 2> at java.base/jdk.internal.platform.CgroupUtil.lambda$readStringValue$1(CgroupUtil.java:67) 2> at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) 2> at java.base/jdk.internal.platform.CgroupUtil.readStringValue(CgroupUtil.java:69) 2> at java.base/jdk.internal.platform.CgroupSubsystemController.getStringValue(CgroupSubsystemController.java:65) 2> at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.getEffectiveCpuSetCpus(CgroupV2Subsystem.java:215) 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/jdk.internal.platform.CgroupMetrics.getEffectiveCpuSetCpus(CgroupMetrics.java:105) 2> at jdk.management/com.sun.management.internal.OperatingSystemImpl$ContainerCpuTicks.getContainerCpuLoad(OperatingSystemImpl.java:99) 2> at jdk.management/com.sun.management.internal.OperatingSystemImpl.getProcessCpuLoad(OperatingSystemImpl.java:271) 2> at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) 2> at java.base/java.lang.reflect.Method.invoke(Method.java:580) 2> at org.opensearch.monitor.Probes.getLoadAndScaleToPercent(Probes.java:47) 2> at org.opensearch.monitor.process.ProcessProbe.getProcessCpuPercent(ProcessProbe.java:109) 2> at org.opensearch.wlm.QueryGroupService.lambda$new$0(QueryGroupService.j 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 2> ava:81) 2> at org.opensearch.search.backpressure.trackers.NodeDuressTrackers$NodeDuressTracker.test(NodeDuressTrackers.java:80) 2> at org.opensearch.search.backpressure.trackers.NodeDuressTrackers.isResourceInDuress(NodeDuressTrackers.java:35) 2> at org.opensearch.search.backpressure.trackers.NodeDuressTrackers.isNodeInDuress(NodeDuressTrackers.java:44) 2> at org.opensearch.wlm.cancellation.QueryGroupTaskCancellationService.handleNodeDuress(QueryGroupTaskCancellationService.java:120) 2> at org.opensearch.wlm.cancellation.QueryGroupTaskCancellationService.cancelTasks(QueryGroupTaskCancellationService.java:89) 2> at org.opensearch.wlm.QueryGroupService.doRun(QueryGroupService.java:129) 2> at org.opensearch.wlm.QueryGroupService.lambda$doStart$3(QueryGroupService.java:140) 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) 2> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 65) Thread[id=49, name=opensearch[node_s1][generic][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 66) Thread[id=95, name=opensearch[node_s4][generic][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/java.lang.StackTraceElement.of(StackTraceElement.java:563) 2> at java.base/java.lang.Throwable.getOurStackTrace(Throwable.java:861) 2> at java.base/java.lang.Throwable.getStackTrace(Throwable.java:853) 2> at java.base/java.lang.Thread.getStackTrace(Thread.java:2451) 2> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:251) 2> at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:4165) 2> at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:4153) 2> at org.opensearch.index.shard.IndexShard.runUnderPrimaryPermit(IndexShard.java:4243) 2> at org.opensearch.index.IndexService.sync(IndexService.java:1350) 2> at org.opensearch.index.IndexService.maybeSyncGlobalCheckpoints(IndexService.java:1330) 2> at org.opensearch.index.IndexService$AsyncGlobalCheckpointTask.runInternal(IndexService.java:1561) 2> at org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:159) 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 67) Thread[id=96, name=opensearch[node_s3][generic][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 68) Thread[id=163, name=opensearch[node_s4][write][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 69) Thread[id=158, name=opensearch[node_s3][generic][T#9], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 70) Thread[id=100, name=opensearch[node_s1][generic][T#6], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 71) Thread[id=190, name=opensearch[node_s1][generic][T#10], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue 1> [2025-05-08T13:53:13,455][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:45155, remoteAddress=/127.0.0.1:60272}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 2> .java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 72) Thread[id=66, name=opensearch[node_s5][scheduler][T#1], state=BLOCKED, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1763) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 73) Thread[id=56, name=opensearch[node_s3][scheduler][T#1], state=BLOCKED, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/java.util.stream.AbstractPipeline.wrapSink(AbstractPipeline.java:547) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendErrorResponse(NativeOutboundHandler.java:180) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:105) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.telemetry.tracing.channels.TraceableTcpTransportChannel.sendResponse(TraceableTcpTransportChannel.java:101) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.initPartialTraversalState(StreamSpliterators.java:291) 1> at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:81) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:158) 1> at org.opensearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:75) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298) 2> at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681) 2> at org.opensearch.indices.IndexingMemoryController.availableShards(IndexingMemoryController.java:198) 2> at org.opensearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker.runUnlocked(IndexingMemoryController.java:340) 2> at org.opensearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:326) 2> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) 2> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) 2> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) 2> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) 1> at org.opensearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:70) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$6.onFailure(ActionListener.java:311) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:1213) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:1171) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at org.opensearch.telemetry.tracing.handler.TraceableTransportResponseHandler.handleException(TraceableTransportResponseHandler.java:81) ~[opensearch-2.19.1.jar:2.19.1] 2> 74) Thread[id=143, name=opensearch[node_s1][clusterManagerService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at org.opensearch.transport.TransportService$9.handleException(TransportService.java:1741) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 1> at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1527) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 1> at org.opensearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1641) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1615) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:81) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:75) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:70) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 75) Thread[id=184, name=opensearch[node_s2][search][T#2], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 1> at org.opensearch.core.action.ActionListener$6.onFailure(ActionListener.java:311) ~[opensearch-core-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onFailure(TransportReplicationAction.java:669) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.handleException(TransportReplicationAction.java:663) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$runWithPrimaryShardReference$3(TransportReplicationAction.java:628) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$4.onFailure(ActionListener.java:192) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.ReplicationOperation.finishAsFailed(ReplicationOperation.java:437) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 76) Thread[id=53, name=opensearch[node_s2][generic][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/java.io.BufferedReader.(BufferedReader.java:115) 2> at java.base/java.nio.file.Files.newBufferedReader(Files.java:2911) 2> at java.base/java.nio.file.Files.newBufferedReader(Files.java:2943) 1> at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$4.onFailure(ActionListener.java:192) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:347) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:100) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/jdk.internal.platform.CgroupUtil.lambda$readStringValue$1(CgroupUtil.java:67) 2> at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:57) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/jdk.internal.platform.CgroupUtil.readStringValue(CgroupUtil.java:69) 2> at java.base/jdk.internal.platform.CgroupSubsystemController.getStringValue(CgroupSubsystemController.java:65) 2> at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.getFromCpuMax(CgroupV2Subsystem.java:140) 2> at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.getCpuQuota(CgroupV2Subsystem.java:136) 2> at java.base/jdk.internal.platform.CgroupMetrics.getCpuQuota(CgroupMetrics.java:70) 2> at jdk.management/com.sun.management.internal.OperatingSystemImpl$ContainerCpuTicks.getContainerCpuLoad(OperatingSystemImpl.java:85) 2> at jdk.management/com.sun.management.internal.OperatingSystemImpl.getProcessCpuLoad(OperatingSystemImpl.java:271) 2> at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) 1> at org.opensearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1333) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:150) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:654) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:547) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$36(IndexShard.java:4194) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.lang.reflect.Method.invoke(Method.java:580) 1> at org.opensearch.core.action.ActionListener$3.onResponse(ActionListener.java:132) ~[opensearch-core-2.19.1.jar:2.19.1] 2> at org.opensearch.monitor.Probes.getLoadAndScaleToPercent(Probes.java:47) 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:310) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.monitor.process.ProcessProbe.getProcessCpuPercent(ProcessProbe.java:109) 2> at org.opensearch.node.resource.tracker.AverageCpuUsageTracker.getUsage(AverageCpuUsageTracker.java:33) 2> at org.opensearch.node.resource.tracker.AbstractAverageUsageTracker.lambda$doStart$0(AbstractAverageUsageTracker.java:86) 2> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) 2> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base/java.util.concurrent.ThreadPoolExec 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:255) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:4165) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1262) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:544) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction. 2> utor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 77) Thread[id=52, name=opensearch[node_s1][generic][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 1> handlePrimaryRequest(TransportReplicationAction.java:483) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1071) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:13,656][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=1, [cluster.fault_detection.follower_check.retry_count]=3} failed, retrying 2> 78) Thread[id=92, name=opensearch[node_s4][transport_worker][T#2], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at org.opensearch.nio.NioSelector.getIoBuffer(NioSelector.java:97) 1> org.opensearch.transport.SendRequestTransportException: [node_s3][127.0.0.1:45155][internal:coordination/fault_detection/follower_check] 2> at org.opensearch.nio.SocketChannelContext.flushToChannel(SocketChannelContext.java:315) 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.nio.BytesChannelContext.singleFlush(BytesChannelContext.java:99) 2> at org.opensearch.nio.BytesChannelContext.flushChannel(BytesChannelContext.java:71) 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.nio.EventHandler.handleWrite(EventHandler.java:172) 2> at org.opensearch.transport.nio.TestEventHandler.handleWrite(TestEventHandler.java:191) 2> at org.opensearch.nio.NioSelector.handleWrite(NioSelector.java:430) 2> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:383) 2> at org.opensearch.nio.NioSelector.queueWrite(NioSelector.java:307) 2> at org.opensearch.nio.SocketChannelContext.sendMessage(SocketChannelContext.java:179) 2> at org.opensearch.transport.nio.MockNioTransport$MockSocketChannel.sendMessage(MockNioTransport.java:416) 2> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:83) 2> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:71) 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.transport.TransportKeepAlive.sendPing(TransportKeepAlive.java:133) 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.transport.TransportKeepAlive.receiveKeepAlive(TransportKeepAlive.java:120) 2> at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:124) 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 2> at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 2> at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 2> at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:132) 2> at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 2> at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 2> at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 2> at org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2> at org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 2> at org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 2> at org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 2> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 79) Thread[id=180, name=opensearch[node_s5][search][T#6], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 2> 80) Thread[id=42, name=opensearch[node_s0][scheduler][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269) 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1763) 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 12 more 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 81) Thread[id=67, name=opensearch[node_s4][generic][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at org.gradle.internal.time.MonotonicClock.syncIsDue(MonotonicClock.java) 2> at org.gradle.internal.time.MonotonicClock.getCurrentTime(MonotonicClock.java:83) 2> at org.gradle.api.internal.tasks.testing.redirector.TestOutputRedirector$Forwarder.onOutput(TestOutputRedirector.java:77) 2> at org.gradle.api.internal.tasks.testing.redirector.DefaultStandardOutputRedirector$WriteAction.text(DefaultStandardOutputRedirector.java:85) 2> at org.gradle.internal.io.LineBufferingOutputStream.flush(LineBufferingOutputStream.java:97) 2> at org.gradle.internal.io.LineBufferingOutputStream.write(LineBufferingOutputStream.java:81) 2> at java.base/java.io.OutputStream.write(OutputStream.java:167) 2> at java.base/java.io.PrintStream.implWrite(PrintStream.java:643) 2> at java.base/java.io.PrintStream.write(PrintStream.java:623) 2> at org.gradle.internal.io.LinePerThreadBufferingOutputStream.write(LinePerThreadBufferingOutputStream.java:250) 2> at org.apache.lucene.tests.util.TestRuleLimitSysouts$DelegateStream.write(TestRuleLimitSysouts.java:173) 2> at java.base/java.io.PrintStream.implWrite(PrintStream.java:643) 2> at java.base/java.io.PrintStream.write(PrintStream.java:623) 2> at org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.java:53) 2> at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStreamManager.java:250) 2> at org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager.java:283) 2> at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:294) 2> at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:221) 2> at org.apache.logging.log4j.cor 2> e.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:212) 2> at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:203) 2> at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161) 2> at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134) 2> at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125) 2> at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89) 2> at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:686) 2> at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:644) 2> at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:620) 2> at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:556) 2> at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:81) 2> at org.apache.logging.log4j.core.Logger.log(Logger.java:163) 2> at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2165) 1> [2025-05-08T21:53:13,855][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][8] start check index 2> at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2119) 2> at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2102) 2> at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1963) 2> at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1804) 2> at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2681) 2> at org.opensearch.transport.OutboundHandler$SendContext.innerOnFailure(OutboundHandler.java:152) 2> at org.opensearch.core.action.NotifyOnceListener.onFailure(NotifyOnceListener.java:65) 2> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:135) 2> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) 2> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) 2> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) 2> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) 2> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) 2> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) 2> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) 2> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) 2> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:856) 1> [2025-05-08T13:53:13,709][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=1, [cluster.fault_detection.follower_check.retry_count]=3} failed, retrying 2> at org.opensearch.cluster.action.shard.ShardStateAction.sendShardAction(ShardStateAction.java:192) 1> org.opensearch.transport.SendRequestTransportException: [node_s4][127.0.0.1:45869][internal:coordination/fault_detection/follower_check] 2> at org.opensearch.cluster.action.shard.ShardStateAction.localShardFailed(ShardStateAction.java:298) 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.indices.cluster.IndicesClusterStateService.sendFailShard(IndicesClusterStateService.java:845) 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.indices.cluster.IndicesClusterStateService.failAndRemoveShard(IndicesClusterStateService.java:834) 2> at org.opensearch.indices.cluster.IndicesClusterStateService$FailedShardHandler.lambda$accept$0(IndicesClusterStateService.java:866) 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 82) Thread[id=170, name=opensearch[node_s4][search][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 83) Thread[id=171, name=opensearch[node_s3][search][T#2], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 2> 84) Thread[id=172, name=opensearch[node_s3][search][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 85) Thread[id=154, name=opensearch[node_s3][DanglingIndices#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 86) Thread[id=133, name=opensearch[node_s4][generic][T#7], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.b 2> ase/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 87) Thread[id=160, name=opensearch[node_s4][management][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 88) Thread[id=144, name=opensearch[node_s4][AsyncLucenePersistedState#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 2> 89) Thread[id=152, name=opensearch[node_s3][management][T#1], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 12 more 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 90) Thread[id=156, name=opensearch[node_s3][refresh][T#1], state=BLOCKED, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at org.opensearch.index.translog.Translog.closeFilesIfNoPendingRetentionLocks(Translog.java:1843) 2> at org.opensearch.index.translog.LocalTranslog.close(LocalTranslog.java:158) 2> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) 2> at org.opensearch.common.util.io.IOUtils.closeWhileHandlingException(IOUtils.java:179) 1> [2025-05-08T13:53:13,818][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s5}{7Bx5ewP5Q42abT6xeM4uBA}{OVcy3CfWRJeYN8aHI5nZRg}{127.0.0.1}{127.0.0.1:41581}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=1, [cluster.fault_detection.follower_check.retry_count]=3} failed, retrying 2> at org.opensearch.index.translog.InternalTranslogManager.close(InternalTranslogManager.java:475) 2> at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) 2> at org.opensearch.index.engine.InternalEngine.closeNoLock(InternalEngine.java:2396) 2> at org.opensearch.index.engine.Engine.failEngine(Engine.java:1342) 2> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1906) 2> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1864) 1> org.opensearch.transport.SendRequestTransportException: [node_s5][127.0.0.1:41581][internal:coordination/fault_detection/follower_check] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.index.shard.IndexShard.refresh(IndexShard.java:1407) 2> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.lambda$shardOperationOnPrimary$0(TransportShardRefreshAction.java:101) 2> at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344) 2> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:100) 2> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:57) 2> at org.opensearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1333) 2> at org.opensearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:150) 2> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:654) 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:547) 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$36(IndexShard.java:4194) 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 2> at org.opensearch.core.action.ActionListener$3.onResponse(ActionListener.java:132) 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 2> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:310) 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 2> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:255) 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 2> at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:4165) 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 2> at org.opensearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1262) 2> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:544) 2> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) 2> at org.opensearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:483) 2> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) 2> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) 2> at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1071) 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) 2> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> 91) Thread[id=136, name=opensearch[node_s5][generic][T#8], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSuppor 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 2> t.parkNanos(LockSupport.java:410) 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 92) Thread[id=101, name=opensearch[node_s5][generic][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 93) Thread[id=57, name=opensearch[node_s2][generic][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 94) Thread[id=48, name=opensearch[node_s1][scheduler][T#1], state=BLOCKED, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at org.apache.logging.log4j.util.LoaderUtil.getThreadContextClassLoader(LoaderUtil.java:90) 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.apache.logging.log4j.util.LoaderUtil.loadClass(LoaderUtil.java:141) 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 2> at org.apache.logging.log4j.core.impl.ThrowableProxyHelper.loadClass(ThrowableProxyHelper.java:214) 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.apache.logging.log4j.core.impl.ThrowableProxyHelper.toExtendedStackTrace(ThrowableProxyHelper.java:112) 1> ... 12 more 2> at org.apache.logging.log4j.core.impl.ThrowableProxy.(ThrowableProxy.java:113) 2> at org.apache.logging.log4j.core.impl.ThrowableProxy.(ThrowableProxy.java:97) 1> [2025-05-08T13:53:13,769][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:56156, remoteAddress=127.0.0.1/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 2> at org.apache.logging.log4j.core.impl.MutableLogEvent.getThrownProxy(MutableLogEvent.java:354) 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> at org.apache.logging.log4j.core.pattern.ExtendedThrowablePatternConverter.format(ExtendedThrowablePatternConverter.java:63) 2> at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:44) 2> at org.apache.logging.log4j.core.layout.PatternLayout$PatternFormatterPatternSerializer.toSerializable(PatternLayout.java:384) 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 2> at org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:240) 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 2> at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:225) 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:59) 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:219) 2> at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:212) 2> at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:203) 2> at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161) 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134) 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:686) 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:644) 2> at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:620) 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 2> at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:556) 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 2> at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:81) 2> at org.apache.logging.log4j.core.Logger.log(Logger.java:163) 2> at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2165) 2> at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2119) 2> at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2102) 2> at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1963) 2> at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1804) 2> at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2681) 2> at org.opensearch.transport.OutboundHandler$SendContext.innerOnFailure(OutboundHandler.java:152) 2> at org.opensearch.core.action.NotifyOnceListener.onFailure(NotifyOnceListener.java:65) 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 2> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:135) 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at 2> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) 2> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) 1> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 2> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) 2> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) 2> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) 1> [2025-05-08T21:53:13,810][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=1, [cluster.fault_detection.follower_check.retry_count]=3} failed, retrying 2> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) 1> org.opensearch.transport.SendRequestTransportException: [node_s0][127.0.0.1:40859][internal:coordination/fault_detection/follower_check] 2> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) 2> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 95) Thread[id=103, name=opensearch[node_s0][generic][T#6], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at ja 2> va.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 96) Thread[id=181, name=opensearch[node_s5][search][T#7], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 97) Thread[id=157, name=opensearch[node_s4][refresh][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 12 more 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 98) Thread[id=64, name=opensearch[node_s2][generic][T#4], state=BLOCKED, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) 2> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) 2> at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:941) 2> at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) 2> at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) 2> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) 2> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) 2> at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) 2> at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) 2> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) 2> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) 2> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) 1> [2025-05-08T13:53:13,956][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:43491, remoteAddress=/127.0.0.1:55672}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) 2> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 99) Thread[id=99, name=opensearch[node_s0][generic][T#5], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 100) Thread[id=82, name=opensearch[node_s5][generic][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendResponse(NativeOutboundHandler.java:152) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:96) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.telemetry.tracing.channels.TraceableTcpTransportChannel.sendResponse(TraceableTcpTransportChannel.java:89) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:72) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:153) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:278) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 101) Thread[id=186, name=opensearch[node_s2][search][T#4], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 1> at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) ~[framework-2.19.1.jar:2.19.1] 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 2> 102) Thread[id=148, name=opensearch[node_s3][generic][T#8], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 2> at org.apache.logging.log4j.core.appender.OutputStreamManager.writeBytes(OutputStreamManager.java:352) 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 2> at org.apache.logging.log4j.core.layout.TextEncoderHelper.writeEncodedText(TextEncoderHelper.java:94) 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2> at org.apache.logging.log4j.core.layout.TextEncoderHelper.encodeChunkedText(TextEncoderHelper.java:141) 2> at org.apache.logging.log4j.core.layout.TextEncoderHelper.encodeText(TextEncoderHelper.java:56) 2> at org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:67) 2> at org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:32) 2> at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:227) 2> at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:59) 2> at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:219) 2> at org.apache.logging.log 2> 4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:212) 2> at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:203) 2> at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161) 2> at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134) 2> at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125) 2> at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89) 2> at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:686) 2> at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:644) 2> at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:620) 2> at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:556) 2> at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:81) 2> at org.apache.logging.log4j.core.Logger.log(Logger.java:163) 2> at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2165) 2> at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2119) 2> at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2102) 2> at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1963) 2> at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1804) 2> at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2681) 2> at org.opensearch.transport.OutboundHandler$SendContext.innerOnFailure(OutboundHandler.java:152) 2> at org.opensearch.core.action.NotifyOnceListener.onFailure(NotifyOnceListener.java:65) 2> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:135) 2> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) 2> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) 2> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) 2> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) 2> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) 2> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) 2> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) 2> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) 2> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:856) 2> at org.opensearch.cluster.action.shard.ShardStateAction.sendShardAction(ShardStateAction.java:192) 2> at org.opensearch.cluster.action.shard.ShardStateAction.localShardFailed(ShardStateAction.java:298) 2> at org.opensearch.indices.cluster.IndicesClusterStateService.sendFailShard(IndicesClusterStateService.java:845) 2> at org.opensearch.indices.cluster.IndicesClusterStateService.failAndRemoveShard(IndicesClusterStateService.java:834) 2> at org.opensearch.indices.cluster.IndicesClusterStateService$FailedShardHandler.lambda$accept$0(IndicesClusterStateService.java:866) 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 103) Thread[id=46, name=opensearch[node_s0][generic][T#4], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 104) Thread[id=179, name=opensearch[node_s3][generic][T#10], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 105) Thread[id=153, name=opensearch[node_s1][refresh][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 106) Thread[id=174, name=opensearch[node_s5][search][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 107) Thread[id=129, name=opensearch[node_s5][generic][T#6], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.L 2> inkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 108) Thread[id=61, name=opensearch[node_s4][scheduler][T#1], state=BLOCKED, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1763) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 109) Thread[id=127, name=opensearch[node_s5][clusterApplierService#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519) 2> at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780) 2> at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725) 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712) 2> at java.base/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 110) Thread[id=138, name=opensearch[node_s0][generic][T#8], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 1> [2025-05-08T13:53:14,236][WARN ][o.o.t.OutboundHandler ] [node_s5] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:55714, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 2> 111) Thread[id=94, name=opensearch[node_s5][transport_worker][T#2], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.lang.System$2.blockedOn(System.java:2374) 2> at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.blockedOn(AbstractInterruptibleChannel.java:209) 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.nio.channels.spi.AbstractSelector.end(AbstractSelector.java:249) 2> at java.base/sun.nio.ch.SelectorImpl.end(SelectorImpl.java:107) 2> at java.base/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:138) 2> at java.base/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130) 2> at java.base/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:142) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 112) Thread[id=132, name=opensearch[node_s0][generic][T#7], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 2> 113) Thread[id=137, name=opensearch[node_s2][generic][T#8], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 2> 114) Thread[id=79, name=opensearch[node_s4][generic][T#3], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 115) Thread[id=44, name=opensearch[node_s0][generic][T#2], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:12 2> 68) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 116) Thread[id=84, name=opensearch[node_s1][transport_worker][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at org.opensearch.nio.NioSelector.assertOnSelectorThread(NioSelector.java:131) 2> at org.opensearch.nio.SocketChannelContext.readyForFlush(SocketChannelContext.java:255) 2> at org.opensearch.nio.NioSelector.writeToChannel(NioSelector.java:371) 2> at org.opensearch.nio.NioSelector.queueWrite(NioSelector.java:307) 2> at org.opensearch.nio.SocketChannelContext.sendMessage(SocketChannelContext.java:179) 2> at org.opensearch.transport.nio.MockNioTransport$MockSocketChannel.sendMessage(MockNioTransport.java:416) 2> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:83) 2> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:71) 2> at org.opensearch.transport.TransportKeepAlive.sendPing(TransportKeepAlive.java:133) 2> at org.opensearch.transport.TransportKeepAlive.receiveKeepAlive(TransportKeepAlive.java:120) 2> at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:124) 2> at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) 2> at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) 2> at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 2> at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:132) 2> at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) 2> at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 2> at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 2> at org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 2> at org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 2> at org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 2> at org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 2> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 2> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 2> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 117) Thread[id=98, name=opensearch[node_s1][generic][T#5], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/sun.invoke.util.VerifyAccess.isTypeVisible(VerifyAccess.java:346) 2> at java.base/java.lang.invoke.MemberName.checkForTypeAlias(MemberName.java:816) 2> at java.base/java.lang.invoke.MemberName$Factory.resolve(MemberName.java:966) 2> at java.base/java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:991) 2> at java.base/java.lang.invoke.MethodHandles$Lookup.resolveOrFail(MethodHandles.java:3758) 2> at java.base/java.lang.invoke.MethodHandles$Lookup.linkMethodHandleConstant(MethodHandles.java:4261) 2> at java.base/java.lang.invoke.MethodHandleNatives.linkMethodHandleConstant(MethodHandleNatives.java:627) 1> [2025-05-08T21:53:14,264][WARN ][o.o.t.ThreadPool ] [node_s1] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x00007b49d884b598@15554127] on thread pool [generic] 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s1/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:34:54.491447622Z) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$1.handleException(FollowersChecker.java:409) 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.telemetry.tracing.handler.TraceableTransportResponseHandler.handleException(TraceableTransportResponseHandler.java:81) 1> at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:941) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1527) 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.transport.TransportService$6.doRun(TransportService.java:1044) 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 118) Thread[id=83, name=opensearch[node_s4][transport_worker][T#1], state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:504) 2> at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:187) 2> at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:172) 2> at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:54) 2> at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) 2> at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) 2> at org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) 2> at org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) 2> at org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) 2> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) 2> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 2> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 2> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> [2025-05-08T13:53:14,237][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:43491, remoteAddress=/127.0.0.1:55672}] 1> java.lang.RuntimeException: The rule is not currently executing. 2> 119) Thread[id=159, name=opensearch[node_s4][generic][T#10], state=TIMED_WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410) 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 2> at java.base/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1268) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 2> 120) Thread[id=189, name=opensearch[node_s2][search][T#7], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java. 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendErrorResponse(NativeOutboundHandler.java:180) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:105) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.telemetry.tracing.channels.TraceableTcpTransportChannel.sendResponse(TraceableTcpTransportChannel.java:101) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.sendErrorResponse(NativeMessageHandler.java:390) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:280) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:14,253][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=1, [cluster.fault_detection.follower_check.retry_count]=3} failed, retrying 1> org.opensearch.transport.SendRequestTransportException: [node_s2][127.0.0.1:39703][internal:coordination/fault_detection/follower_check] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 2> base/java.lang.Thread.run(Thread.java:1583) 2> 121) Thread[id=145, name=opensearch[node_s3][AsyncLucenePersistedState#updateTask][T#1], state=WAITING, group=TGRP-ICUCollationKeywordFieldMapperIT] 2> at java.base/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371) 2> at java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:458) 2> at java.base/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:613) 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1257) 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 2> at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:178) 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070) 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 2> at java.base/java.lang.Thread.run(Thread.java:1583) 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializ 1> er.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 12 more 1> [2025-05-08T13:53:14,402][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:55652, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:14,422][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:55728, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:14,143][WARN ][o.o.t.TransportChannel ] [node_s3] Failed to send error response for action [indices:admin/refresh[s]] and request [BasicReplicationRequest{[foo][8]}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendErrorResponse(NativeOutboundHandler.java:180) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:105) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.telemetry.tracing.channels.TraceableTcpTransportChannel.sendResponse(TraceableTcpTransportChannel.java:101) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:81) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:75) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:70) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$6.onFailure(ActionListener.java:311) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:1213) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:1171) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.telemetry.tracing.handler.TraceableTransportResponseHandler.handleException(TraceableTransportResponseHandler.java:81) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$9.handleException(TransportService.java:1741) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1527) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1641) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1615) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:81) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:75) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:70) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$6.onFailure(ActionListener.java:311) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onFailure(TransportReplicationAction.java:669) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.handleException(TransportReplicationAction.java:663) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$runWithPrimaryShardReference$3(TransportReplicationAction.java:628) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$4.onFailure(ActionListener.java:192) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.ReplicationOperation.finishAsFailed(ReplicationOperation.java:437) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$4.onFailure(ActionListener.java:192) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:347) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:57) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1333) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:150) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:654) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:547) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$36(IndexShard.java:4194) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$3.onResponse(ActionListener.java:132) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:310) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:255) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:4165) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1262) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:544) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAc 1> tion.handlePrimaryRequest(TransportReplicationAction.java:483) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1071) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Suppressed: org.opensearch.transport.RemoteTransportException: [node_s3][127.0.0.1:45155][indices:admin/refresh[s][p]] 1> Caused by: org.opensearch.index.engine.RefreshFailedEngineException: Refresh failed 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1910) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1864) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.refresh(IndexShard.java:1407) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.lambda$shardOperationOnPrimary$0(TransportShardRefreshAction.java:101) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:57) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1333) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:150) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:654) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:547) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$36(IndexShard.java:4194) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$3.onResponse(ActionListener.java:132) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:310) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:255) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:4165) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1262) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:544) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:483) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1071) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.nio.channels.ClosedChannelException 1> at java.base/sun.nio.ch.FileLockImpl.release(FileLockImpl.java:58) ~[?:?] 1> at java.base/java.nio.channels.FileLock.close(FileLock.java:336) ~[?:?] 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.close(NativeFSLockFactory.java:195) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.util.IOUtils.close(IOUtils.java:85) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.util.IOUtils.close(IOUtils.java:72) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2561) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2466) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:5774) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:719) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:72) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.Refe 1> renceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:434) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:414) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1887) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1864) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.refresh(IndexShard.java:1407) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.lambda$shardOperationOnPrimary$0(TransportShardRefreshAction.java:101) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:57) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1333) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:150) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:654) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:547) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$36(IndexShard.java:4194) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$3.onResponse(ActionListener.java:132) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:310) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:255) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:4165) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1262) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:544) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:483) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) [opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) [opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1071) [opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) [opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:14,507][WARN ][o.o.t.NativeMessageHandler] [node_s1] Failed to send error message back to client for action [internal:coordination/fault_detection/leader_check] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendErrorResponse(NativeOutboundHandler.java:180) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:105) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.telemetry.tracing.channels.TraceableTcpTransportChannel.sendResponse(TraceableTcpTransportChannel.java:101) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.sendErrorResponse(NativeMessageHandler.java:390) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:280) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) 1> ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Suppressed: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendResponse(NativeOutboundHandler.java:152) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:96) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.telemetry.tracing.channels.TraceableTcpTransportChannel.sendResponse(TraceableTcpTransportChannel.java:89) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:72) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:153) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:278) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:14,802][WARN ][o.o.t.TcpTransport ] [node_s4] exception caught on transport layer [NioSocketChannel{localAddress=/127.0.0.1:55728, remoteAddress=127.0.0.1/127.0.0.1:43491}], closing connection 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.MockNioTransport$MockTcpChannelFactory.lambda$createChannel$1(MockNioTransport.java:253) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.InboundChannelBuffer.ensureCapacity(InboundChannelBuffer.java:100) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.readFromChannel(SocketChannelContext.java:298) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.Byt 2> java.lang.Exception: Suite timeout exceeded (>= 1200000 msec). at __randomizedtesting.SeedInfo.seed([298D86C07ECB082C]:0) 1> esChannelContext.read(BytesChannelContext.java:53) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2> REPRODUCE WITH: ./gradlew ':plugins:analysis-icu:internalClusterTest' --tests "org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT" -Dtests.seed=298D86C07ECB082C -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en -Dtests.timezone=GMT -Druntime.java=21 1> [2025-05-08T13:53:15,104][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:37330, remoteAddress=127.0.0.1/127.0.0.1:45155}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T21:53:15,147][INFO ][o.o.c.c.Coordinator ] [node_s4] cluster-manager node [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}] failed, restarting discovery 1> org.opensearch.OpenSearchException: node [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}] failed [3] consecutive checks 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:329) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.telemetry.tracing.handler.TraceableTransportResponseHandler.handleException(TraceableTransportResponseHandler.java:81) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1527) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$6.doRun(TransportService.java:1044) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: org.opensearch.transport.SendRequestTransportException: [node_s1][127.0.0.1:43491][internal:coordination/fault_detection/leader_check] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> ... 3 more 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> ... 3 more 1> [2025-05-08T13:53:15,173][INFO ][o.o.c.c.Coordinator ] [node_s3] cluster-manager node [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}] failed, restarting discovery 1> org.opensearch.OpenSearchException: node [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}] failed [3] consecutive checks 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:329) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.telemetry.tracing.handler.TraceableTransportResponseHandler.handleException(TraceableTransportResponseHandler.java:81) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1527) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$6.doRun(TransportService.java:1044) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: org.opensearch.transport.SendRequestTransportException: [node_s1][127.0.0.1:43491][internal:coordination/fault_detection/leader_check] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> ... 3 more 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> ... 3 more 1> [2025-05-08T21:53:15,162][INFO ][o.o.c.c.Coordinator ] [node_s5] cluster-manager node [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}] failed, restarting discovery 1> org.opensearch.OpenSearchException: node [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}] failed [3] consecutive checks 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:329) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.telemetry.tracing.handler.TraceableTransportResponseHandler.handleException(TraceableTransportResponseHandler.java:81) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1527) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$6.doRun(TransportService.java:1044) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: org.opensearch.transport.SendRequestTransportException: [node_s1][127.0.0.1:43491][internal:coordination/fault_detection/leader_check] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> ... 3 more 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> ... 3 more 1> [2025-05-08T13:53:15,416][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=2, [cluster.fault_detection.follower_check.retry_count]=3} failed, retrying 1> org.opensearch.transport.SendRequestTransportException: [node_s3][127.0.0.1:45155][internal:coordination/fault_detection/follower_check] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 12 more 1> [2025-05-08T13:53:15,433][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:38088, remoteAddress=127.0.0.1/127.0.0.1:45869}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:15,869][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:34632, remoteAddress=127.0.0.1/127.0.0.1:41581}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:15,865][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=2, [cluster.fault_detection.follower_check.retry_count]=3} failed, retrying 1> org.opensearch.transport.SendRequestTransportException: [node_s4][127.0.0.1:45869][internal:coordination/fault_detection/follower_check] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 12 more 1> [2025-05-08T13:53:16,103][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:35246, remoteAddress=127.0.0.1/127.0.0.1:40859}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:16,049][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s5}{7Bx5ewP5Q42abT6xeM4uBA}{OVcy3CfWRJeYN8aHI5nZRg}{127.0.0.1}{127.0.0.1:41581}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=2, [cluster.fault_detection.follower_check.retry_count]=3} failed, retrying 1> org.opensearch.transport.SendRequestTransportException: [node_s5][127.0.0.1:41581][internal:coordination/fault_detection/follower_check] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 12 more 1> [2025-05-08T13:53:16,299][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:56178, remoteAddress=127.0.0.1/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:16,266][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:38098, remoteAddress=127.0.0.1/127.0.0.1:40859}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:16,275][WARN ][o.o.t.OutboundHandler ] [node_s5] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:35572, remoteAddress=127.0.0.1/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:16,248][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:40188, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurre 1> nt.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:16,268][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=2, [cluster.fault_detection.follower_check.retry_count]=3} failed, retrying 1> org.opensearch.transport.SendRequestTransportException: [node_s0][127.0.0.1:40859][internal:coordination/fault_detection/follower_check] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 12 more 1> [2025-05-08T13:53:16,325][WARN ][o.o.t.OutboundHandler ] [node_s5] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:38102, remoteAddress=127.0.0.1/127.0.0.1:40859}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensear 1> ch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:16,239][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:35560, remoteAddress=127.0.0.1/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:16,265][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:38086, remoteAddress=127.0.0.1/127.0.0.1:40859}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:16,408][INFO ][o.o.c.s.ClusterApplierService] [node_s5] cluster-manager node changed {previous [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}], current []}, term: 1, version: 48, reason: becoming candidate: onLeaderFailure 1> [2025-05-08T13:53:16,408][INFO ][o.o.c.s.ClusterApplierService] [node_s3] cluster-manager node changed {previous [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}], current []}, term: 1, version: 48, reason: becoming candidate: onLeaderFailure 1> [2025-05-08T21:53:16,420][INFO ][o.o.t.s.M.Listener ] [node_s3] [foo][8] end check index 1> [2025-05-08T13:53:16,410][INFO ][o.o.c.s.ClusterApplierService] [node_s4] cluster-manager node changed {previous [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}], current []}, term: 1, version: 48, reason: becoming candidate: onLeaderFailure 1> [2025-05-08T13:53:16,455][WARN ][o.o.t.OutboundHandler ] [node_s0] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:60972, remoteAddress=127.0.0.1/127.0.0.1:45155}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T21:53:16,462][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=2, [cluster.fault_detection.follower_check.retry_count]=3} failed, retrying 1> org.opensearch.transport.SendRequestTransportException: [node_s2][127.0.0.1:39703][internal:coordination/fault_detection/follower_check] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 12 more 1> [2025-05-08T13:53:16,582][WARN ][o.o.t.OutboundHandler ] [node_s5] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:40190, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:19,607][WARN ][o.o.t.OutboundHandler ] [node_s5] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:40904, remoteAddress=127.0.0.1/127.0.0.1:40859}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T21:53:19,311][WARN ][o.o.t.ThreadPool ] [node_s5] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x00007b49d884b598@27cc7d94] on thread pool [generic] 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s5/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:35:08.444736755Z) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:941) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T21:53:19,347][WARN ][o.o.t.ThreadPool ] [node_s4] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x00007b49d884b598@c30f50f] on thread pool [generic] 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s4/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:35:05.730680516Z) 1> at org.apache.lucene.st 1> ore.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:941) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:19,314][WARN ][o.o.t.ThreadPool ] [node_s3] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x00007b49d884b598@44e26291] on thread pool [generic] 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s3/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:35:02.283609087Z) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:941) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:17,053][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:35588, remoteAddress=127.0.0.1/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete 1> .tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:18,133][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:37344, remoteAddress=127.0.0.1/127.0.0.1:45155}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:18,183][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:40200, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:18,154][WARN ][o.o.t.OutboundHandler ] [node_s0] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:55680, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java 1> .base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T21:53:16,550][WARN ][o.o.i.c.IndicesClusterStateService] [node_s3] [foo][8] marking and sending shard failed due to [shard failure, reason [refresh failed source[api]]] 1> java.nio.channels.ClosedChannelException: null 1> at java.base/sun.nio.ch.FileLockImpl.release(FileLockImpl.java:58) ~[?:?] 1> at java.base/java.nio.channels.FileLock.close(FileLock.java:336) ~[?:?] 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.close(NativeFSLockFactory.java:195) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.util.IOUtils.close(IOUtils.java:85) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.util.IOUtils.close(IOUtils.java:72) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2561) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2466) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:5774) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:719) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:72) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:434) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:414) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1887) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1864) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.refresh(IndexShard.java:1407) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.lambda$shardOperationOnPrimary$0(TransportShardRefreshAction.java:101) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:57) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1333) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:150) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:654) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:547) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$36(IndexShard.java:4194) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener$3.onResponse(ActionListener.java:132) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:310) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:255) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:4165) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1262) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:544) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:483) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1071) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/ja 1> va.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:18,134][WARN ][o.o.t.OutboundHandler ] [node_s2] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:55664, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:20,530][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:57754, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:20,439][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:38082, remoteAddress=127.0.0.1/127.0.0.1:45869}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~ 1> [?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:20,099][WARN ][o.o.t.OutboundHandler ] [node_s5] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:50398, remoteAddress=127.0.0.1/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:19,941][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:50388, remoteAddress=127.0.0.1/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:19,592][WARN ][o.o.c.NodeConnectionsService] [node_s0] failed to connect to {node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true} (tried [1] times) 1> org.opensearch.transport.ConnectTransportException: [node_s3][127.0.0.1:45155] failure to send internal:tcp/handshake 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 20 more 1> [2025-05-08T13:53:19,651][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:57774, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:20,924][WARN ][o.o.t.ThreadPool ] [node_s0] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x00007b49d884b598@2f4df60d] on thread pool [generic] 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s0/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:32:59.594066779Z) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:941) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:21,159][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:34626, remoteAddress=127.0.0.1/127.0.0.1:41581}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:21,306][WARN ][o.o.t.ThreadPool ] [node_s2] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x00007b49d884b598@482791d3] on thread pool [generic] 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s2/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:34:59.176544704Z) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:941) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T21:53:21,290][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:55730, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:856) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.action.shard.ShardStateAction.sendShardAction(ShardStateAction.java:192) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.action.shard.ShardStateAction.localShardFailed(ShardStateAction.java:298) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService.sendFailShard(IndicesClusterStateService.java:845) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService.failAndRemoveShard(IndicesClusterStateService.java:834) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService$FailedShardHandler.lambda$accept$0(IndicesClusterStateService.java:866) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:21,301][WARN ][o.o.c.NodeConnectionsService] [node_s4] failed to connect to {node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true} (tried [1] times) 1> org.opensearch.transport.ConnectTransportException: [node_s1][127.0.0.1:43491] failure to send internal:tcp/handshake 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch 1> -2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 20 more 1> [2025-05-08T13:53:21,321][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:40910, remoteAddress=127.0.0.1/127.0.0.1:40859}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:21,384][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:35220, remoteAddress=127.0.0.1/127.0.0.1:40859}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInt 1> ernal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:21,454][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=3, [cluster.fault_detection.follower_check.retry_count]=3} failed too many times 1> org.opensearch.transport.SendRequestTransportException: [node_s3][127.0.0.1:45155][internal:coordination/fault_detection/follower_check] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 12 more 1> [2025-05-08T13:53:21,557][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:40916, remoteAddress=127.0.0.1/127.0.0.1:40859}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:21,680][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:56128, remoteAddress=127.0.0.1/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:21,677][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s5}{7Bx5ewP5Q42abT6xeM4uBA}{OVcy3CfWRJeYN8aHI5nZRg}{127.0.0.1}{127.0.0.1:41581}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=3, [cluster.fault_detection.follower_check.retry_count]=3} failed too many times 1> org.opensearch.transport.SendRequestTransportException: [node_s5][127.0.0.1:41581][internal:coordination/fault_detection/follower_check] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 12 more 1> [2025-05-08T13:53:21,712][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=3, [cluster.fault_detection.follower_check.retry_count]=3} failed too many times 1> org.opensearch.transport.SendRequestTransportException: [node_s4][127.0.0.1:45869][internal:coordination/fault_detection/follower_check] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 12 more 1> [2025-05-08T13:53:21,654][WARN ][o.o.t.OutboundHandler ] [node_s5] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:57790, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:21,824][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:57792, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:21,824][WARN ][o.o.t.OutboundHandler ] [node_s5] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:50406, remoteAddress=127.0.0.1/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:21,865][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:50420, remoteAddress=/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:21,889][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=3, [cluster.fault_detection.follower_check.retry_count]=3} failed too many times 1> org.opensearch.transport.SendRequestTransportException: [node_s0][127.0.0.1:40859][internal:coordination/fault_detection/follower_check] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opense 1> arch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 12 more 1> [2025-05-08T13:53:21,933][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:57804, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:21,972][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:50426, remoteAddress=127.0.0.1/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T21:53:21,900][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=3, [cluster.fault_detection.follower_check.retry_count]=3} failed too many times 1> org.opensearch.transport.SendRequestTransportException: [node_s2][127.0.0.1:39703][internal:coordination/fault_detection/follower_check] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:366) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker$3.run(FollowersChecker.java:453) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at o 1> rg.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 12 more 1> [2025-05-08T13:53:22,002][WARN ][o.o.t.OutboundHandler ] [node_s0] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:55710, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T21:53:21,909][WARN ][o.o.c.a.s.ShardStateAction] [node_s3] unexpected failure while sending request [internal:cluster/shard/failure] to [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}] for shard entry [shard id [[foo][8]], allocation id [cGyJdexFQSGtYywGywBJ4w], primary term [0], message [shard failure, reason [refresh failed source[api]]], failure [ClosedChannelException[null]], markAsStale [true]] 1> org.opensearch.transport.SendRequestTransportException: [node_s1][127.0.0.1:43491][internal:cluster/shard/failure] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:856) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.action.shard.ShardStateAction.sendShardAction(ShardStateAction.java:192) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.action.shard.ShardStateAction.localShardFailed(ShardStateAction.java:298) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService.sendFailShard(IndicesClusterStateService.java:845) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService.failAndRemoveShard(IndicesClusterStateService.java:834) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.indices.cluster.IndicesClusterStateService$FailedShardHandler.lambda$accept$0(IndicesClusterStateService.java:866) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opense 1> arch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> ... 12 more 1> [2025-05-08T21:53:22,003][WARN ][o.o.t.ThreadPool ] [node_s1] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x00007b49d884b598@15554127] on thread pool [generic] 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s1/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:34:54.491447622Z) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:941) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 2> NOTE: test params are: codec=Lucene912, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=hi-IN, timezone=SystemV/PST8 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:22,205][WARN ][o.o.t.OutboundHandler ] [node_s2] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:55688, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:22,054][WARN ][o.o.t.OutboundHandler ] [node_s5] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:40922, remoteAddress=127.0.0.1/127.0.0.1:40859}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T21:53:22,169][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:37340, remoteAddress=127.0.0.1/127.0.0.1:45155}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:270) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:153) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:70) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:220) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.TransportAction.execute(TransportAction.java:109) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:112) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:99) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:480) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:792) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.support.AbstractClient$ClusterAdmin.nodesStats(AbstractClient.java:890) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.InternalClusterInfoService.updateNodeStats(InternalClusterInfoService.java:231) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:260) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.InternalClusterInfoService$RefreshRunnable.doRun(InternalClusterInfoService.java:520) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.InternalClusterInfoService$RefreshAndRescheduleRunnable.doRun(InternalClusterInfoService.java:552) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:22,436][INFO ][o.o.c.c.Coordinator ] [node_s2] cluster-manager node [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}] failed, restarting discovery 1> org.opensearch.OpenSearchException: node [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}] failed [3] consecutive checks 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:329) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.telemetry.tracing.handler.TraceableTransportResponseHandler.handleException(TraceableTransportResponseHandler.java:81) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1527) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$6.doRun(TransportService.java:1044) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: org.opensearch.transport.SendRequestTransportException: [node_s1][127.0.0.1:43491][internal:coordination/fault_detection/leader_check] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> ... 3 more 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> ... 3 more 1> [2025-05-08T13:53:22,454][INFO ][o.o.c.c.Coordinator ] [node_s0] cluster-manager node [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}] failed, restarting discovery 1> org.opensearch.OpenSearchException: node [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}] failed [3] consecutive checks 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:329) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.telemetry.tracing.handler.TraceableTransportResponseHandler.handleException(TraceableTransportResponseHandler.java:81) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1527) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService$6.doRun(TransportService.java:1044) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> Caused by: org.opensearch.transport.SendRequestTransportException: [node_s1][127.0.0.1:43491][internal:coordination/fault_detection/leader_check] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> ... 3 more 1> Caused by: java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:279) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] 1> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] 1> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 1> ... 3 more 1> [2025-05-08T21:53:22,641][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:38084, remoteAddress=127.0.0.1/127.0.0.1:45869}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:270) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:153) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:70) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:220) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.TransportAction.execute(TransportAction.java:109) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:112) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:99) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:480) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:792) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.support.AbstractClient$ClusterAdmin.nodesStats(AbstractClient.java:890) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.InternalClusterInfoService.updateNodeStats(InternalClusterInfoService.java:231) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:260) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.InternalClusterInfoService$RefreshRunnable.doRun(InternalClusterInfoService.java:520) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.InternalClusterInfoService$RefreshAndRescheduleRunnable.doRun(InternalClusterInfoService.java:552) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:22,775][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:57814, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:22,789][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:50444, remoteAddress=127.0.0.1/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:22,765][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:50442, remoteAddress=127.0.0.1/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFutu 1> re.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:22,811][WARN ][o.o.t.OutboundHandler ] [node_s5] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:50454, remoteAddress=127.0.0.1/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:22,807][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:57812, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTran 1> sport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:22,806][WARN ][o.o.t.OutboundHandler ] [node_s5] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:40926, remoteAddress=127.0.0.1/127.0.0.1:40859}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:22,994][WARN ][o.o.t.OutboundHandler ] [node_s0] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:57820, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:22,993][WARN ][o.o.t.OutboundHandler ] [node_s2] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:40930, remoteAddress=127.0.0.1/127.0.0.1:40859}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:22,992][WARN ][o.o.t.OutboundHandler ] [node_s0] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:50466, remoteAddress=127.0.0.1/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFutu 1> re.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T21:53:23,036][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:37330, remoteAddress=127.0.0.1/127.0.0.1:45155}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.sendNodeRequest(TransportBroadcastByNodeAction.java:370) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.start(TransportBroadcastByNodeAction.java:355) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:273) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:92) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:220) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.TransportAction.execute(TransportAction.java:109) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:112) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:99) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:480) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1546) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.support.AbstractClient$IndicesAdmin.stats(AbstractClient.java:1826) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.InternalClusterInfoService.updateIndicesStats(InternalClusterInfoService.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:301) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.InternalClusterInfoService$RefreshRunnable.doRun(InternalClusterInfoService.java:520) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.InternalClusterInfoService$RefreshAndRescheduleRunnable.doRun(InternalClusterInfoService.java:552) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:23,019][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:40942, remoteAddress=127.0.0.1/127.0.0.1:40859}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:23,091][WARN ][o.o.t.OutboundHandler ] [node_s2] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:57826, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFutu 1> re.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:23,088][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:40948, remoteAddress=127.0.0.1/127.0.0.1:40859}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:23,097][INFO ][o.o.c.s.ClusterApplierService] [node_s0] cluster-manager node changed {previous [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}], current []}, term: 1, version: 48, reason: becoming candidate: onLeaderFailure 1> [2025-05-08T13:53:23,098][INFO ][o.o.c.s.ClusterApplierService] [node_s2] cluster-manager node changed {previous [{node_s1}{yZZ3wNx4St-kTZad4B03Jg}{24Z-MQIaRf-KQiwEgtgoCA}{127.0.0.1}{127.0.0.1:43491}{imr}{shard_indexing_pressure_enabled=true}], current []}, term: 1, version: 48, reason: becoming candidate: onLeaderFailure 1> [2025-05-08T13:53:23,136][WARN ][o.o.t.OutboundHandler ] [node_s5] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:57836, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T21:53:23,252][WARN ][o.o.t.OutboundHandler ] [node_s1] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:38088, remoteAddress=127.0.0.1/127.0.0.1:45869}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:338) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:248) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1001) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1762) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:897) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportService.sendRequest(TransportService.java:874) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.sendNodeRequest(TransportBroadcastByNodeAction.java:370) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.start(TransportBroadcastByNodeAction.java:355) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:273) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:92) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:220) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.action.support.TransportAction.execute(TransportAction.java:109) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:112) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:99) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:480) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1546) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.client.support.AbstractClient$IndicesAdmin.stats(AbstractClient.java:1826) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.InternalClusterInfoService.updateIndicesStats(InternalClusterInfoService.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:301) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.InternalClusterInfoService$RefreshRunnable.doRun(InternalClusterInfoService.java:520) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.cluster.InternalClusterInfoService$RefreshAndRescheduleRunnable.doRun(InternalClusterInfoService.java:552) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:23,522][WARN ][o.o.t.OutboundHandler ] [node_s0] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:39370, remoteAddress=127.0.0.1/127.0.0.1:45155}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:23,740][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:40950, remoteAddress=127.0.0.1/127.0.0.1:40859}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:23,737][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:50468, remoteAddress=127.0.0.1/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:23,718][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:50476, remoteAddress=127.0.0.1/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:23,734][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:57842, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFutu 1> re.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:23,774][WARN ][o.o.t.OutboundHandler ] [node_s5] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:57858, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:23,774][WARN ][o.o.t.OutboundHandler ] [node_s5] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:40954, remoteAddress=127.0.0.1/127.0.0.1:40859}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:23,877][WARN ][o.o.t.OutboundHandler ] [node_s0] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:57868, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:23,906][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:57884, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:23,916][WARN ][o.o.t.OutboundHandler ] [node_s2] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:57888, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:23,921][WARN ][o.o.t.OutboundHandler ] [node_s0] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:50486, remoteAddress=127.0.0.1/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFutu 1> re.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:23,989][WARN ][o.o.t.OutboundHandler ] [node_s3] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:40968, remoteAddress=127.0.0.1/127.0.0.1:40859}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:23,939][WARN ][o.o.t.OutboundHandler ] [node_s2] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:40960, remoteAddress=127.0.0.1/127.0.0.1:40859}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTran 1> sport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:24,047][WARN ][o.o.t.OutboundHandler ] [node_s4] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:57880, remoteAddress=127.0.0.1/127.0.0.1:43491}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:24,057][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s3}{RyCADUdWTC-RdM7abPqxWQ}{1vZP3-WERtyuAbqIbPb7lg}{127.0.0.1}{127.0.0.1:45155}{dir}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=3, [cluster.fault_detection.follower_check.retry_count]=3} marking node as faulty 1> [2025-05-08T13:53:24,041][WARN ][o.o.t.OutboundHandler ] [node_s5] send message failed [channel: NioSocketChannel{localAddress=/127.0.0.1:50492, remoteAddress=127.0.0.1/127.0.0.1:39703}] 1> java.lang.RuntimeException: The rule is not currently executing. 1> at org.apache.lucene.tests.util.TestRuleStoreClassName.getTestClass(TestRuleStoreClassName.java:50) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.apache.lucene.tests.util.LuceneTestCase.getTestClass(LuceneTestCase.java:822) ~[lucene-test-framework-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:93) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:142) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.BigArrays.newByteArray(BigArrays.java:537) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:86) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:59) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:55) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:215) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:202) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:130) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:186) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendRequest(NativeOutboundHandler.java:121) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.lambda$new$1(TcpTransport.java:206) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TransportHandshaker.sendHandshake(TransportHandshaker.java:104) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport.executeHandshake(TcpTransport.java:900) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1049) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onResponse(TcpTransport.java:1022) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-core-2.19.1.jar:2.19.1] 1> at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:54) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] 1> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] 1> at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:76) ~[opensearch-common-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:165) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-2.19.1.jar:2.19.1] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:24,183][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s2}{3DHw9gajTnaQD82aEN-0fw}{LezWU5xoT3WFOoQcodvIiw}{127.0.0.1}{127.0.0.1:39703}{imr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=3, [cluster.fault_detection.follower_check.retry_count]=3} marking node as faulty 1> [2025-05-08T21:53:24,235][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s5}{7Bx5ewP5Q42abT6xeM4uBA}{OVcy3CfWRJeYN8aHI5nZRg}{127.0.0.1}{127.0.0.1:41581}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=3, [cluster.fault_detection.follower_check.retry_count]=3} marking node as faulty 2> NOTE: Linux 6.13.5-arch1-1 riscv64/Eclipse Adoptium 21.0.6 (64-bit)/cpus=12,threads=135,free=351162568,total=536870912 1> [2025-05-08T13:53:24,282][ERROR][o.o.m.f.FsHealthService ] [node_s5] health check failed 2> NOTE: All tests run in this JVM: [ICUCollationKeywordFieldMapperIT] 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s5/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:35:08.444736755Z) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodeDataPaths(NodeEnvironment.java:911) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsHealthService$FsHealthMonitor.monitorFSHealth(FsHealthService.java:212) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsHealthService$FsHealthMonitor.run(FsHealthService.java:195) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T21:53:24,292][ERROR][o.o.m.f.FsHealthService ] [node_s0] health check failed 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s0/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:32:59.594066779Z) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodeDataPaths(NodeEnvironment.java:911) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsHealthService$FsHealthMonitor.monitorFSHealth(FsHealthService.java:212) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsHealthService$FsHealthMonitor.run(FsHealthService.java:195) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:24,420][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s0}{48nm_5e6SW27203haHcNtA}{2R3dLqhRT4imeGvXN1B3kQ}{127.0.0.1}{127.0.0.1:40859}{imr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=3, [cluster.fault_detection.follower_check.retry_count]=3} marking node as faulty 1> [2025-05-08T13:53:24,299][ERROR][o.o.m.f.FsHealthService ] [node_s3] health check failed 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s3/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:35:02.283609087Z) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodeDataPaths(NodeEnvironment.java:911) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsHealthService$FsHealthMonitor.monitorFSHealth(FsHealthService.java:212) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsHealthService$FsHealthMonitor.run(FsHealthService.java:195) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:24,299][ERROR][o.o.m.f.FsHealthService ] [node_s1] health check failed 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s1/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:34:54.491447622Z) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodeDataPaths(NodeEnvironment.java:911) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsHealthService$FsHealthMonitor.monitorFSHealth(FsHealthService.java:212) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsHealthService$FsHealthMonitor.run(FsHealthService.java:195) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:24,294][ERROR][o.o.m.f.FsHealthService ] [node_s4] health check failed 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s4/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:35:05.730680516Z) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodeDataPaths(NodeEnvironment.java:911) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsHealthService$FsHealthMonitor.monitorFSHealth(FsHealthService.java:212) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsHealthService$FsHealthMonitor.run(FsHealthService.java:195) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:24,316][ERROR][o.o.m.f.FsHealthService ] [node_s2] health check failed 1> org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/testrun/internalClusterTest/temp/org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT_298D86C07ECB082C-001/tempDir-002/node_s2/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2025-05-08T21:34:59.176544704Z) 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165) ~[lucene-core-9.12.1.jar:9.12.1 7a97a05a239d6fb9f1f347aa09bfa52e875be092 - 2024-12-09 16:47:48] 1> at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1160) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.env.NodeEnvironment.nodeDataPaths(NodeEnvironment.java:911) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsHealthService$FsHealthMonitor.monitorFSHealth(FsHealthService.java:212) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.monitor.fs.FsHealthService$FsHealthMonitor.run(FsHealthService.java:195) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) ~[opensearch-2.19.1.jar:2.19.1] 1> at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.19.1.jar:2.19.1] 1> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] 1> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] 1> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 1> [2025-05-08T13:53:24,719][INFO ][o.o.c.c.FollowersChecker ] [node_s1] FollowerChecker{discoveryNode={node_s4}{MxhvN93ARFSXSbr2W4-NUQ}{qKVBTNNVT9ejQsJU8IjnEQ}{127.0.0.1}{127.0.0.1:45869}{dir}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=3, [cluster.fault_detection.follower_check.retry_count]=3} marking node as faulty Tests with failures: - org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT.testNormalization - org.opensearch.index.mapper.ICUCollationKeywordFieldMapperIT.classMethod 6 tests completed, 2 failed > Task :plugins:analysis-icu:internalClusterTest FAILED [Incubating] Problems report is available at: file:///build/opensearch/src/OpenSearch-2.19.1/build/reports/problems/problems-report.html FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':plugins:analysis-icu:internalClusterTest'. > There were failing tests. See the report at: file:///build/opensearch/src/OpenSearch-2.19.1/plugins/analysis-icu/build/reports/tests/internalClusterTest/index.html BUILD FAILED in 1h 3m 47s 229 actionable tasks: 59 executed, 170 up-to-date ==> ERROR: A failure occurred in build().  Aborting... ==> ERROR: Build failed, check /var/lib/archbuild/extra-riscv64/felix-1/build [?25h[?25h[?25hreceiving incremental file list opensearch-2.19.1-2-riscv64-build.log opensearch-2.19.1-2-riscv64-prepare.log sent 62 bytes received 96,952 bytes 194,028.00 bytes/sec total size is 1,817,280 speedup is 18.73