==>(B Building on aurorus(B ==>(B Checking for remote environment...(B ==>(B Syncing package to remote host...(B sending incremental file list ./ .SRCINFO 700 39% 0.00kB/s 0:00:00 1,772 100% 1.02MB/s 0:00:00 (xfr#1, to-chk=7/9) .nvchecker.toml 84 100% 82.03kB/s 0:00:00 84 100% 82.03kB/s 0:00:00 (xfr#2, to-chk=6/9) LICENSE 646 100% 630.86kB/s 0:00:00 646 100% 630.86kB/s 0:00:00 (xfr#3, to-chk=5/9) PKGBUILD 700 12% 683.59kB/s 0:00:00 5,506 100% 5.25MB/s 0:00:00 (xfr#4, to-chk=4/9) REUSE.toml 375 100% 366.21kB/s 0:00:00 375 100% 366.21kB/s 0:00:00 (xfr#5, to-chk=3/9) python-distributed-2026.8.0-1.log 817 100% 797.85kB/s 0:00:00 817 100% 797.85kB/s 0:00:00 (xfr#6, to-chk=2/9) LICENSES/ sent 1,521 bytes received 252 bytes 709.20 bytes/sec total size is 8,599 speedup is 4.85 ==>(B Applying RISC-V patches...(B sending incremental file list ./ fix-identity-json.patch 551 100% 0.00kB/s 0:00:00 551 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=6/8) fix-ipv6-scope.patch 1,219 100% 1.16MB/s 0:00:00 1,219 100% 1.16MB/s 0:00:00 (xfr#2, to-chk=5/8) fix-lost-dependency-accounting.patch 449 100% 438.48kB/s 0:00:00 449 100% 438.48kB/s 0:00:00 (xfr#3, to-chk=4/8) fix-popen-signal.patch 1,049 100% 1.00MB/s 0:00:00 1,049 100% 1.00MB/s 0:00:00 (xfr#4, to-chk=3/8) fix-test-cpu-count.patch 1,154 100% 1.10MB/s 0:00:00 1,154 100% 1.10MB/s 0:00:00 (xfr#5, to-chk=2/8) relax-test-timing.patch 700 6% 683.59kB/s 0:00:00 10,865 100% 5.18MB/s 0:00:00 (xfr#6, to-chk=1/8) riscv64.patch 2,850 100% 927.73kB/s 0:00:00 2,850 100% 927.73kB/s 0:00:00 (xfr#7, to-chk=0/8) sent 6,552 bytes received 224 bytes 4,517.33 bytes/sec total size is 18,137 speedup is 2.68 patching file PKGBUILD ==>(B Running pkgctl build --arch riscv64 on remote host...(B ==> WARNING:(B invalid architecture: riscv64(B ==>(B Updating pacman database cache(B [?25l:: Synchronizing package databases... core downloading... extra downloading... multilib downloading... [?25h==>(B Building python-distributed(B  ->(B repo: extra(B  ->(B arch: riscv64(B  ->(B worker: felix-3(B ==>(B Building python-distributed for [extra] (riscv64)(B ]3008;start=5848493057f7442fafba90741c8b156a;user=felix;hostname=aurorus.felixc.at;machineid=01ca1404c26b4b3a80152fb4f25f4ca7;bootid=62bf108769c34cbbb3b3844801966040;pid=519487;comm=sudo;targetuser=root;type=session\==>(B Locking clean chroot...(Bdone(B Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file. ]3008;start=beef862fa92d44509703bddc02dac251;user=root;hostname=aurorus.felixc.at;machineid=01ca1404c26b4b3a80152fb4f25f4ca7;bootid=62bf108769c34cbbb3b3844801966040;pid=519621;comm=systemd-nspawn;container=arch-nspawn-519621;type=container\]11;?\]2;🔵 Container arch-nspawn-519621 on aurorus.felixc.at\[?25l:: Synchronizing package databases... core downloading... extra downloading... :: Starting full system upgrade... there is nothing to do [?25h[!p]104\[?7h]3008;end=beef862fa92d44509703bddc02dac251\==>(B Building in chroot for [extra] (riscv64)...(B ==>(B Synchronizing chroot copy [/var/lib/archbuild/extra-riscv64/root] -> [felix-3]...(Bdone(B ]3008;start=4772b8c428e2492da2367f7949e4d98f;user=root;hostname=aurorus.felixc.at;machineid=01ca1404c26b4b3a80152fb4f25f4ca7;bootid=62bf108769c34cbbb3b3844801966040;pid=521641;comm=sudo;targetuser=felix;type=session\==>(B Making package: python-distributed 2026.8.0-1 (Tue Sep 8 21:48:26 2026)(B ==>(B Retrieving sources...(B  ->(B Found python-distributed-2026.8.0.tar.gz(B  ->(B Found fix-identity-json.patch(B  ->(B Found fix-lost-dependency-accounting.patch(B  ->(B Found fix-test-cpu-count.patch(B  ->(B Found relax-test-timing.patch(B  ->(B Found fix-ipv6-scope.patch(B  ->(B Found fix-popen-signal.patch(B ==>(B Validating source files with b2sums...(B python-distributed-2026.8.0.tar.gz ... Passed fix-identity-json.patch ... Passed fix-lost-dependency-accounting.patch ... Passed fix-test-cpu-count.patch ... Passed relax-test-timing.patch ... Passed fix-ipv6-scope.patch ... Passed fix-popen-signal.patch ... Passed ]3008;end=4772b8c428e2492da2367f7949e4d98f\Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file. ]3008;start=75832fa3e18a4dadbb386d1ec0d91f1a;user=root;hostname=aurorus.felixc.at;machineid=01ca1404c26b4b3a80152fb4f25f4ca7;bootid=62bf108769c34cbbb3b3844801966040;pid=522736;comm=systemd-nspawn;container=makechrootpkg-python-distributed.build.519508;type=container\]11;?\]2;🔵 Container makechrootpkg-python-distributed.build.519508 on aurorus.felixc.at\]3008;start=e71d3dabc47a43cfa0d58f062d2a3c48;user=root;hostname=makechrootpkg-python-distributed.build.519508;machineid=0d405df5a1cc4a8aa1ca96c60fd93bc2;bootid=ffc539e47d1c4e9699f6a2118caae337;pid=14;comm=sudo;targetuser=builduser;type=session\==>(B Making package: python-distributed 2026.8.0-1 (Tue Sep 8 21:48:51 2026)(B ==>(B Checking runtime dependencies...(B ==>(B Installing missing dependencies...(B ]3008;start=a3fb9fba18df4eb182b00abaa8bd0e89;user=builduser;hostname=makechrootpkg-python-distributed.build.519508;machineid=0d405df5a1cc4a8aa1ca96c60fd93bc2;bootid=ffc539e47d1c4e9699f6a2118caae337;pid=890;comm=sudo;targetuser=root;type=session\[?25lresolving dependencies... looking for conflicting packages... Package (20) New Version Net Change extra/libyaml 0.2.5-3 0.16 MiB extra/python-fsspec 2026.7.0-1 2.58 MiB extra/python-heapdict 1.0.1-9 0.02 MiB extra/python-markupsafe 3.0.3-1 0.09 MiB extra/python-partd 1.4.2-2 0.20 MiB extra/python-click 8.3.3-1 1.38 MiB extra/python-cloudpickle 3.1.2-3 0.18 MiB extra/python-dask 2026.8.0-1 22.43 MiB extra/python-jinja 1:3.1.6-3.1 2.04 MiB extra/python-locket 1.0.0-7 0.03 MiB extra/python-msgpack 1.1.2-2 0.30 MiB extra/python-packaging 26.3-1 1.58 MiB extra/python-psutil 7.2.2-1 1.14 MiB extra/python-sortedcontainers 2.4.0-8 0.38 MiB extra/python-tblib 3.2.2-2 0.10 MiB extra/python-toolz 1.1.0-2 0.74 MiB extra/python-tornado 6.5.8-1 6.22 MiB extra/python-urllib3 2.6.3-1 1.44 MiB extra/python-yaml 6.0.3-2 0.93 MiB extra/python-zict 3.0.0-5 0.62 MiB Total Installed Size: 42.56 MiB :: Proceed with installation? [Y/n] checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... installing python-click... installing python-cloudpickle... installing python-fsspec... Optional dependencies for python-fsspec python-aiohttp: HTTP support python-distributed: Dask support python-libarchive-c: archives support python-lz4: LZ4 compression support python-paramiko: SFTP support python-pyarrow: Arrow/Parquet support python-pygit2: git support python-requests: web protocols support python-smbprotocol: SMB support python-snappy: snappy compression support python-tqdm: progress bar support installing python-packaging... installing python-locket... installing python-toolz... installing python-partd... Optional dependencies for python-partd python-blosc python-numpy python-pandas python-pyzmq installing libyaml... installing python-yaml... installing python-dask... Optional dependencies for python-dask python-distributed: distributed computing python-graphviz: graph visualization using the graphviz engine python-jinja: HTML representations of Dask objects in Jupyter notebooks [pending] python-lz4: transparent use of lz4 compression algorithm python-matplotlib: color map support for graph visualization python-numpy: arrays and dataframes support python-pandas: dataframes support python-psutil: more accurate CPU count [pending] python-pyarrow: Apache Arrow, for parquet support python-scipy: for dask.array.stats, dask.array.fft, and dask.array.linalg.lu python-snappy: snappy compression for reading/writing Avro or Parquet files python-sqlalchemy: writing and reading from SQL databases python-xarray: N-D labeled arrays and datasets python-xxhash: faster hashing of arrays installing python-markupsafe... installing python-jinja... Optional dependencies for python-jinja python-babel: for i18n support installing python-msgpack... installing python-psutil... installing python-sortedcontainers... installing python-tblib... installing python-tornado... Optional dependencies for python-tornado python-pycurl: for tornado.curl_httpclient python-twisted: for tornado.platform.twisted installing python-urllib3... Optional dependencies for python-urllib3 python-brotli: Brotli support python-brotlicffi: Brotli support python-h2: HTTP/2 support python-pysocks: SOCKS support installing python-heapdict... installing python-zict... :: Running post-transaction hooks... (1/1) Arming ConditionNeedsUpdate... [?25h]3008;end=a3fb9fba18df4eb182b00abaa8bd0e89\==>(B Checking buildtime dependencies...(B ==>(B Installing missing dependencies...(B ]3008;start=5adbc497f975446cae7f95f2fac4538c;user=builduser;hostname=makechrootpkg-python-distributed.build.519508;machineid=0d405df5a1cc4a8aa1ca96c60fd93bc2;bootid=ffc539e47d1c4e9699f6a2118caae337;pid=1004;comm=sudo;targetuser=root;type=session\[?25lresolving dependencies... looking for conflicting packages... warning: insufficient columns available for table display Packages (124) abseil-cpp-20260817.0-2 apache-orc-2.3.1-4 arrow-25.0.1-4 aws-c-auth-0.10.1-1 aws-c-cal-0.9.13-1 aws-c-common-0.12.6-1 aws-c-compression-0.3.2-1 aws-c-event-stream-0.7.0-1 aws-c-http-0.10.14-1 aws-c-io-0.26.3-1 aws-c-mqtt-0.15.2-1 aws-c-s3-0.12.2-1 aws-c-sdkutils-0.2.4-1 aws-checksums-0.2.10-1 aws-crt-cpp-0.38.5-1 aws-sdk-cpp-core-1.11.792-1 aws-sdk-cpp-iam-1.11.792-1 aws-sdk-cpp-s3-1.11.792-1 blas-3.12.1-2 blosc-1.21.6-2 c-ares-1.34.8-1 cblas-3.12.1-2 gflags-2.2.2-6 google-glog-0.7.1-2 grpc-1.83.1-1 gtest-1.18.0-1 hdf5-2.2.0-1 lapack-3.12.1-2 libaec-1.1.7-1 liblzf-3.6-5 libpgm-5.3.128-4 libsodium-1.0.22-1 libutf8proc-2.11.3-1 libzip-1.11.4-1 netcdf-4.9.3-4 protobuf-36.1-1 python-argcomplete-3.7.2-1 python-asttokens-3.0.2-1 python-attrs-26.1.0-1 python-autocommand-2.2.2-9 python-bcrypt-5.0.0-2 python-certifi-2026.07.22-1 python-cffi-2.1.1-1 python-cftime-1.6.5-1 python-charset-normalizer-3.5.1-1 python-comm-0.2.3-2 python-cramjam-2.11.0-3 python-dateutil-2.9.0-8.1 python-decorator-5.3.1-1 python-execnet-2.1.2-3 python-executing-2.2.1-3.1 python-fluidity-0.2.1-8 python-idna-3.19-1 python-iniconfig-2.3.0-1 python-invoke-3.0.3-1 python-ipython-pygments-lexers-1.1.1-3.1 python-jaraco.collections-5.1.0-3 python-jaraco.context-6.1.2-1 python-jaraco.functools-4.1.0-3 python-jaraco.text-4.0.0-4 python-jedi-0.20.0-1 python-jsonschema-specifications-2025.9.1-2 python-jupyter-client-8.10.0-1.1 python-jupyter-core-5.9.1-2 python-lexicon-3.0.0-2 python-matplotlib-inline-0.2.2-1.1 python-more-itertools-11.1.0-1 python-nest-asyncio-1.6.0-5 python-parso-1:0.8.7-1 python-pexpect-4.9.0-7.1 python-pkg_resources-81.0.0-1 python-platformdirs-4.11.7-1 python-pluggy-1.6.0-3.1 python-pooch-1.9.0-1 python-prompt_toolkit-3.0.53-1 python-ptyprocess-0.7.0-9 python-pure-eval-0.2.3-3 python-pycparser-3.00-1 python-pygments-2.21.0-1 python-pynacl-1.6.2-1 python-pyproject-hooks-1.2.0-6 python-pytz-2026.1-1 python-pyzmq-27.2.0-1 python-referencing-0.37.0-3.1 python-rpds-py-2026.6.3-1 python-six-1.17.0-3 python-stack-data-0.6.3-5 python-traitlets-5.16.1-1 python-typing_extensions-4.16.0-1 python-vcs-versioning-2.3.4-1 python-wcwidth-0.8.3-1 re2-2:2025.11.05-6 s2n-tls-1.7.2-1 snappy-1.2.2-3 thrift-0.22.0-6 zeromq-4.3.5-3 ipython-9.17.0-1 python-blosc-1.11.4-1 python-build-1.6.0-1 python-cryptography-50.0.1-1 python-flaky-3.8.1-4 python-h5py-3.16.0-1 python-installer-1.0.1-1 python-ipykernel-7.3.0-1.1 python-ipywidgets-8.1.9-1 python-jsonschema-4.26.0-1.1 python-lz4-4.4.5-2 python-netcdf4-1.7.2-5 python-numpy-2.5.3-1 python-pandas-2.3.3-2 python-paramiko-5.0.0-1 python-prometheus_client-0.26.0-1 python-pyarrow-25.0.1-1 python-pytest-1:9.1.1-1 python-pytest-repeat-0.9.4-1 python-pytest-timeout-2.5.0-1 python-pytest-xdist-3.8.0-3 python-requests-2.34.2-1 python-scipy-1.18.1-1 python-setuptools-1:82.0.1-1 python-setuptools-scm-10.2.3-1 python-snappy-0.7.2-3 python-wheel-0.48.0-1 python-zstandard-0.25.0-3 Total Download Size: 0.59 MiB Total Installed Size: 568.13 MiB :: Proceed with installation? [Y/n] :: Retrieving packages... apache-orc-2.3.1-4-riscv64 downloading... python-matplotlib-inline-0.2.2-1.1-any downloading... checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... installing python-pyproject-hooks... installing python-build... Optional dependencies for python-build python-pip: to use as the Python package installer (default) python-uv: to use as the Python package installer python-virtualenv: to use virtualenv for build isolation installing python-installer... installing python-more-itertools... installing python-jaraco.functools... installing python-jaraco.context... installing python-autocommand... installing python-jaraco.text... Optional dependencies for python-jaraco.text python-inflect: for show-newlines script installing python-jaraco.collections... installing python-platformdirs... installing python-wheel... Optional dependencies for python-wheel python-keyring: for wheel.signatures python-xdg: for wheel.signatures python-setuptools: for legacy bdist_wheel subcommand [pending] installing python-typing_extensions... installing python-pkg_resources... installing python-setuptools... installing python-vcs-versioning... Optional dependencies for python-vcs-versioning python-rich: formatting of log messages installing python-setuptools-scm... Optional dependencies for python-setuptools-scm python-rich: use rich as console log handler installing python-decorator... installing python-pygments... installing python-ipython-pygments-lexers... installing python-parso... installing python-jedi... installing python-argcomplete... installing python-traitlets... installing python-matplotlib-inline... Optional dependencies for python-matplotlib-inline python-matplotlib installing python-ptyprocess... installing python-pexpect... installing python-wcwidth... installing python-prompt_toolkit... Optional dependencies for python-prompt_toolkit python-pygments: for color schemes and lexers [installed] installing python-asttokens... installing python-executing... installing python-pure-eval... installing python-stack-data... installing ipython... Optional dependencies for ipython python-black: to auto format with Black python-pickleshare: for the ip.db database yapf: to auto format with YAPF installing python-iniconfig... installing python-pluggy... installing python-pytest... installing python-pytest-repeat... installing python-pytest-timeout... installing python-execnet... installing python-pytest-xdist... Optional dependencies for python-pytest-xdist python-psutil: to use psutil for detection of the number of CPUs available [installed] python-setproctitle: to update the process title installing python-flaky... installing snappy... installing blosc... installing python-blosc... Optional dependencies for python-blosc python-numpy: for tests and benchmarks [pending] installing python-pycparser... installing python-cffi... Optional dependencies for python-cffi python-setuptools: "limited api" version checking in cffi.setuptools_ext [installed] installing python-cryptography... installing libaec... installing hdf5... installing liblzf... installing blas... installing cblas... installing lapack... installing python-numpy... Optional dependencies for python-numpy blas-openblas: faster linear algebra installing python-h5py... installing python-comm... installing python-six... installing python-dateutil... installing python-jupyter-core... installing libsodium... installing libpgm... installing zeromq... Optional dependencies for zeromq cppzmq: C++ binding for libzmq installing python-pyzmq... installing python-jupyter-client... installing python-nest-asyncio... installing python-ipykernel... Optional dependencies for python-ipykernel python-debugpy: debugger support installing python-ipywidgets... Optional dependencies for python-ipywidgets jupyterlab-widgets: widgets for Jupyter lab installing python-attrs... installing python-rpds-py... installing python-referencing... installing python-jsonschema-specifications... installing python-jsonschema... Optional dependencies for python-jsonschema python-isoduration: for duration format python-fqdn: for hostname format python-idna: for idn-hostname format [pending] python-jsonpointer: for json-pointer & relative-json-pointer format python-rfc3339-validator: for date-time format python-rfc3987: for iri, iri-reference, uri & uri-reference format python-uri-template: for uri-template format python-webcolors: for color format installing python-lz4... installing python-cftime... installing libzip... installing netcdf... Optional dependencies for netcdf netcdf-fortran: fortran bindings netcdf-cxx: c++ bindings installing python-certifi... installing python-netcdf4... installing python-pytz... installing python-pandas... Optional dependencies for python-pandas python-pandas-datareader: pandas.io.data replacement (recommended) python-numexpr: accelerating certain numerical operations (recommended) python-bottleneck: accelerating certain types of nan evaluations (recommended) python-matplotlib: plotting python-jinja: conditional formatting with DataFrame.style [installed] python-tabulate: printing in Markdown-friendly format python-scipy: miscellaneous statistical functions [pending] python-numba: alternative execution engine python-xarray: pandas-like API for N-dimensional data python-xlrd: Excel XLS input python-xlwt: Excel XLS output python-openpyxl: Excel XLSX input/output python-xlsxwriter: alternative Excel XLSX output python-beautifulsoup4: read_html function (in any case) python-html5lib: read_html function (and/or python-lxml) python-lxml: read_xml, to_xml and read_html function (and/or python-html5lib) python-sqlalchemy: SQL database support python-psycopg2: PostgreSQL engine for sqlalchemy python-pymysql: MySQL engine for sqlalchemy python-pytables: HDF5-based reading / writing python-blosc: for msgpack compression using blosc [installed] zlib: compression for msgpack [installed] python-pyarrow: Parquet, ORC and feather reading/writing [pending] python-fsspec: handling files aside from local and HTTP [installed] python-qtpy: read_clipboard function (only one needed) xclip: read_clipboard function (only one needed) xsel: read_clipboard function (only one needed) python-brotli: Brotli compression python-snappy: Snappy compression [pending] python-zstandard: Zstandard (zstd) compression [pending] installing python-bcrypt... installing python-pynacl... installing python-fluidity... installing python-lexicon... installing python-invoke... installing python-paramiko... installing python-prometheus_client... Optional dependencies for python-prometheus_client python-snappy: snappy compression support [pending] python-aiohttp: aiohttp server support python-django: django exporter support installing gtest... Optional dependencies for gtest python: gmock generator [installed] installing abseil-cpp... installing protobuf... installing apache-orc... installing aws-c-common... installing aws-c-cal... installing aws-c-compression... installing s2n-tls... installing aws-c-io... installing aws-c-http... installing aws-c-sdkutils... installing aws-c-auth... installing aws-checksums... installing aws-c-event-stream... installing aws-c-mqtt... installing aws-c-s3... installing aws-crt-cpp... installing aws-sdk-cpp-core... installing aws-sdk-cpp-iam... installing aws-sdk-cpp-s3... installing gflags... installing google-glog... installing c-ares... installing re2... installing grpc... installing libutf8proc... installing thrift... Optional dependencies for thrift qt5-base: TQTcpServer (Qt5) support installing arrow... installing python-pyarrow... Optional dependencies for python-pyarrow python-cffi: interact with C code [installed] python-pandas: Pandas integration [installed] python-fsspec: Filesystem Spec support [installed] installing python-charset-normalizer... installing python-idna... installing python-requests... Optional dependencies for python-requests python-chardet: alternative character encoding library python-pysocks: SOCKS proxy support installing python-pooch... Optional dependencies for python-pooch python-paramiko: for SFTP downloads [installed] python-tqdm: for printing a download progress bar installing python-scipy... Optional dependencies for python-scipy python-pillow: for image saving module installing python-cramjam... installing python-snappy... installing python-zstandard... Optional dependencies for python-zstandard python-cffi [installed] :: Running post-transaction hooks... (1/1) Arming ConditionNeedsUpdate... [?25h]3008;end=5adbc497f975446cae7f95f2fac4538c\==>(B Retrieving sources...(B  ->(B Found python-distributed-2026.8.0.tar.gz(B  ->(B Found fix-identity-json.patch(B  ->(B Found fix-lost-dependency-accounting.patch(B  ->(B Found fix-test-cpu-count.patch(B  ->(B Found relax-test-timing.patch(B  ->(B Found fix-ipv6-scope.patch(B  ->(B Found fix-popen-signal.patch(B ==> WARNING:(B Skipping all source file integrity checks.(B ==>(B Extracting sources...(B  ->(B Extracting python-distributed-2026.8.0.tar.gz with bsdtar(B ==>(B Starting prepare()...(B patching file distributed/http/scheduler/json.py patching file distributed/scheduler.py patching file distributed/utils.py patching file distributed/tests/test_worker.py patching file distributed/deploy/tests/test_slow_adaptive.py patching file distributed/cli/tests/test_dask_worker.py patching file distributed/cli/tests/test_tls_cli.py patching file distributed/deploy/tests/test_adaptive.py patching file distributed/deploy/tests/test_spec_cluster.py patching file distributed/deploy/tests/test_subprocess.py patching file distributed/diagnostics/tests/test_progress.py patching file distributed/tests/test_asyncprocess.py patching file distributed/tests/test_failed_workers.py patching file distributed/tests/test_priorities.py patching file distributed/tests/test_semaphore.py patching file distributed/tests/test_steal.py patching file distributed/tests/test_utils_test.py patching file distributed/tests/test_worker.py patching file distributed/tests/test_worker_memory.py patching file distributed/utils_test.py patching file pyproject.toml patching file distributed/tests/test_utils_test.py ==>(B Starting build()...(B * Getting build dependencies for wheel... running egg_info creating distributed.egg-info writing distributed.egg-info/PKG-INFO writing dependency_links to distributed.egg-info/dependency_links.txt writing entry points to distributed.egg-info/entry_points.txt writing requirements to distributed.egg-info/requires.txt writing top-level names to distributed.egg-info/top_level.txt writing manifest file 'distributed.egg-info/SOURCES.txt' reading manifest file 'distributed.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching '*.coffee' under directory 'distributed' no previously-included directories found matching '.github' no previously-included directories found matching 'continuous_integration' no previously-included directories found matching 'docs/_build' adding license file 'LICENSE.txt' writing manifest file 'distributed.egg-info/SOURCES.txt' * Building wheel... running bdist_wheel running build running build_py creating build/lib/distributed copying distributed/lock.py -> build/lib/distributed copying distributed/preloading.py -> build/lib/distributed copying distributed/semaphore.py -> build/lib/distributed copying distributed/metrics.py -> build/lib/distributed copying distributed/actor.py -> build/lib/distributed copying distributed/exceptions.py -> build/lib/distributed copying distributed/system.py -> build/lib/distributed copying distributed/worker_client.py -> build/lib/distributed copying distributed/profile.py -> build/lib/distributed copying distributed/queues.py -> build/lib/distributed copying distributed/process.py -> build/lib/distributed copying distributed/sizeof.py -> build/lib/distributed copying distributed/utils_test.py -> build/lib/distributed copying distributed/utils.py -> build/lib/distributed copying distributed/worker_state_machine.py -> build/lib/distributed copying distributed/node.py -> build/lib/distributed copying distributed/broker.py -> build/lib/distributed copying distributed/versions.py -> build/lib/distributed copying distributed/__init__.py -> build/lib/distributed copying distributed/collections.py -> build/lib/distributed copying distributed/_stories.py -> build/lib/distributed copying distributed/recreate_tasks.py -> build/lib/distributed copying distributed/utils_comm.py -> build/lib/distributed copying distributed/variable.py -> build/lib/distributed copying distributed/batched.py -> build/lib/distributed copying distributed/client.py -> build/lib/distributed copying distributed/gc.py -> build/lib/distributed copying distributed/worker.py -> build/lib/distributed copying distributed/event.py -> build/lib/distributed copying distributed/core.py -> build/lib/distributed copying distributed/_signals.py -> build/lib/distributed copying distributed/chaos.py -> build/lib/distributed copying distributed/cfexecutor.py -> build/lib/distributed copying distributed/nanny.py -> build/lib/distributed copying distributed/threadpoolexecutor.py -> build/lib/distributed copying distributed/diskutils.py -> build/lib/distributed copying distributed/itertools.py -> build/lib/distributed copying distributed/spans.py -> build/lib/distributed copying distributed/cluster_dump.py -> build/lib/distributed copying distributed/spill.py -> build/lib/distributed copying distributed/multi_lock.py -> build/lib/distributed copying distributed/scheduler.py -> build/lib/distributed copying distributed/pytest_resourceleaks.py -> build/lib/distributed copying distributed/config.py -> build/lib/distributed copying distributed/stealing.py -> build/lib/distributed copying distributed/publish.py -> build/lib/distributed copying distributed/active_memory_manager.py -> build/lib/distributed copying distributed/_asyncio.py -> build/lib/distributed copying distributed/_async_taskgroup.py -> build/lib/distributed copying distributed/bokeh.py -> build/lib/distributed copying distributed/_concurrent_futures_thread.py -> build/lib/distributed copying distributed/worker_memory.py -> build/lib/distributed copying distributed/system_monitor.py -> build/lib/distributed copying distributed/compatibility.py -> build/lib/distributed copying distributed/_version.py -> build/lib/distributed copying distributed/counter.py -> build/lib/distributed copying distributed/objects.py -> build/lib/distributed copying distributed/security.py -> build/lib/distributed copying distributed/proctitle.py -> build/lib/distributed creating build/lib/distributed/widgets copying distributed/widgets/__init__.py -> build/lib/distributed/widgets creating build/lib/distributed/deploy copying distributed/deploy/adaptive_core.py -> build/lib/distributed/deploy copying distributed/deploy/ssh.py -> build/lib/distributed/deploy copying distributed/deploy/utils.py -> build/lib/distributed/deploy copying distributed/deploy/adaptive.py -> build/lib/distributed/deploy copying distributed/deploy/__init__.py -> build/lib/distributed/deploy copying distributed/deploy/cluster.py -> build/lib/distributed/deploy copying distributed/deploy/old_ssh.py -> build/lib/distributed/deploy copying distributed/deploy/spec.py -> build/lib/distributed/deploy copying distributed/deploy/subprocess.py -> build/lib/distributed/deploy copying distributed/deploy/local.py -> build/lib/distributed/deploy creating build/lib/distributed/diagnostics copying distributed/diagnostics/memory_sampler.py -> build/lib/distributed/diagnostics copying distributed/diagnostics/websocket.py -> build/lib/distributed/diagnostics copying distributed/diagnostics/eventstream.py -> build/lib/distributed/diagnostics copying distributed/diagnostics/__init__.py -> build/lib/distributed/diagnostics copying distributed/diagnostics/nvml.py -> build/lib/distributed/diagnostics copying distributed/diagnostics/progress.py -> build/lib/distributed/diagnostics copying distributed/diagnostics/memray.py -> build/lib/distributed/diagnostics copying distributed/diagnostics/task_stream.py -> build/lib/distributed/diagnostics copying distributed/diagnostics/cudf.py -> build/lib/distributed/diagnostics copying distributed/diagnostics/rmm.py -> build/lib/distributed/diagnostics copying distributed/diagnostics/cluster_dump.py -> build/lib/distributed/diagnostics copying distributed/diagnostics/progress_stream.py -> build/lib/distributed/diagnostics copying distributed/diagnostics/graph_layout.py -> build/lib/distributed/diagnostics copying distributed/diagnostics/progressbar.py -> build/lib/distributed/diagnostics copying distributed/diagnostics/plugin.py -> build/lib/distributed/diagnostics creating build/lib/distributed/http copying distributed/http/statics.py -> build/lib/distributed/http copying distributed/http/prometheus.py -> build/lib/distributed/http copying distributed/http/utils.py -> build/lib/distributed/http copying distributed/http/__init__.py -> build/lib/distributed/http copying distributed/http/routing.py -> build/lib/distributed/http copying distributed/http/health.py -> build/lib/distributed/http copying distributed/http/proxy.py -> build/lib/distributed/http creating build/lib/distributed/shuffle copying distributed/shuffle/_buffer.py -> build/lib/distributed/shuffle copying distributed/shuffle/_limiter.py -> build/lib/distributed/shuffle copying distributed/shuffle/__init__.py -> build/lib/distributed/shuffle copying distributed/shuffle/_rechunk.py -> build/lib/distributed/shuffle copying distributed/shuffle/_core.py -> build/lib/distributed/shuffle copying distributed/shuffle/_worker_plugin.py -> build/lib/distributed/shuffle copying distributed/shuffle/_shuffle.py -> build/lib/distributed/shuffle copying distributed/shuffle/_comms.py -> build/lib/distributed/shuffle copying distributed/shuffle/_memory.py -> build/lib/distributed/shuffle copying distributed/shuffle/_arrow.py -> build/lib/distributed/shuffle copying distributed/shuffle/_pickle.py -> build/lib/distributed/shuffle copying distributed/shuffle/_disk.py -> build/lib/distributed/shuffle copying distributed/shuffle/_scheduler_plugin.py -> build/lib/distributed/shuffle copying distributed/shuffle/_merge.py -> build/lib/distributed/shuffle copying distributed/shuffle/_exceptions.py -> build/lib/distributed/shuffle creating build/lib/distributed/dashboard copying distributed/dashboard/utils.py -> build/lib/distributed/dashboard copying distributed/dashboard/__init__.py -> build/lib/distributed/dashboard copying distributed/dashboard/export_tool.py -> build/lib/distributed/dashboard copying distributed/dashboard/worker.py -> build/lib/distributed/dashboard copying distributed/dashboard/core.py -> build/lib/distributed/dashboard copying distributed/dashboard/scheduler.py -> build/lib/distributed/dashboard creating build/lib/distributed/protocol copying distributed/protocol/cuda.py -> build/lib/distributed/protocol copying distributed/protocol/numba.py -> build/lib/distributed/protocol copying distributed/protocol/utils_test.py -> build/lib/distributed/protocol copying distributed/protocol/utils.py -> build/lib/distributed/protocol copying distributed/protocol/__init__.py -> build/lib/distributed/protocol copying distributed/protocol/cupy.py -> build/lib/distributed/protocol copying distributed/protocol/torch.py -> build/lib/distributed/protocol copying distributed/protocol/pickle.py -> build/lib/distributed/protocol copying distributed/protocol/numpy.py -> build/lib/distributed/protocol copying distributed/protocol/keras.py -> build/lib/distributed/protocol copying distributed/protocol/h5py.py -> build/lib/distributed/protocol copying distributed/protocol/core.py -> build/lib/distributed/protocol copying distributed/protocol/netcdf4.py -> build/lib/distributed/protocol copying distributed/protocol/sparse.py -> build/lib/distributed/protocol copying distributed/protocol/rmm.py -> build/lib/distributed/protocol copying distributed/protocol/serialize.py -> build/lib/distributed/protocol copying distributed/protocol/arrow.py -> build/lib/distributed/protocol copying distributed/protocol/scipy.py -> build/lib/distributed/protocol copying distributed/protocol/compression.py -> build/lib/distributed/protocol creating build/lib/distributed/cli copying distributed/cli/dask_ssh.py -> build/lib/distributed/cli copying distributed/cli/utils.py -> build/lib/distributed/cli copying distributed/cli/__init__.py -> build/lib/distributed/cli copying distributed/cli/dask_spec.py -> build/lib/distributed/cli copying distributed/cli/dask_scheduler.py -> build/lib/distributed/cli copying distributed/cli/dask_worker.py -> build/lib/distributed/cli creating build/lib/distributed/comm copying distributed/comm/utils.py -> build/lib/distributed/comm copying distributed/comm/__init__.py -> build/lib/distributed/comm copying distributed/comm/registry.py -> build/lib/distributed/comm copying distributed/comm/tcp.py -> build/lib/distributed/comm copying distributed/comm/inproc.py -> build/lib/distributed/comm copying distributed/comm/core.py -> build/lib/distributed/comm copying distributed/comm/ucx.py -> build/lib/distributed/comm copying distributed/comm/addressing.py -> build/lib/distributed/comm copying distributed/comm/ws.py -> build/lib/distributed/comm creating build/lib/distributed/widgets/templates copying distributed/widgets/templates/__init__.py -> build/lib/distributed/widgets/templates creating build/lib/distributed/http/worker copying distributed/http/worker/__init__.py -> build/lib/distributed/http/worker creating build/lib/distributed/http/scheduler copying distributed/http/scheduler/__init__.py -> build/lib/distributed/http/scheduler copying distributed/http/scheduler/api.py -> build/lib/distributed/http/scheduler copying distributed/http/scheduler/json.py -> build/lib/distributed/http/scheduler copying distributed/http/scheduler/missing_bokeh.py -> build/lib/distributed/http/scheduler copying distributed/http/scheduler/info.py -> build/lib/distributed/http/scheduler creating build/lib/distributed/http/templates copying distributed/http/templates/__init__.py -> build/lib/distributed/http/templates creating build/lib/distributed/http/static copying distributed/http/static/__init__.py -> build/lib/distributed/http/static creating build/lib/distributed/http/worker/prometheus copying distributed/http/worker/prometheus/__init__.py -> build/lib/distributed/http/worker/prometheus copying distributed/http/worker/prometheus/core.py -> build/lib/distributed/http/worker/prometheus creating build/lib/distributed/http/scheduler/prometheus copying distributed/http/scheduler/prometheus/semaphore.py -> build/lib/distributed/http/scheduler/prometheus copying distributed/http/scheduler/prometheus/__init__.py -> build/lib/distributed/http/scheduler/prometheus copying distributed/http/scheduler/prometheus/core.py -> build/lib/distributed/http/scheduler/prometheus copying distributed/http/scheduler/prometheus/stealing.py -> build/lib/distributed/http/scheduler/prometheus creating build/lib/distributed/http/static/js copying distributed/http/static/js/__init__.py -> build/lib/distributed/http/static/js creating build/lib/distributed/http/static/css copying distributed/http/static/css/__init__.py -> build/lib/distributed/http/static/css creating build/lib/distributed/http/static/images copying distributed/http/static/images/__init__.py -> build/lib/distributed/http/static/images creating build/lib/distributed/dashboard/templates copying distributed/dashboard/templates/__init__.py -> build/lib/distributed/dashboard/templates creating build/lib/distributed/dashboard/components copying distributed/dashboard/components/__init__.py -> build/lib/distributed/dashboard/components copying distributed/dashboard/components/nvml.py -> build/lib/distributed/dashboard/components copying distributed/dashboard/components/shared.py -> build/lib/distributed/dashboard/components copying distributed/dashboard/components/worker.py -> build/lib/distributed/dashboard/components copying distributed/dashboard/components/rmm.py -> build/lib/distributed/dashboard/components copying distributed/dashboard/components/scheduler.py -> build/lib/distributed/dashboard/components running egg_info writing distributed.egg-info/PKG-INFO writing dependency_links to distributed.egg-info/dependency_links.txt writing entry points to distributed.egg-info/entry_points.txt writing requirements to distributed.egg-info/requires.txt writing top-level names to distributed.egg-info/top_level.txt reading manifest file 'distributed.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching '*.coffee' under directory 'distributed' no previously-included directories found matching '.github' no previously-included directories found matching 'continuous_integration' no previously-included directories found matching 'docs/_build' adding license file 'LICENSE.txt' writing manifest file 'distributed.egg-info/SOURCES.txt' copying distributed/distributed-schema.yaml -> build/lib/distributed copying distributed/distributed.yaml -> build/lib/distributed copying distributed/py.typed -> build/lib/distributed copying distributed/http/templates/status.html -> build/lib/distributed/http/templates copying distributed/http/templates/task.html -> build/lib/distributed/http/templates copying distributed/http/templates/workers.html -> build/lib/distributed/http/templates copying distributed/http/templates/worker-table.html -> build/lib/distributed/http/templates copying distributed/http/templates/base.html -> build/lib/distributed/http/templates copying distributed/http/templates/main.html -> build/lib/distributed/http/templates copying distributed/http/templates/gpu.html -> build/lib/distributed/http/templates copying distributed/http/templates/simple.html -> build/lib/distributed/http/templates copying distributed/http/templates/worker.html -> build/lib/distributed/http/templates copying distributed/http/templates/exceptions.html -> build/lib/distributed/http/templates copying distributed/http/templates/logs.html -> build/lib/distributed/http/templates copying distributed/http/templates/json-index.html -> build/lib/distributed/http/templates copying distributed/http/templates/call-stack.html -> build/lib/distributed/http/templates copying distributed/dashboard/export_tool.js -> build/lib/distributed/dashboard copying distributed/dashboard/theme.yaml -> build/lib/distributed/dashboard copying distributed/widgets/templates/client.html.j2 -> build/lib/distributed/widgets/templates copying distributed/widgets/templates/cluster.html.j2 -> build/lib/distributed/widgets/templates copying distributed/widgets/templates/computation.html.j2 -> build/lib/distributed/widgets/templates copying distributed/widgets/templates/future.html.j2 -> build/lib/distributed/widgets/templates copying distributed/widgets/templates/has_what.html.j2 -> build/lib/distributed/widgets/templates copying distributed/widgets/templates/local_cluster.html.j2 -> build/lib/distributed/widgets/templates copying distributed/widgets/templates/log.html.j2 -> build/lib/distributed/widgets/templates copying distributed/widgets/templates/logs.html.j2 -> build/lib/distributed/widgets/templates copying distributed/widgets/templates/process_interface.html.j2 -> build/lib/distributed/widgets/templates copying distributed/widgets/templates/scheduler.html.j2 -> build/lib/distributed/widgets/templates copying distributed/widgets/templates/scheduler_info.html.j2 -> build/lib/distributed/widgets/templates copying distributed/widgets/templates/security.html.j2 -> build/lib/distributed/widgets/templates copying distributed/widgets/templates/task_state.html.j2 -> build/lib/distributed/widgets/templates copying distributed/widgets/templates/who_has.html.j2 -> build/lib/distributed/widgets/templates copying distributed/widgets/templates/worker_state.html.j2 -> build/lib/distributed/widgets/templates copying distributed/http/static/individual-cluster-map.html -> build/lib/distributed/http/static copying distributed/http/static/js/anime.min.js -> build/lib/distributed/http/static/js copying distributed/http/static/js/individual-cluster-map.js -> build/lib/distributed/http/static/js copying distributed/http/static/js/reconnecting-websocket.min.js -> build/lib/distributed/http/static/js copying distributed/http/static/js/sortable.min.js -> build/lib/distributed/http/static/js copying distributed/http/static/css/base.css -> build/lib/distributed/http/static/css copying distributed/http/static/css/gpu.css -> build/lib/distributed/http/static/css copying distributed/http/static/css/individual-cluster-map.css -> build/lib/distributed/http/static/css copying distributed/http/static/css/sortable.min.css -> build/lib/distributed/http/static/css copying distributed/http/static/css/status.css -> build/lib/distributed/http/static/css copying distributed/http/static/images/dask-logo.svg -> build/lib/distributed/http/static/images copying distributed/http/static/images/fa-bars.svg -> build/lib/distributed/http/static/images copying distributed/http/static/images/favicon.ico -> build/lib/distributed/http/static/images copying distributed/http/static/images/jupyter.svg -> build/lib/distributed/http/static/images copying distributed/http/static/images/numpy.png -> build/lib/distributed/http/static/images copying distributed/http/static/images/pandas.png -> build/lib/distributed/http/static/images copying distributed/http/static/images/python.png -> build/lib/distributed/http/static/images copying distributed/dashboard/templates/file.html -> build/lib/distributed/dashboard/templates copying distributed/dashboard/templates/performance_report.html -> build/lib/distributed/dashboard/templates installing to build/bdist.linux-riscv64/wheel running install running install_lib creating build/bdist.linux-riscv64/wheel creating build/bdist.linux-riscv64/wheel/distributed copying build/lib/distributed/lock.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/preloading.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/semaphore.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/metrics.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/actor.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/exceptions.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/system.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/worker_client.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/profile.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/queues.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/process.py -> build/bdist.linux-riscv64/wheel/./distributed creating build/bdist.linux-riscv64/wheel/distributed/widgets copying build/lib/distributed/widgets/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/widgets creating build/bdist.linux-riscv64/wheel/distributed/widgets/templates copying build/lib/distributed/widgets/templates/task_state.html.j2 -> build/bdist.linux-riscv64/wheel/./distributed/widgets/templates copying build/lib/distributed/widgets/templates/cluster.html.j2 -> build/bdist.linux-riscv64/wheel/./distributed/widgets/templates copying build/lib/distributed/widgets/templates/computation.html.j2 -> build/bdist.linux-riscv64/wheel/./distributed/widgets/templates copying build/lib/distributed/widgets/templates/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/widgets/templates copying build/lib/distributed/widgets/templates/log.html.j2 -> build/bdist.linux-riscv64/wheel/./distributed/widgets/templates copying build/lib/distributed/widgets/templates/scheduler.html.j2 -> build/bdist.linux-riscv64/wheel/./distributed/widgets/templates copying build/lib/distributed/widgets/templates/logs.html.j2 -> build/bdist.linux-riscv64/wheel/./distributed/widgets/templates copying build/lib/distributed/widgets/templates/has_what.html.j2 -> build/bdist.linux-riscv64/wheel/./distributed/widgets/templates copying build/lib/distributed/widgets/templates/scheduler_info.html.j2 -> build/bdist.linux-riscv64/wheel/./distributed/widgets/templates copying build/lib/distributed/widgets/templates/local_cluster.html.j2 -> build/bdist.linux-riscv64/wheel/./distributed/widgets/templates copying build/lib/distributed/widgets/templates/security.html.j2 -> build/bdist.linux-riscv64/wheel/./distributed/widgets/templates copying build/lib/distributed/widgets/templates/worker_state.html.j2 -> build/bdist.linux-riscv64/wheel/./distributed/widgets/templates copying build/lib/distributed/widgets/templates/process_interface.html.j2 -> build/bdist.linux-riscv64/wheel/./distributed/widgets/templates copying build/lib/distributed/widgets/templates/client.html.j2 -> build/bdist.linux-riscv64/wheel/./distributed/widgets/templates copying build/lib/distributed/widgets/templates/future.html.j2 -> build/bdist.linux-riscv64/wheel/./distributed/widgets/templates copying build/lib/distributed/widgets/templates/who_has.html.j2 -> build/bdist.linux-riscv64/wheel/./distributed/widgets/templates copying build/lib/distributed/sizeof.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/utils_test.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/utils.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/worker_state_machine.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/node.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/broker.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/versions.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/collections.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/_stories.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/recreate_tasks.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/utils_comm.py -> build/bdist.linux-riscv64/wheel/./distributed creating build/bdist.linux-riscv64/wheel/distributed/deploy copying build/lib/distributed/deploy/adaptive_core.py -> build/bdist.linux-riscv64/wheel/./distributed/deploy copying build/lib/distributed/deploy/ssh.py -> build/bdist.linux-riscv64/wheel/./distributed/deploy copying build/lib/distributed/deploy/utils.py -> build/bdist.linux-riscv64/wheel/./distributed/deploy copying build/lib/distributed/deploy/adaptive.py -> build/bdist.linux-riscv64/wheel/./distributed/deploy copying build/lib/distributed/deploy/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/deploy copying build/lib/distributed/deploy/cluster.py -> build/bdist.linux-riscv64/wheel/./distributed/deploy copying build/lib/distributed/deploy/old_ssh.py -> build/bdist.linux-riscv64/wheel/./distributed/deploy copying build/lib/distributed/deploy/spec.py -> build/bdist.linux-riscv64/wheel/./distributed/deploy copying build/lib/distributed/deploy/subprocess.py -> build/bdist.linux-riscv64/wheel/./distributed/deploy copying build/lib/distributed/deploy/local.py -> build/bdist.linux-riscv64/wheel/./distributed/deploy copying build/lib/distributed/variable.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/batched.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/client.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/distributed.yaml -> build/bdist.linux-riscv64/wheel/./distributed creating build/bdist.linux-riscv64/wheel/distributed/diagnostics copying build/lib/distributed/diagnostics/memory_sampler.py -> build/bdist.linux-riscv64/wheel/./distributed/diagnostics copying build/lib/distributed/diagnostics/websocket.py -> build/bdist.linux-riscv64/wheel/./distributed/diagnostics copying build/lib/distributed/diagnostics/eventstream.py -> build/bdist.linux-riscv64/wheel/./distributed/diagnostics copying build/lib/distributed/diagnostics/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/diagnostics copying build/lib/distributed/diagnostics/nvml.py -> build/bdist.linux-riscv64/wheel/./distributed/diagnostics copying build/lib/distributed/diagnostics/progress.py -> build/bdist.linux-riscv64/wheel/./distributed/diagnostics copying build/lib/distributed/diagnostics/memray.py -> build/bdist.linux-riscv64/wheel/./distributed/diagnostics copying build/lib/distributed/diagnostics/task_stream.py -> build/bdist.linux-riscv64/wheel/./distributed/diagnostics copying build/lib/distributed/diagnostics/cudf.py -> build/bdist.linux-riscv64/wheel/./distributed/diagnostics copying build/lib/distributed/diagnostics/rmm.py -> build/bdist.linux-riscv64/wheel/./distributed/diagnostics copying build/lib/distributed/diagnostics/cluster_dump.py -> build/bdist.linux-riscv64/wheel/./distributed/diagnostics copying build/lib/distributed/diagnostics/progress_stream.py -> build/bdist.linux-riscv64/wheel/./distributed/diagnostics copying build/lib/distributed/diagnostics/graph_layout.py -> build/bdist.linux-riscv64/wheel/./distributed/diagnostics copying build/lib/distributed/diagnostics/progressbar.py -> build/bdist.linux-riscv64/wheel/./distributed/diagnostics copying build/lib/distributed/diagnostics/plugin.py -> build/bdist.linux-riscv64/wheel/./distributed/diagnostics copying build/lib/distributed/gc.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/worker.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/event.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/core.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/_signals.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/chaos.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/cfexecutor.py -> build/bdist.linux-riscv64/wheel/./distributed creating build/bdist.linux-riscv64/wheel/distributed/http copying build/lib/distributed/http/statics.py -> build/bdist.linux-riscv64/wheel/./distributed/http copying build/lib/distributed/http/prometheus.py -> build/bdist.linux-riscv64/wheel/./distributed/http creating build/bdist.linux-riscv64/wheel/distributed/http/worker copying build/lib/distributed/http/worker/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/http/worker creating build/bdist.linux-riscv64/wheel/distributed/http/worker/prometheus copying build/lib/distributed/http/worker/prometheus/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/http/worker/prometheus copying build/lib/distributed/http/worker/prometheus/core.py -> build/bdist.linux-riscv64/wheel/./distributed/http/worker/prometheus copying build/lib/distributed/http/utils.py -> build/bdist.linux-riscv64/wheel/./distributed/http copying build/lib/distributed/http/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/http creating build/bdist.linux-riscv64/wheel/distributed/http/scheduler copying build/lib/distributed/http/scheduler/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/http/scheduler copying build/lib/distributed/http/scheduler/api.py -> build/bdist.linux-riscv64/wheel/./distributed/http/scheduler copying build/lib/distributed/http/scheduler/json.py -> build/bdist.linux-riscv64/wheel/./distributed/http/scheduler copying build/lib/distributed/http/scheduler/missing_bokeh.py -> build/bdist.linux-riscv64/wheel/./distributed/http/scheduler creating build/bdist.linux-riscv64/wheel/distributed/http/scheduler/prometheus copying build/lib/distributed/http/scheduler/prometheus/semaphore.py -> build/bdist.linux-riscv64/wheel/./distributed/http/scheduler/prometheus copying build/lib/distributed/http/scheduler/prometheus/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/http/scheduler/prometheus copying build/lib/distributed/http/scheduler/prometheus/core.py -> build/bdist.linux-riscv64/wheel/./distributed/http/scheduler/prometheus copying build/lib/distributed/http/scheduler/prometheus/stealing.py -> build/bdist.linux-riscv64/wheel/./distributed/http/scheduler/prometheus copying build/lib/distributed/http/scheduler/info.py -> build/bdist.linux-riscv64/wheel/./distributed/http/scheduler creating build/bdist.linux-riscv64/wheel/distributed/http/templates copying build/lib/distributed/http/templates/status.html -> build/bdist.linux-riscv64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/task.html -> build/bdist.linux-riscv64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/workers.html -> build/bdist.linux-riscv64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/worker-table.html -> build/bdist.linux-riscv64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/base.html -> build/bdist.linux-riscv64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/main.html -> build/bdist.linux-riscv64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/gpu.html -> build/bdist.linux-riscv64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/simple.html -> build/bdist.linux-riscv64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/worker.html -> build/bdist.linux-riscv64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/exceptions.html -> build/bdist.linux-riscv64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/logs.html -> build/bdist.linux-riscv64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/json-index.html -> build/bdist.linux-riscv64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/call-stack.html -> build/bdist.linux-riscv64/wheel/./distributed/http/templates copying build/lib/distributed/http/routing.py -> build/bdist.linux-riscv64/wheel/./distributed/http copying build/lib/distributed/http/health.py -> build/bdist.linux-riscv64/wheel/./distributed/http creating build/bdist.linux-riscv64/wheel/distributed/http/static copying build/lib/distributed/http/static/individual-cluster-map.html -> build/bdist.linux-riscv64/wheel/./distributed/http/static creating build/bdist.linux-riscv64/wheel/distributed/http/static/js copying build/lib/distributed/http/static/js/individual-cluster-map.js -> build/bdist.linux-riscv64/wheel/./distributed/http/static/js copying build/lib/distributed/http/static/js/reconnecting-websocket.min.js -> build/bdist.linux-riscv64/wheel/./distributed/http/static/js copying build/lib/distributed/http/static/js/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/http/static/js copying build/lib/distributed/http/static/js/sortable.min.js -> build/bdist.linux-riscv64/wheel/./distributed/http/static/js copying build/lib/distributed/http/static/js/anime.min.js -> build/bdist.linux-riscv64/wheel/./distributed/http/static/js copying build/lib/distributed/http/static/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/http/static creating build/bdist.linux-riscv64/wheel/distributed/http/static/css copying build/lib/distributed/http/static/css/individual-cluster-map.css -> build/bdist.linux-riscv64/wheel/./distributed/http/static/css copying build/lib/distributed/http/static/css/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/http/static/css copying build/lib/distributed/http/static/css/status.css -> build/bdist.linux-riscv64/wheel/./distributed/http/static/css copying build/lib/distributed/http/static/css/gpu.css -> build/bdist.linux-riscv64/wheel/./distributed/http/static/css copying build/lib/distributed/http/static/css/sortable.min.css -> build/bdist.linux-riscv64/wheel/./distributed/http/static/css copying build/lib/distributed/http/static/css/base.css -> build/bdist.linux-riscv64/wheel/./distributed/http/static/css creating build/bdist.linux-riscv64/wheel/distributed/http/static/images copying build/lib/distributed/http/static/images/fa-bars.svg -> build/bdist.linux-riscv64/wheel/./distributed/http/static/images copying build/lib/distributed/http/static/images/pandas.png -> build/bdist.linux-riscv64/wheel/./distributed/http/static/images copying build/lib/distributed/http/static/images/numpy.png -> build/bdist.linux-riscv64/wheel/./distributed/http/static/images copying build/lib/distributed/http/static/images/python.png -> build/bdist.linux-riscv64/wheel/./distributed/http/static/images copying build/lib/distributed/http/static/images/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/http/static/images copying build/lib/distributed/http/static/images/jupyter.svg -> build/bdist.linux-riscv64/wheel/./distributed/http/static/images copying build/lib/distributed/http/static/images/favicon.ico -> build/bdist.linux-riscv64/wheel/./distributed/http/static/images copying build/lib/distributed/http/static/images/dask-logo.svg -> build/bdist.linux-riscv64/wheel/./distributed/http/static/images copying build/lib/distributed/http/proxy.py -> build/bdist.linux-riscv64/wheel/./distributed/http copying build/lib/distributed/nanny.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/threadpoolexecutor.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/diskutils.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/itertools.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/spans.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/py.typed -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/cluster_dump.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/spill.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/multi_lock.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/scheduler.py -> build/bdist.linux-riscv64/wheel/./distributed creating build/bdist.linux-riscv64/wheel/distributed/shuffle copying build/lib/distributed/shuffle/_buffer.py -> build/bdist.linux-riscv64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_limiter.py -> build/bdist.linux-riscv64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_rechunk.py -> build/bdist.linux-riscv64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_core.py -> build/bdist.linux-riscv64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_worker_plugin.py -> build/bdist.linux-riscv64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_shuffle.py -> build/bdist.linux-riscv64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_comms.py -> build/bdist.linux-riscv64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_memory.py -> build/bdist.linux-riscv64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_arrow.py -> build/bdist.linux-riscv64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_pickle.py -> build/bdist.linux-riscv64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_disk.py -> build/bdist.linux-riscv64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_scheduler_plugin.py -> build/bdist.linux-riscv64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_merge.py -> build/bdist.linux-riscv64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_exceptions.py -> build/bdist.linux-riscv64/wheel/./distributed/shuffle copying build/lib/distributed/pytest_resourceleaks.py -> build/bdist.linux-riscv64/wheel/./distributed creating build/bdist.linux-riscv64/wheel/distributed/dashboard copying build/lib/distributed/dashboard/utils.py -> build/bdist.linux-riscv64/wheel/./distributed/dashboard copying build/lib/distributed/dashboard/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/dashboard creating build/bdist.linux-riscv64/wheel/distributed/dashboard/templates copying build/lib/distributed/dashboard/templates/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/dashboard/templates copying build/lib/distributed/dashboard/templates/performance_report.html -> build/bdist.linux-riscv64/wheel/./distributed/dashboard/templates copying build/lib/distributed/dashboard/templates/file.html -> build/bdist.linux-riscv64/wheel/./distributed/dashboard/templates copying build/lib/distributed/dashboard/export_tool.js -> build/bdist.linux-riscv64/wheel/./distributed/dashboard copying build/lib/distributed/dashboard/export_tool.py -> build/bdist.linux-riscv64/wheel/./distributed/dashboard copying build/lib/distributed/dashboard/worker.py -> build/bdist.linux-riscv64/wheel/./distributed/dashboard copying build/lib/distributed/dashboard/core.py -> build/bdist.linux-riscv64/wheel/./distributed/dashboard copying build/lib/distributed/dashboard/theme.yaml -> build/bdist.linux-riscv64/wheel/./distributed/dashboard copying build/lib/distributed/dashboard/scheduler.py -> build/bdist.linux-riscv64/wheel/./distributed/dashboard creating build/bdist.linux-riscv64/wheel/distributed/dashboard/components copying build/lib/distributed/dashboard/components/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/dashboard/components copying build/lib/distributed/dashboard/components/nvml.py -> build/bdist.linux-riscv64/wheel/./distributed/dashboard/components copying build/lib/distributed/dashboard/components/shared.py -> build/bdist.linux-riscv64/wheel/./distributed/dashboard/components copying build/lib/distributed/dashboard/components/worker.py -> build/bdist.linux-riscv64/wheel/./distributed/dashboard/components copying build/lib/distributed/dashboard/components/rmm.py -> build/bdist.linux-riscv64/wheel/./distributed/dashboard/components copying build/lib/distributed/dashboard/components/scheduler.py -> build/bdist.linux-riscv64/wheel/./distributed/dashboard/components copying build/lib/distributed/config.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/stealing.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/publish.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/active_memory_manager.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/_asyncio.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/_async_taskgroup.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/distributed-schema.yaml -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/bokeh.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/_concurrent_futures_thread.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/worker_memory.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/system_monitor.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/compatibility.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/_version.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/counter.py -> build/bdist.linux-riscv64/wheel/./distributed creating build/bdist.linux-riscv64/wheel/distributed/protocol copying build/lib/distributed/protocol/cuda.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/protocol/numba.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/protocol/utils_test.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/protocol/utils.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/protocol/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/protocol/cupy.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/protocol/torch.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/protocol/pickle.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/protocol/numpy.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/protocol/keras.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/protocol/h5py.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/protocol/core.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/protocol/netcdf4.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/protocol/sparse.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/protocol/rmm.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/protocol/serialize.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/protocol/arrow.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/protocol/scipy.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/protocol/compression.py -> build/bdist.linux-riscv64/wheel/./distributed/protocol copying build/lib/distributed/objects.py -> build/bdist.linux-riscv64/wheel/./distributed creating build/bdist.linux-riscv64/wheel/distributed/cli copying build/lib/distributed/cli/dask_ssh.py -> build/bdist.linux-riscv64/wheel/./distributed/cli copying build/lib/distributed/cli/utils.py -> build/bdist.linux-riscv64/wheel/./distributed/cli copying build/lib/distributed/cli/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/cli copying build/lib/distributed/cli/dask_spec.py -> build/bdist.linux-riscv64/wheel/./distributed/cli copying build/lib/distributed/cli/dask_scheduler.py -> build/bdist.linux-riscv64/wheel/./distributed/cli copying build/lib/distributed/cli/dask_worker.py -> build/bdist.linux-riscv64/wheel/./distributed/cli creating build/bdist.linux-riscv64/wheel/distributed/comm copying build/lib/distributed/comm/utils.py -> build/bdist.linux-riscv64/wheel/./distributed/comm copying build/lib/distributed/comm/__init__.py -> build/bdist.linux-riscv64/wheel/./distributed/comm copying build/lib/distributed/comm/registry.py -> build/bdist.linux-riscv64/wheel/./distributed/comm copying build/lib/distributed/comm/tcp.py -> build/bdist.linux-riscv64/wheel/./distributed/comm copying build/lib/distributed/comm/inproc.py -> build/bdist.linux-riscv64/wheel/./distributed/comm copying build/lib/distributed/comm/core.py -> build/bdist.linux-riscv64/wheel/./distributed/comm copying build/lib/distributed/comm/ucx.py -> build/bdist.linux-riscv64/wheel/./distributed/comm copying build/lib/distributed/comm/addressing.py -> build/bdist.linux-riscv64/wheel/./distributed/comm copying build/lib/distributed/comm/ws.py -> build/bdist.linux-riscv64/wheel/./distributed/comm copying build/lib/distributed/security.py -> build/bdist.linux-riscv64/wheel/./distributed copying build/lib/distributed/proctitle.py -> build/bdist.linux-riscv64/wheel/./distributed running install_egg_info Copying distributed.egg-info to build/bdist.linux-riscv64/wheel/./distributed-2026.8.0-py3.14.egg-info running install_scripts creating build/bdist.linux-riscv64/wheel/distributed-2026.8.0.dist-info/WHEEL creating '/build/python-distributed/src/distributed-2026.8.0/dist/.tmp-6e2qwoqt/distributed-2026.8.0-py3-none-any.whl' and adding 'build/bdist.linux-riscv64/wheel' to it adding 'distributed/__init__.py' adding 'distributed/_async_taskgroup.py' adding 'distributed/_asyncio.py' adding 'distributed/_concurrent_futures_thread.py' adding 'distributed/_signals.py' adding 'distributed/_stories.py' adding 'distributed/_version.py' adding 'distributed/active_memory_manager.py' adding 'distributed/actor.py' adding 'distributed/batched.py' adding 'distributed/bokeh.py' adding 'distributed/broker.py' adding 'distributed/cfexecutor.py' adding 'distributed/chaos.py' adding 'distributed/client.py' adding 'distributed/cluster_dump.py' adding 'distributed/collections.py' adding 'distributed/compatibility.py' adding 'distributed/config.py' adding 'distributed/core.py' adding 'distributed/counter.py' adding 'distributed/diskutils.py' adding 'distributed/distributed-schema.yaml' adding 'distributed/distributed.yaml' adding 'distributed/event.py' adding 'distributed/exceptions.py' adding 'distributed/gc.py' adding 'distributed/itertools.py' adding 'distributed/lock.py' adding 'distributed/metrics.py' adding 'distributed/multi_lock.py' adding 'distributed/nanny.py' adding 'distributed/node.py' adding 'distributed/objects.py' adding 'distributed/preloading.py' adding 'distributed/process.py' adding 'distributed/proctitle.py' adding 'distributed/profile.py' adding 'distributed/publish.py' adding 'distributed/py.typed' adding 'distributed/pytest_resourceleaks.py' adding 'distributed/queues.py' adding 'distributed/recreate_tasks.py' adding 'distributed/scheduler.py' adding 'distributed/security.py' adding 'distributed/semaphore.py' adding 'distributed/sizeof.py' adding 'distributed/spans.py' adding 'distributed/spill.py' adding 'distributed/stealing.py' adding 'distributed/system.py' adding 'distributed/system_monitor.py' adding 'distributed/threadpoolexecutor.py' adding 'distributed/utils.py' adding 'distributed/utils_comm.py' adding 'distributed/utils_test.py' adding 'distributed/variable.py' adding 'distributed/versions.py' adding 'distributed/worker.py' adding 'distributed/worker_client.py' adding 'distributed/worker_memory.py' adding 'distributed/worker_state_machine.py' adding 'distributed/cli/__init__.py' adding 'distributed/cli/dask_scheduler.py' adding 'distributed/cli/dask_spec.py' adding 'distributed/cli/dask_ssh.py' adding 'distributed/cli/dask_worker.py' adding 'distributed/cli/utils.py' adding 'distributed/comm/__init__.py' adding 'distributed/comm/addressing.py' adding 'distributed/comm/core.py' adding 'distributed/comm/inproc.py' adding 'distributed/comm/registry.py' adding 'distributed/comm/tcp.py' adding 'distributed/comm/ucx.py' adding 'distributed/comm/utils.py' adding 'distributed/comm/ws.py' adding 'distributed/dashboard/__init__.py' adding 'distributed/dashboard/core.py' adding 'distributed/dashboard/export_tool.js' adding 'distributed/dashboard/export_tool.py' adding 'distributed/dashboard/scheduler.py' adding 'distributed/dashboard/theme.yaml' adding 'distributed/dashboard/utils.py' adding 'distributed/dashboard/worker.py' adding 'distributed/dashboard/components/__init__.py' adding 'distributed/dashboard/components/nvml.py' adding 'distributed/dashboard/components/rmm.py' adding 'distributed/dashboard/components/scheduler.py' adding 'distributed/dashboard/components/shared.py' adding 'distributed/dashboard/components/worker.py' adding 'distributed/dashboard/templates/__init__.py' adding 'distributed/dashboard/templates/file.html' adding 'distributed/dashboard/templates/performance_report.html' adding 'distributed/deploy/__init__.py' adding 'distributed/deploy/adaptive.py' adding 'distributed/deploy/adaptive_core.py' adding 'distributed/deploy/cluster.py' adding 'distributed/deploy/local.py' adding 'distributed/deploy/old_ssh.py' adding 'distributed/deploy/spec.py' adding 'distributed/deploy/ssh.py' adding 'distributed/deploy/subprocess.py' adding 'distributed/deploy/utils.py' adding 'distributed/diagnostics/__init__.py' adding 'distributed/diagnostics/cluster_dump.py' adding 'distributed/diagnostics/cudf.py' adding 'distributed/diagnostics/eventstream.py' adding 'distributed/diagnostics/graph_layout.py' adding 'distributed/diagnostics/memory_sampler.py' adding 'distributed/diagnostics/memray.py' adding 'distributed/diagnostics/nvml.py' adding 'distributed/diagnostics/plugin.py' adding 'distributed/diagnostics/progress.py' adding 'distributed/diagnostics/progress_stream.py' adding 'distributed/diagnostics/progressbar.py' adding 'distributed/diagnostics/rmm.py' adding 'distributed/diagnostics/task_stream.py' adding 'distributed/diagnostics/websocket.py' adding 'distributed/http/__init__.py' adding 'distributed/http/health.py' adding 'distributed/http/prometheus.py' adding 'distributed/http/proxy.py' adding 'distributed/http/routing.py' adding 'distributed/http/statics.py' adding 'distributed/http/utils.py' adding 'distributed/http/scheduler/__init__.py' adding 'distributed/http/scheduler/api.py' adding 'distributed/http/scheduler/info.py' adding 'distributed/http/scheduler/json.py' adding 'distributed/http/scheduler/missing_bokeh.py' adding 'distributed/http/scheduler/prometheus/__init__.py' adding 'distributed/http/scheduler/prometheus/core.py' adding 'distributed/http/scheduler/prometheus/semaphore.py' adding 'distributed/http/scheduler/prometheus/stealing.py' adding 'distributed/http/static/__init__.py' adding 'distributed/http/static/individual-cluster-map.html' adding 'distributed/http/static/css/__init__.py' adding 'distributed/http/static/css/base.css' adding 'distributed/http/static/css/gpu.css' adding 'distributed/http/static/css/individual-cluster-map.css' adding 'distributed/http/static/css/sortable.min.css' adding 'distributed/http/static/css/status.css' adding 'distributed/http/static/images/__init__.py' adding 'distributed/http/static/images/dask-logo.svg' adding 'distributed/http/static/images/fa-bars.svg' adding 'distributed/http/static/images/favicon.ico' adding 'distributed/http/static/images/jupyter.svg' adding 'distributed/http/static/images/numpy.png' adding 'distributed/http/static/images/pandas.png' adding 'distributed/http/static/images/python.png' adding 'distributed/http/static/js/__init__.py' adding 'distributed/http/static/js/anime.min.js' adding 'distributed/http/static/js/individual-cluster-map.js' adding 'distributed/http/static/js/reconnecting-websocket.min.js' adding 'distributed/http/static/js/sortable.min.js' adding 'distributed/http/templates/__init__.py' adding 'distributed/http/templates/base.html' adding 'distributed/http/templates/call-stack.html' adding 'distributed/http/templates/exceptions.html' adding 'distributed/http/templates/gpu.html' adding 'distributed/http/templates/json-index.html' adding 'distributed/http/templates/logs.html' adding 'distributed/http/templates/main.html' adding 'distributed/http/templates/simple.html' adding 'distributed/http/templates/status.html' adding 'distributed/http/templates/task.html' adding 'distributed/http/templates/worker-table.html' adding 'distributed/http/templates/worker.html' adding 'distributed/http/templates/workers.html' adding 'distributed/http/worker/__init__.py' adding 'distributed/http/worker/prometheus/__init__.py' adding 'distributed/http/worker/prometheus/core.py' adding 'distributed/protocol/__init__.py' adding 'distributed/protocol/arrow.py' adding 'distributed/protocol/compression.py' adding 'distributed/protocol/core.py' adding 'distributed/protocol/cuda.py' adding 'distributed/protocol/cupy.py' adding 'distributed/protocol/h5py.py' adding 'distributed/protocol/keras.py' adding 'distributed/protocol/netcdf4.py' adding 'distributed/protocol/numba.py' adding 'distributed/protocol/numpy.py' adding 'distributed/protocol/pickle.py' adding 'distributed/protocol/rmm.py' adding 'distributed/protocol/scipy.py' adding 'distributed/protocol/serialize.py' adding 'distributed/protocol/sparse.py' adding 'distributed/protocol/torch.py' adding 'distributed/protocol/utils.py' adding 'distributed/protocol/utils_test.py' adding 'distributed/shuffle/__init__.py' adding 'distributed/shuffle/_arrow.py' adding 'distributed/shuffle/_buffer.py' adding 'distributed/shuffle/_comms.py' adding 'distributed/shuffle/_core.py' adding 'distributed/shuffle/_disk.py' adding 'distributed/shuffle/_exceptions.py' adding 'distributed/shuffle/_limiter.py' adding 'distributed/shuffle/_memory.py' adding 'distributed/shuffle/_merge.py' adding 'distributed/shuffle/_pickle.py' adding 'distributed/shuffle/_rechunk.py' adding 'distributed/shuffle/_scheduler_plugin.py' adding 'distributed/shuffle/_shuffle.py' adding 'distributed/shuffle/_worker_plugin.py' adding 'distributed/widgets/__init__.py' adding 'distributed/widgets/templates/__init__.py' adding 'distributed/widgets/templates/client.html.j2' adding 'distributed/widgets/templates/cluster.html.j2' adding 'distributed/widgets/templates/computation.html.j2' adding 'distributed/widgets/templates/future.html.j2' adding 'distributed/widgets/templates/has_what.html.j2' adding 'distributed/widgets/templates/local_cluster.html.j2' adding 'distributed/widgets/templates/log.html.j2' adding 'distributed/widgets/templates/logs.html.j2' adding 'distributed/widgets/templates/process_interface.html.j2' adding 'distributed/widgets/templates/scheduler.html.j2' adding 'distributed/widgets/templates/scheduler_info.html.j2' adding 'distributed/widgets/templates/security.html.j2' adding 'distributed/widgets/templates/task_state.html.j2' adding 'distributed/widgets/templates/who_has.html.j2' adding 'distributed/widgets/templates/worker_state.html.j2' adding 'distributed-2026.8.0.dist-info/licenses/LICENSE.txt' adding 'distributed-2026.8.0.dist-info/METADATA' adding 'distributed-2026.8.0.dist-info/WHEEL' adding 'distributed-2026.8.0.dist-info/entry_points.txt' adding 'distributed-2026.8.0.dist-info/top_level.txt' adding 'distributed-2026.8.0.dist-info/RECORD' removing build/bdist.linux-riscv64/wheel Successfully built distributed-2026.8.0-py3-none-any.whl ==>(B Starting check()...(B ============================= test session starts ============================== platform linux -- Python 3.14.7, pytest-9.1.1, pluggy-1.6.0 -- /build/python-distributed/src/distributed-2026.8.0/test-env/bin/python cachedir: .pytest_cache rootdir: /build/python-distributed/src/distributed-2026.8.0 configfile: pyproject.toml plugins: timeout-2.4.0, flaky-3.8.1, repeat-0.9.4, xdist-3.8.0 timeout: 300.0s timeout method: signal timeout func_only: False collecting ... collected 3922 items / 516 deselected / 17 skipped / 3406 selected distributed/cli/tests/test_dask_scheduler.py::test_defaults FAILED [ 0%] distributed/cli/tests/test_dask_scheduler.py::test_no_dashboard PASSED [ 0%] distributed/cli/tests/test_dask_scheduler.py::test_dashboard SKIPPED [ 0%] distributed/cli/tests/test_dask_scheduler.py::test_dashboard_non_standard_ports SKIPPED [ 0%] distributed/cli/tests/test_dask_scheduler.py::test_multiple_protocols PASSED [ 0%] distributed/cli/tests/test_dask_scheduler.py::test_dashboard_allowlist SKIPPED [ 0%] distributed/cli/tests/test_dask_scheduler.py::test_interface PASSED [ 0%] distributed/cli/tests/test_dask_scheduler.py::test_scheduler_port_zero PASSED [ 0%] distributed/cli/tests/test_dask_scheduler.py::test_dashboard_port_zero SKIPPED [ 0%] distributed/cli/tests/test_dask_scheduler.py::test_preload_file PASSED [ 0%] distributed/cli/tests/test_dask_scheduler.py::test_preload_module PASSED [ 0%] distributed/cli/tests/test_dask_scheduler.py::test_preload_remote_module PASSED [ 0%] distributed/cli/tests/test_dask_scheduler.py::test_preload_config PASSED [ 0%] distributed/cli/tests/test_dask_scheduler.py::test_preload_command PASSED [ 0%] distributed/cli/tests/test_dask_scheduler.py::test_preload_command_default PASSED [ 0%] distributed/cli/tests/test_dask_scheduler.py::test_version_option PASSED [ 0%] distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers_2 PASSED [ 0%] distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers PASSED [ 0%] distributed/cli/tests/test_dask_scheduler.py::test_uvloop SKIPPED (could not import 'uvloop': No module named 'uvloop') [ 0%] distributed/cli/tests/test_dask_spec.py::test_file PASSED [ 0%] distributed/cli/tests/test_dask_spec.py::test_errors PASSED [ 0%] distributed/cli/tests/test_dask_spec.py::test_signal_handling_worker[2-Nanny] PASSED [ 0%] distributed/cli/tests/test_dask_spec.py::test_signal_handling_worker[2-Worker] PASSED [ 0%] distributed/cli/tests/test_dask_spec.py::test_signal_handling_worker[15-Nanny] PASSED [ 0%] distributed/cli/tests/test_dask_spec.py::test_signal_handling_worker[15-Worker] PASSED [ 0%] distributed/cli/tests/test_dask_spec.py::test_signal_handling_scheduler[2] PASSED [ 0%] distributed/cli/tests/test_dask_spec.py::test_signal_handling_scheduler[15] PASSED [ 0%] distributed/cli/tests/test_dask_spec.py::test_uvloop SKIPPED (could not import 'uvloop': No module named 'uvloop') [ 0%] distributed/cli/tests/test_dask_ssh.py::test_version_option PASSED [ 0%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports[args0-expect0] PASSED [ 0%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports[args1-expect1] PASSED [ 0%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports[args2-expect2] PASSED [ 0%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports[args3-expect3] PASSED [ 0%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports[args4-expect4] PASSED [ 0%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports[args5-expect5] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports[args6-expect6] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports[args7-expect7] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports[args8-expect8] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports[args9-expect9] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports[args10-expect10] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports[args11-expect11] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports[args12-expect12] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports[args13-expect13] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports[args14-expect14] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports[args15-expect15] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports[args16-expect16] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports[args17-expect17] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_apportion_ports_bad PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_nanny_worker_port_range_too_many_workers_raises PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_no_nanny PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_nworkers_requires_nanny PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_version_option PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_unknown_option_reports_no_such_option PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_stray_positional_hints_at_help_and_preload PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_signal_handling[2---nanny] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_signal_handling[2---no-nanny] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_signal_handling[15---nanny] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_signal_handling[15---no-nanny] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_error_during_startup[--nanny] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_error_during_startup[--no-nanny] PASSED [ 1%] distributed/cli/tests/test_dask_worker.py::test_uvloop[--nanny] SKIPPED [ 1%] distributed/cli/tests/test_dask_worker.py::test_uvloop[--no-nanny] SKIPPED [ 1%] distributed/cli/tests/test_tls_cli.py::test_basic PASSED [ 1%] distributed/cli/tests/test_tls_cli.py::test_sni PASSED [ 1%] distributed/cli/tests/test_tls_cli.py::test_use_config_file PASSED [ 1%] distributed/comm/tests/test_comms.py::test_parse_host_port[tornado] PASSED [ 1%] distributed/comm/tests/test_comms.py::test_unparse_host_port[tornado] PASSED [ 1%] distributed/comm/tests/test_comms.py::test_get_address_host[tornado] PASSED [ 1%] distributed/comm/tests/test_comms.py::test_resolve_address[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_get_local_address_for[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_tcp_listener_does_not_call_handler_on_handshake_error[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_tcp_specific[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_tls_specific[tornado-None] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_tls_specific[tornado-localhost] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_comm_failure_threading[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_inproc_specific_same_thread PASSED [ 2%] distributed/comm/tests/test_comms.py::test_inproc_specific_different_threads PASSED [ 2%] distributed/comm/tests/test_comms.py::test_inproc_continues_listening_after_handshake_error PASSED [ 2%] distributed/comm/tests/test_comms.py::test_inproc_handshakes_concurrently PASSED [ 2%] distributed/comm/tests/test_comms.py::test_default_client_server_ipv4[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_default_client_server_ipv6[tornado] FAILED [ 2%] distributed/comm/tests/test_comms.py::test_tcp_client_server_ipv4[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_tcp_client_server_ipv6[tornado] FAILED [ 2%] distributed/comm/tests/test_comms.py::test_tls_client_server_ipv4[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_tls_client_server_ipv6[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_inproc_client_server PASSED [ 2%] distributed/comm/tests/test_comms.py::test_tls_reject_certificate[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_tcp_comm_closed_implicit[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_tls_comm_closed_implicit[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_inproc_comm_closed_implicit PASSED [ 2%] distributed/comm/tests/test_comms.py::test_tcp_comm_closed_explicit[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_tls_comm_closed_explicit[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_inproc_comm_closed_explicit PASSED [ 2%] distributed/comm/tests/test_comms.py::test_inproc_comm_closed_explicit_2 PASSED [ 2%] distributed/comm/tests/test_comms.py::test_comm_closed_on_write_error[tornado-BufferError] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_comm_closed_on_write_error[tornado-CustomBase] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_comm_closed_on_read_error[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_retry_connect[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_stop_listener_during_handle_stream[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_tcp_connect_timeout[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_inproc_connect_timeout PASSED [ 2%] distributed/comm/tests/test_comms.py::test_tcp_many_listeners[tornado] PASSED [ 2%] distributed/comm/tests/test_comms.py::test_inproc_many_listeners PASSED [ 3%] distributed/comm/tests/test_comms.py::test_tcp_deserialize[tornado] PASSED [ 3%] distributed/comm/tests/test_comms.py::test_inproc_deserialize PASSED [ 3%] distributed/comm/tests/test_comms.py::test_inproc_deserialize_roundtrip PASSED [ 3%] distributed/comm/tests/test_comms.py::test_tcp_deserialize_roundtrip[tornado] PASSED [ 3%] distributed/comm/tests/test_comms.py::test_tcp_deserialize_eoferror[tornado] PASSED [ 3%] distributed/comm/tests/test_comms.py::test_tcp_repr[tornado] PASSED [ 3%] distributed/comm/tests/test_comms.py::test_tls_repr[tornado] PASSED [ 3%] distributed/comm/tests/test_comms.py::test_inproc_repr PASSED [ 3%] distributed/comm/tests/test_comms.py::test_tcp_addresses[tornado] PASSED [ 3%] distributed/comm/tests/test_comms.py::test_get_stream_address_raises_if_closed PASSED [ 3%] distributed/comm/tests/test_comms.py::test_tls_addresses[tornado] PASSED [ 3%] distributed/comm/tests/test_comms.py::test_inproc_addresses PASSED [ 3%] distributed/comm/tests/test_comms.py::test_register_backend_entrypoint PASSED [ 3%] distributed/comm/tests/test_comms.py::test_do_not_share_buffers[tornado-list] PASSED [ 3%] distributed/comm/tests/test_comms.py::test_do_not_share_buffers[tornado-OpaqueList] PASSED [ 3%] distributed/comm/tests/test_comms.py::test_share_buffer_with_header[tornado-1-0-False] PASSED [ 3%] distributed/comm/tests/test_comms.py::test_share_buffer_with_header[tornado-1-1800-False] PASSED [ 3%] distributed/comm/tests/test_comms.py::test_share_buffer_with_header[tornado-1-2100-True] PASSED [ 3%] distributed/comm/tests/test_comms.py::test_share_buffer_with_header[tornado-200000-9500-False] PASSED [ 3%] distributed/comm/tests/test_comms.py::test_share_buffer_with_header[tornado-200000-10500-True] PASSED [ 3%] distributed/comm/tests/test_comms.py::test_share_buffer_with_header[tornado-350000-0-False] PASSED [ 3%] distributed/comm/tests/test_tcp.py::test_getaddrinfo_invalid_af PASSED [ 3%] distributed/dashboard/tests/test_bokeh.py::test_old_import PASSED [ 3%] distributed/deploy/tests/test_adaptive.py::test_adaptive_local_cluster PASSED [ 3%] distributed/deploy/tests/test_adaptive.py::test_adaptive_local_cluster_multi_workers PASSED [ 3%] distributed/deploy/tests/test_adaptive.py::test_min_max PASSED [ 3%] distributed/deploy/tests/test_adaptive.py::test_avoid_churn PASSED [ 3%] distributed/deploy/tests/test_adaptive.py::test_adapt_quickly PASSED [ 3%] distributed/deploy/tests/test_adaptive.py::test_no_more_workers_than_tasks PASSED [ 3%] distributed/deploy/tests/test_adaptive.py::test_basic_no_loop PASSED [ 3%] distributed/deploy/tests/test_adaptive.py::test_target_duration[5] PASSED [ 3%] distributed/deploy/tests/test_adaptive.py::test_target_duration[1] PASSED [ 3%] distributed/deploy/tests/test_adaptive.py::test_worker_keys PASSED [ 3%] distributed/deploy/tests/test_adaptive.py::test_adapt_cores_memory PASSED [ 4%] distributed/deploy/tests/test_adaptive.py::test_adaptive_config PASSED [ 4%] distributed/deploy/tests/test_adaptive.py::test_update_adaptive PASSED [ 4%] distributed/deploy/tests/test_adaptive.py::test_adaptive_no_memory_limit PASSED [ 4%] distributed/deploy/tests/test_adaptive.py::test_adapt_gets_stopped_on_cluster_close PASSED [ 4%] distributed/deploy/tests/test_adaptive.py::test_scale_needs_to_be_awaited PASSED [ 4%] distributed/deploy/tests/test_adaptive.py::test_adaptive_stopped PASSED [ 4%] distributed/deploy/tests/test_adaptive.py::test_scale_up_large_tasks[1] PASSED [ 4%] distributed/deploy/tests/test_adaptive.py::test_scale_up_large_tasks[inf] PASSED [ 4%] distributed/deploy/tests/test_adaptive.py::test_respect_average_nthreads PASSED [ 4%] distributed/deploy/tests/test_adaptive.py::test_adaptive_stops_on_cluster_status_change PASSED [ 4%] distributed/deploy/tests/test_adaptive.py::test_interval PASSED [ 4%] distributed/deploy/tests/test_adaptive.py::test_adapt_logs_error_in_safe_target PASSED [ 4%] distributed/deploy/tests/test_adaptive.py::test_adapt_callback_logs_error_in_scale_down PASSED [ 4%] distributed/deploy/tests/test_adaptive.py::test_adaptive_logs_stopping_once[True] PASSED [ 4%] distributed/deploy/tests/test_adaptive.py::test_adaptive_logs_stopping_once[False] PASSED [ 4%] distributed/deploy/tests/test_adaptive.py::test_adapt_stop_del PASSED [ 4%] distributed/deploy/tests/test_adaptive_core.py::test_safe_target PASSED [ 4%] distributed/deploy/tests/test_adaptive_core.py::test_scale_up PASSED [ 4%] distributed/deploy/tests/test_adaptive_core.py::test_scale_down PASSED [ 4%] distributed/deploy/tests/test_cluster.py::test_eq PASSED [ 4%] distributed/deploy/tests/test_cluster.py::test_repr PASSED [ 4%] distributed/deploy/tests/test_cluster.py::test_cluster_wait_for_worker PASSED [ 4%] distributed/deploy/tests/test_cluster.py::test_exponential_backoff PASSED [ 4%] distributed/deploy/tests/test_cluster.py::test_sync_context_manager_used_with_async_cluster PASSED [ 4%] distributed/deploy/tests/test_deploy_utils.py::test_default_process_thread_breakdown PASSED [ 4%] distributed/deploy/tests/test_local.py::test_simple PASSED [ 4%] distributed/deploy/tests/test_local.py::test_local_cluster_supports_blocked_handlers PASSED [ 4%] distributed/deploy/tests/test_local.py::test_close_twice PASSED [ 4%] distributed/deploy/tests/test_local.py::test_procs PASSED [ 4%] distributed/deploy/tests/test_local.py::test_move_unserializable_data PASSED [ 4%] distributed/deploy/tests/test_local.py::test_transports_inproc PASSED [ 4%] distributed/deploy/tests/test_local.py::test_transports_tcp PASSED [ 4%] distributed/deploy/tests/test_local.py::test_transports_tcp_port PASSED [ 4%] distributed/deploy/tests/test_local.py::test_cores PASSED [ 5%] distributed/deploy/tests/test_local.py::test_submit PASSED [ 5%] distributed/deploy/tests/test_local.py::test_context_manager PASSED [ 5%] distributed/deploy/tests/test_local.py::test_no_workers_sync PASSED [ 5%] distributed/deploy/tests/test_local.py::test_Client_with_local PASSED [ 5%] distributed/deploy/tests/test_local.py::test_Client_solo PASSED [ 5%] distributed/deploy/tests/test_local.py::test_duplicate_clients SKIPPED [ 5%] distributed/deploy/tests/test_local.py::test_Client_kwargs PASSED [ 5%] distributed/deploy/tests/test_local.py::test_Client_unused_kwargs_with_cluster PASSED [ 5%] distributed/deploy/tests/test_local.py::test_Client_unused_kwargs_with_address PASSED [ 5%] distributed/deploy/tests/test_local.py::test_Client_twice PASSED [ 5%] distributed/deploy/tests/test_local.py::test_client_constructor_with_temporary_security PASSED [ 5%] distributed/deploy/tests/test_local.py::test_defaults PASSED [ 5%] distributed/deploy/tests/test_local.py::test_defaults_2 PASSED [ 5%] distributed/deploy/tests/test_local.py::test_defaults_3 PASSED [ 5%] distributed/deploy/tests/test_local.py::test_defaults_4 PASSED [ 5%] distributed/deploy/tests/test_local.py::test_defaults_6 PASSED [ 5%] distributed/deploy/tests/test_local.py::test_worker_params PASSED [ 5%] distributed/deploy/tests/test_local.py::test_memory_limit_none PASSED [ 5%] distributed/deploy/tests/test_local.py::test_cleanup PASSED [ 5%] distributed/deploy/tests/test_local.py::test_repeated PASSED [ 5%] distributed/deploy/tests/test_local.py::test_bokeh[True] SKIPPED (could not import 'bokeh': No module named 'bokeh') [ 5%] distributed/deploy/tests/test_local.py::test_bokeh[False] SKIPPED (could not import 'bokeh': No module named 'bokeh') [ 5%] distributed/deploy/tests/test_local.py::test_blocks_until_full PASSED [ 5%] distributed/deploy/tests/test_local.py::test_scale_up_and_down PASSED [ 5%] distributed/deploy/tests/test_local.py::test_silent_startup FAILED [ 5%] distributed/deploy/tests/test_local.py::test_only_local_access PASSED [ 5%] distributed/deploy/tests/test_local.py::test_remote_access FAILED [ 5%] distributed/deploy/tests/test_local.py::test_memory[None] PASSED [ 5%] distributed/deploy/tests/test_local.py::test_memory[3] PASSED [ 5%] distributed/deploy/tests/test_local.py::test_memory_nanny[None] PASSED [ 5%] distributed/deploy/tests/test_local.py::test_memory_nanny[3] PASSED [ 5%] distributed/deploy/tests/test_local.py::test_death_timeout_raises PASSED [ 5%] distributed/deploy/tests/test_local.py::test_bokeh_kwargs SKIPPED (could not import 'bokeh': No module named 'bokeh') [ 5%] distributed/deploy/tests/test_local.py::test_io_loop_periodic_callbacks PASSED [ 6%] distributed/deploy/tests/test_local.py::test_logging PASSED [ 6%] distributed/deploy/tests/test_local.py::test_ipywidgets PASSED [ 6%] distributed/deploy/tests/test_local.py::test_ipywidgets_loop PASSED [ 6%] distributed/deploy/tests/test_local.py::test_no_ipywidgets PASSED [ 6%] distributed/deploy/tests/test_local.py::test_scale PASSED [ 6%] distributed/deploy/tests/test_local.py::test_adapt PASSED [ 6%] distributed/deploy/tests/test_local.py::test_adapt_then_manual PASSED [ 6%] distributed/deploy/tests/test_local.py::test_scale_retires_workers PASSED [ 6%] distributed/deploy/tests/test_local.py::test_local_tls_restart PASSED [ 6%] distributed/deploy/tests/test_local.py::test_asynchronous_property PASSED [ 6%] distributed/deploy/tests/test_local.py::test_protocol_inproc PASSED [ 6%] distributed/deploy/tests/test_local.py::test_protocol_tcp PASSED [ 6%] distributed/deploy/tests/test_local.py::test_protocol_ip PASSED [ 6%] distributed/deploy/tests/test_local.py::test_worker_class_worker PASSED [ 6%] distributed/deploy/tests/test_local.py::test_worker_class_nanny PASSED [ 6%] distributed/deploy/tests/test_local.py::test_worker_class_nanny_async PASSED [ 6%] distributed/deploy/tests/test_local.py::test_starts_up_sync PASSED [ 6%] distributed/deploy/tests/test_local.py::test_dont_select_closed_worker PASSED [ 6%] distributed/deploy/tests/test_local.py::test_client_cluster_synchronous PASSED [ 6%] distributed/deploy/tests/test_local.py::test_scale_memory_cores PASSED [ 6%] distributed/deploy/tests/test_local.py::test_repr[2 GiB] PASSED [ 6%] distributed/deploy/tests/test_local.py::test_repr[None] PASSED [ 6%] distributed/deploy/tests/test_local.py::test_capture_security[True] PASSED [ 6%] distributed/deploy/tests/test_local.py::test_capture_security[False] PASSED [ 6%] distributed/deploy/tests/test_local.py::test_no_dangling_asyncio_tasks PASSED [ 6%] distributed/deploy/tests/test_local.py::test_async_with PASSED [ 6%] distributed/deploy/tests/test_local.py::test_no_workers PASSED [ 6%] distributed/deploy/tests/test_local.py::test_cluster_names PASSED [ 6%] distributed/deploy/tests/test_local.py::test_local_cluster_redundant_kwarg[True] PASSED [ 6%] distributed/deploy/tests/test_local.py::test_local_cluster_redundant_kwarg[False] PASSED [ 6%] distributed/deploy/tests/test_local.py::test_cluster_info_sync PASSED [ 6%] distributed/deploy/tests/test_local.py::test_cluster_info_sync_is_robust_to_network_blips PASSED [ 6%] distributed/deploy/tests/test_local.py::test_cluster_host_used_throughout_cluster[True-None] PASSED [ 6%] distributed/deploy/tests/test_local.py::test_cluster_host_used_throughout_cluster[True-127.0.0.1] PASSED [ 7%] distributed/deploy/tests/test_local.py::test_cluster_host_used_throughout_cluster[False-None] PASSED [ 7%] distributed/deploy/tests/test_local.py::test_cluster_host_used_throughout_cluster[False-127.0.0.1] PASSED [ 7%] distributed/deploy/tests/test_local.py::test_connect_to_closed_cluster PASSED [ 7%] distributed/deploy/tests/test_local.py::test_localcluster_start_exception PASSED [ 7%] distributed/deploy/tests/test_local.py::test_localcluster_get_client PASSED [ 7%] distributed/deploy/tests/test_slow_adaptive.py::test_startup PASSED [ 7%] distributed/deploy/tests/test_slow_adaptive.py::test_adaptive PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_specification PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_spec_sync PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_loop_started_in_constructor PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_repr PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_scale PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_unexpected_closed_worker PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_restart PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_broken_worker PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_spec_close_clusters PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_new_worker_spec PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_nanny_port PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_spec_process PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_get_logs PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_scheduler_info PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_dashboard_link PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_widget PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_scale_cores_memory PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_ProcessInterfaceValid PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_MultiWorker PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_run_spec PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_run_spec_cluster_worker_names PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_bad_close PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_shutdown_scheduler_disabled PASSED [ 7%] distributed/deploy/tests/test_spec_cluster.py::test_shutdown_scheduler PASSED [ 7%] distributed/deploy/tests/test_subprocess.py::test_basic PASSED [ 7%] distributed/deploy/tests/test_subprocess.py::test_n_workers PASSED [ 7%] distributed/deploy/tests/test_subprocess.py::test_scale_up_and_down PASSED [ 8%] distributed/deploy/tests/test_subprocess.py::test_raise_if_scheduler_fails_to_start PASSED [ 8%] distributed/deploy/tests/test_subprocess.py::test_raise_on_windows SKIPPED [ 8%] distributed/diagnostics/tests/test_cluster_dump_plugin.py::test_cluster_dump_plugin PASSED [ 8%] distributed/diagnostics/tests/test_eventstream.py::test_eventstream SKIPPED [ 8%] distributed/diagnostics/tests/test_eventstream.py::test_eventstream_remote PASSED [ 8%] distributed/diagnostics/tests/test_graph_layout.py::test_basic PASSED [ 8%] distributed/diagnostics/tests/test_graph_layout.py::test_construct_after_call PASSED [ 8%] distributed/diagnostics/tests/test_graph_layout.py::test_states PASSED [ 8%] distributed/diagnostics/tests/test_graph_layout.py::test_release_tasks PASSED [ 8%] distributed/diagnostics/tests/test_graph_layout.py::test_forget PASSED [ 8%] distributed/diagnostics/tests/test_graph_layout.py::test_unique_positions PASSED [ 8%] distributed/diagnostics/tests/test_graph_layout.py::test_layout_scatter PASSED [ 8%] distributed/diagnostics/tests/test_install_plugin.py::test_pip_install PASSED [ 8%] distributed/diagnostics/tests/test_install_plugin.py::test_conda_install PASSED [ 8%] distributed/diagnostics/tests/test_install_plugin.py::test_pip_install_fails PASSED [ 8%] distributed/diagnostics/tests/test_install_plugin.py::test_conda_install_fails_when_conda_not_found PASSED [ 8%] distributed/diagnostics/tests/test_install_plugin.py::test_conda_install_fails_when_conda_raises PASSED [ 8%] distributed/diagnostics/tests/test_install_plugin.py::test_conda_install_fails_on_returncode PASSED [ 8%] distributed/diagnostics/tests/test_install_plugin.py::test_package_install_failing_does_not_restart_on_nanny PASSED [ 8%] distributed/diagnostics/tests/test_memory_sampler.py::test_sync PASSED [ 8%] distributed/diagnostics/tests/test_memory_sampler.py::test_at_least_one_sample PASSED [ 8%] distributed/diagnostics/tests/test_memory_sampler.py::test_multi_sample PASSED [ 8%] distributed/diagnostics/tests/test_memory_sampler.py::test_pandas[False] SKIPPED [ 8%] distributed/diagnostics/tests/test_memory_sampler.py::test_pandas[True] SKIPPED [ 8%] distributed/diagnostics/tests/test_memory_sampler.py::test_pandas_multiseries[False] PASSED [ 8%] distributed/diagnostics/tests/test_memory_sampler.py::test_pandas_multiseries[True] PASSED [ 8%] distributed/diagnostics/tests/test_nanny_plugin.py::test_register_idempotent_plugin PASSED [ 8%] distributed/diagnostics/tests/test_nanny_plugin.py::test_register_non_idempotent_plugin PASSED [ 8%] distributed/diagnostics/tests/test_nanny_plugin.py::test_register_plugin_with_broken_setup_to_existing_nannies_raises PASSED [ 8%] distributed/diagnostics/tests/test_nanny_plugin.py::test_plugin_with_broken_setup_on_new_nanny_logs PASSED [ 8%] distributed/diagnostics/tests/test_nanny_plugin.py::test_unregister_nanny_plugin_with_broken_teardown_raises PASSED [ 8%] distributed/diagnostics/tests/test_nanny_plugin.py::test_nanny_plugin_with_broken_teardown_logs_on_close PASSED [ 8%] distributed/diagnostics/tests/test_progress.py::test_multiprogress PASSED [ 8%] distributed/diagnostics/tests/test_progress.py::test_multiprogress_cancel PASSED [ 9%] distributed/diagnostics/tests/test_progress.py::test_multiprogress_with_spans PASSED [ 9%] distributed/diagnostics/tests/test_progress.py::test_multiprogress_with_prefix PASSED [ 9%] distributed/diagnostics/tests/test_progress.py::test_robust_to_bad_plugin PASSED [ 9%] distributed/diagnostics/tests/test_progress.py::test_group_timing PASSED [ 9%] distributed/diagnostics/tests/test_progress_widgets.py::test_progressbar_widget PASSED [ 9%] distributed/diagnostics/tests/test_progress_widgets.py::test_multi_progressbar_widget PASSED [ 9%] distributed/diagnostics/tests/test_progress_widgets.py::test_values PASSED [ 9%] distributed/diagnostics/tests/test_progress_widgets.py::test_progressbar_done PASSED [ 9%] distributed/diagnostics/tests/test_progress_widgets.py::test_progressbar_cancel PASSED [ 9%] distributed/diagnostics/tests/test_progress_widgets.py::test_multibar_complete PASSED [ 9%] distributed/diagnostics/tests/test_progress_widgets.py::test_fast PASSED [ 9%] distributed/diagnostics/tests/test_progress_widgets.py::test_multibar_with_spans PASSED [ 9%] distributed/diagnostics/tests/test_progress_widgets.py::test_serializers PASSED [ 9%] distributed/diagnostics/tests/test_progress_widgets.py::test_tls PASSED [ 9%] distributed/diagnostics/tests/test_progressbar.py::test_text_progressbar PASSED [ 9%] distributed/diagnostics/tests/test_progressbar.py::test_TextProgressBar_error PASSED [ 9%] distributed/diagnostics/tests/test_progressbar.py::test_TextProgressBar_empty PASSED [ 9%] distributed/diagnostics/tests/test_progressbar.py::test_progress_function PASSED [ 9%] distributed/diagnostics/tests/test_progressbar.py::test_progress_function_w_kwargs PASSED [ 9%] distributed/diagnostics/tests/test_progressbar.py::test_progress_function_raises PASSED [ 9%] distributed/diagnostics/tests/test_scheduler_plugin.py::test_simple PASSED [ 9%] distributed/diagnostics/tests/test_scheduler_plugin.py::test_add_remove_worker PASSED [ 9%] distributed/diagnostics/tests/test_scheduler_plugin.py::test_async_add_remove_worker PASSED [ 9%] distributed/diagnostics/tests/test_scheduler_plugin.py::test_async_and_sync_add_remove_worker PASSED [ 9%] distributed/diagnostics/tests/test_scheduler_plugin.py::test_failing_async_add_remove_worker PASSED [ 9%] distributed/diagnostics/tests/test_scheduler_plugin.py::test_failing_sync_add_remove_worker PASSED [ 9%] distributed/diagnostics/tests/test_scheduler_plugin.py::test_lifecycle PASSED [ 9%] distributed/diagnostics/tests/test_scheduler_plugin.py::test_register_plugin PASSED [ 9%] distributed/diagnostics/tests/test_scheduler_plugin.py::test_unregister_scheduler_plugin PASSED [ 9%] distributed/diagnostics/tests/test_scheduler_plugin.py::test_unregister_scheduler_plugin_from_client PASSED [ 9%] distributed/diagnostics/tests/test_scheduler_plugin.py::test_register_plugin_on_scheduler PASSED [ 9%] distributed/diagnostics/tests/test_scheduler_plugin.py::test_closing_errors_ok PASSED [ 9%] distributed/diagnostics/tests/test_scheduler_plugin.py::test_update_graph_hook_simple PASSED [ 9%] distributed/diagnostics/tests/test_scheduler_plugin.py::test_update_graph_hook_complex PASSED [ 10%] distributed/diagnostics/tests/test_scheduler_plugin.py::test_register_idempotent_plugin PASSED [ 10%] distributed/diagnostics/tests/test_scheduler_plugin.py::test_register_non_idempotent_plugin PASSED [ 10%] distributed/diagnostics/tests/test_scheduler_plugin.py::test_register_idempotent_plugins_directly PASSED [ 10%] distributed/diagnostics/tests/test_scheduler_plugin.py::test_register_non_idempotent_plugins_directly PASSED [ 10%] distributed/diagnostics/tests/test_task_stream.py::test_TaskStreamPlugin PASSED [ 10%] distributed/diagnostics/tests/test_task_stream.py::test_maxlen PASSED [ 10%] distributed/diagnostics/tests/test_task_stream.py::test_collect PASSED [ 10%] distributed/diagnostics/tests/test_task_stream.py::test_collect_start_index PASSED [ 10%] distributed/diagnostics/tests/test_task_stream.py::test_client PASSED [ 10%] distributed/diagnostics/tests/test_task_stream.py::test_client_sync PASSED [ 10%] distributed/diagnostics/tests/test_task_stream.py::test_client_ctx PASSED [ 10%] distributed/diagnostics/tests/test_task_stream.py::test_client_ctx_sync PASSED [ 10%] distributed/diagnostics/tests/test_task_stream.py::test_get_task_stream_plot SKIPPED [ 10%] distributed/diagnostics/tests/test_task_stream.py::test_get_task_stream_save SKIPPED [ 10%] distributed/diagnostics/tests/test_worker_plugin.py::test_create_with_client PASSED [ 10%] distributed/diagnostics/tests/test_worker_plugin.py::test_remove_with_client PASSED [ 10%] distributed/diagnostics/tests/test_worker_plugin.py::test_remove_with_client_raises PASSED [ 10%] distributed/diagnostics/tests/test_worker_plugin.py::test_create_on_construction PASSED [ 10%] distributed/diagnostics/tests/test_worker_plugin.py::test_normal_task_transitions_called PASSED [ 10%] distributed/diagnostics/tests/test_worker_plugin.py::test_failing_task_transitions_called PASSED [ 10%] distributed/diagnostics/tests/test_worker_plugin.py::test_superseding_task_transitions_called PASSED [ 10%] distributed/diagnostics/tests/test_worker_plugin.py::test_dependent_tasks PASSED [ 10%] distributed/diagnostics/tests/test_worker_plugin.py::test_empty_plugin PASSED [ 10%] distributed/diagnostics/tests/test_worker_plugin.py::test_default_name PASSED [ 10%] distributed/diagnostics/tests/test_worker_plugin.py::test_assert_no_warning_no_overload PASSED [ 10%] distributed/diagnostics/tests/test_worker_plugin.py::test_WorkerPlugin_overwrite PASSED [ 10%] distributed/diagnostics/tests/test_worker_plugin.py::test_register_idempotent_plugin PASSED [ 10%] distributed/diagnostics/tests/test_worker_plugin.py::test_register_non_idempotent_plugin PASSED [ 10%] distributed/diagnostics/tests/test_worker_plugin.py::test_register_plugin_with_broken_setup_to_existing_workers_raises PASSED [ 10%] distributed/diagnostics/tests/test_worker_plugin.py::test_plugin_with_broken_setup_on_new_worker_logs PASSED [ 10%] distributed/diagnostics/tests/test_worker_plugin.py::test_unregister_worker_plugin_with_broken_teardown_raises PASSED [ 10%] distributed/diagnostics/tests/test_worker_plugin.py::test_plugin_with_broken_teardown_logs_on_close PASSED [ 10%] distributed/http/scheduler/tests/test_missing_bokeh.py::test_missing_bokeh PASSED [ 10%] distributed/http/scheduler/tests/test_missing_bokeh.py::test_bokeh_version_too_low SKIPPED [ 11%] distributed/http/scheduler/tests/test_missing_bokeh.py::test_bokeh_version_too_high SKIPPED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_connect PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_worker_404 PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_prefix SKIPPED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_prometheus PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_metrics_when_prometheus_client_not_installed PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_prometheus_collect_client_connections_totals PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_prometheus_collect_task_states PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_prometheus_collect_task_groups PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_prometheus_collect_task_prefix_counts PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_prometheus_collect_worker_totals PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_prometheus_collect_worker_states PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_health SKIPPED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_sitemap PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_task_page[1_0] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_task_page[1_1] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_task_page[x] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_task_page[a+b] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_task_page[a b] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_task_page[key5] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_task_page[key6] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_task_page[key7] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_task_page[key8] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_task_page[(] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_task_page[[] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_task_page['] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_task_page["] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_task_page[key13] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_task_page[key14] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_task_page[key15] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_task_page[key16] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_call_stack_page[1_0] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_call_stack_page[1_1] PASSED [ 11%] distributed/http/scheduler/tests/test_scheduler_http.py::test_call_stack_page[x] PASSED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_call_stack_page[a+b] PASSED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_call_stack_page[a b] PASSED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_call_stack_page[key5] PASSED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_call_stack_page[key6] PASSED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_call_stack_page[key7] PASSED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_call_stack_page[key8] PASSED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_call_stack_page[(] PASSED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_call_stack_page[[] PASSED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_call_stack_page['] PASSED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_call_stack_page["] PASSED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_call_stack_page[key13] PASSED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_call_stack_page[key14] PASSED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_call_stack_page[key15] PASSED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_call_stack_page[key16] PASSED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_allow_websocket_origin SKIPPED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_eventstream PASSED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_api_disabled_by_default PASSED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_api SKIPPED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_retire_workers SKIPPED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_retire_workers_with_tuple_keys SKIPPED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_get_workers SKIPPED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_adaptive_target SKIPPED [ 12%] distributed/http/scheduler/tests/test_scheduler_http.py::test_check_idle SKIPPED [ 12%] distributed/http/scheduler/tests/test_semaphore_http.py::test_prometheus PASSED [ 12%] distributed/http/scheduler/tests/test_semaphore_http.py::test_prometheus_without_semaphore_extension PASSED [ 12%] distributed/http/scheduler/tests/test_stealing_http.py::test_prometheus PASSED [ 12%] distributed/http/scheduler/tests/test_stealing_http.py::test_prometheus_collect_count_total_by_cost_multipliers PASSED [ 12%] distributed/http/scheduler/tests/test_stealing_http.py::test_prometheus_collect_cost_total_by_cost_multipliers PASSED [ 12%] distributed/http/scheduler/tests/test_stealing_http.py::test_prometheus_without_stealing_extension PASSED [ 12%] distributed/http/tests/test_core.py::test_scheduler PASSED [ 12%] distributed/http/tests/test_core.py::test_prometheus_api_doc PASSED [ 12%] distributed/http/tests/test_routing.py::test_basic PASSED [ 12%] distributed/http/worker/tests/test_worker_http.py::test_prometheus PASSED [ 12%] distributed/http/worker/tests/test_worker_http.py::test_metrics_when_prometheus_client_not_installed PASSED [ 13%] distributed/http/worker/tests/test_worker_http.py::test_prometheus_collect_task_states PASSED [ 13%] distributed/http/worker/tests/test_worker_http.py::test_health SKIPPED [ 13%] distributed/http/worker/tests/test_worker_http.py::test_sitemap SKIPPED [ 13%] distributed/http/worker/tests/test_worker_http.py::test_prometheus_collect_memory_metrics PASSED [ 13%] distributed/http/worker/tests/test_worker_http.py::test_prometheus_collect_memory_metrics_spill_disabled PASSED [ 13%] distributed/http/worker/tests/test_worker_http.py::test_prometheus_collect_memory_metrics_bogus_sizeof PASSED [ 13%] distributed/protocol/tests/test_arrow.py::test_roundtrip[RecordBatch] PASSED [ 13%] distributed/protocol/tests/test_arrow.py::test_roundtrip[Table] PASSED [ 13%] distributed/protocol/tests/test_arrow.py::test_scatter[RecordBatch] PASSED [ 13%] distributed/protocol/tests/test_arrow.py::test_scatter[Table] PASSED [ 13%] distributed/protocol/tests/test_arrow.py::test_dumps_compression PASSED [ 13%] distributed/protocol/tests/test_arrow.py::test_dump_fileinfo PASSED [ 13%] distributed/protocol/tests/test_collection.py::test_serialize_collection[y0-dask-tuple] PASSED [ 13%] distributed/protocol/tests/test_collection.py::test_serialize_collection[y0-dask-dict] PASSED [ 13%] distributed/protocol/tests/test_collection.py::test_serialize_collection[y0-dask-list] PASSED [ 13%] distributed/protocol/tests/test_collection.py::test_serialize_collection[y1-pickle-tuple] PASSED [ 13%] distributed/protocol/tests/test_collection.py::test_serialize_collection[y1-pickle-dict] PASSED [ 13%] distributed/protocol/tests/test_collection.py::test_serialize_collection[y1-pickle-list] PASSED [ 13%] distributed/protocol/tests/test_collection.py::test_serialize_collection[None-pickle-tuple] PASSED [ 13%] distributed/protocol/tests/test_collection.py::test_serialize_collection[None-pickle-dict] PASSED [ 13%] distributed/protocol/tests/test_collection.py::test_serialize_collection[None-pickle-list] PASSED [ 13%] distributed/protocol/tests/test_collection.py::test_large_collections_serialize_simply PASSED [ 13%] distributed/protocol/tests/test_collection.py::test_nested_types PASSED [ 13%] distributed/protocol/tests/test_compression.py::test_get_compression_settings PASSED [ 13%] distributed/protocol/tests/test_compression.py::test_auto_compression PASSED [ 13%] distributed/protocol/tests/test_compression.py::test_compression_1 PASSED [ 13%] distributed/protocol/tests/test_compression.py::test_compression_2 PASSED [ 13%] distributed/protocol/tests/test_compression.py::test_compression_3 PASSED [ 13%] distributed/protocol/tests/test_compression.py::test_compression_without_deserialization PASSED [ 13%] distributed/protocol/tests/test_compression.py::test_lz4_decompression_avoids_deep_copy PASSED [ 13%] distributed/protocol/tests/test_compression.py::test_maybe_compress[None-bytes] PASSED [ 13%] distributed/protocol/tests/test_compression.py::test_maybe_compress[None-memoryview] PASSED [ 13%] distributed/protocol/tests/test_compression.py::test_maybe_compress[zlib-bytes] PASSED [ 13%] distributed/protocol/tests/test_compression.py::test_maybe_compress[zlib-memoryview] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_maybe_compress[lz4-bytes] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_maybe_compress[lz4-memoryview] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_maybe_compress[snappy-bytes] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_maybe_compress[snappy-memoryview] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_maybe_compress[zstd-bytes] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_maybe_compress[zstd-memoryview] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_maybe_compress_sample[None] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_maybe_compress_sample[zlib] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_maybe_compress_sample[lz4] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_maybe_compress_sample[snappy] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_maybe_compress_sample[zstd] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_maybe_compress_memoryviews[None] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_maybe_compress_memoryviews[zlib] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_maybe_compress_memoryviews[lz4] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_maybe_compress_memoryviews[snappy] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_maybe_compress_memoryviews[zstd] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_maybe_compress_auto PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_loads_without_deserialization_avoids_compression PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_compress_remote_comms PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_disable_compression_on_localhost[None] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_disable_compression_on_localhost[127.0.0.1] PASSED [ 14%] distributed/protocol/tests/test_compression.py::test_disable_compression_on_localhost[localhost] PASSED [ 14%] distributed/protocol/tests/test_h5py.py::test_serialize_deserialize_file PASSED [ 14%] distributed/protocol/tests/test_h5py.py::test_serialize_deserialize_group PASSED [ 14%] distributed/protocol/tests/test_h5py.py::test_serialize_deserialize_dataset PASSED [ 14%] distributed/protocol/tests/test_h5py.py::test_raise_error_on_serialize_write_permissions PASSED [ 14%] distributed/protocol/tests/test_h5py.py::test_h5py_serialize PASSED [ 14%] distributed/protocol/tests/test_h5py.py::test_h5py_serialize_2 PASSED [ 14%] distributed/protocol/tests/test_highlevelgraph.py::test_combo_of_layer_types PASSED [ 14%] distributed/protocol/tests/test_highlevelgraph.py::test_blockwise PASSED [ 14%] distributed/protocol/tests/test_highlevelgraph.py::test_shuffle PASSED [ 14%] distributed/protocol/tests/test_highlevelgraph.py::test_array_annotations PASSED [ 14%] distributed/protocol/tests/test_highlevelgraph.py::test_dataframe_annotations PASSED [ 14%] distributed/protocol/tests/test_netcdf4.py::test_serialize_deserialize_dataset PASSED [ 15%] distributed/protocol/tests/test_netcdf4.py::test_serialize_deserialize_variable PASSED [ 15%] distributed/protocol/tests/test_netcdf4.py::test_serialize_deserialize_group PASSED [ 15%] distributed/protocol/tests/test_netcdf4.py::test_netcdf4_serialize PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_serialize PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x0] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x1] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x2] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x3] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x4] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x5] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x6] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x7] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x8] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x9] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x10] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x11] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x12] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x13] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x14] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x15] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x16] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x17] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x18] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x19] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x20] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x21] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x22] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x23] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x24] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x25] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x26] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x27] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x28] PASSED [ 15%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x29] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x30] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x31] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x32] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x33] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x34] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x35] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy[x36] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_dumps_numpy_writable[True] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_dumps_numpy_writable[False] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_serialize_numpy_ma_masked_array[x0] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_serialize_numpy_ma_masked_array[x1] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_serialize_numpy_ma_masked_array[x2] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_serialize_numpy_ma_masked_array[x3] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_serialize_numpy_ma_masked_array[x4] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_serialize_numpy_ma_masked_array[x5] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_serialize_numpy_ma_masked PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_dumps_serialize_numpy_custom_dtype SKIPPEDre.test_rational') [ 16%] distributed/protocol/tests/test_numpy.py::test_memmap PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_itemsize[f8-8] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_itemsize[i4-4] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_itemsize[c16-16] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_itemsize[b-1] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_itemsize[S3-3] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_itemsize[M8[us]-8] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_itemsize[M8[s]-8] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_itemsize[U3-12] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_itemsize[dt8-12] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_itemsize[dt9-4] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_itemsize[dt10-8] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_itemsize[dt11-88] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_itemsize[dt12-8] PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_compress_numpy PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_compress_memoryview PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_dumps_large PASSED [ 16%] distributed/protocol/tests/test_numpy.py::test_zero_strided_numpy_array[True-x0] PASSED [ 17%] distributed/protocol/tests/test_numpy.py::test_zero_strided_numpy_array[True-x1] PASSED [ 17%] distributed/protocol/tests/test_numpy.py::test_zero_strided_numpy_array[True-x2] PASSED [ 17%] distributed/protocol/tests/test_numpy.py::test_zero_strided_numpy_array[True-x3] PASSED [ 17%] distributed/protocol/tests/test_numpy.py::test_zero_strided_numpy_array[False-x0] PASSED [ 17%] distributed/protocol/tests/test_numpy.py::test_zero_strided_numpy_array[False-x1] PASSED [ 17%] distributed/protocol/tests/test_numpy.py::test_zero_strided_numpy_array[False-x2] PASSED [ 17%] distributed/protocol/tests/test_numpy.py::test_zero_strided_numpy_array[False-x3] PASSED [ 17%] distributed/protocol/tests/test_numpy.py::test_non_zero_strided_array PASSED [ 17%] distributed/protocol/tests/test_numpy.py::test_serialize_writeable_array_readonly_base_object PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df0] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df1] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df2] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df3] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df4] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df5] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df6] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df7] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df8] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df9] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df10] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df11] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df15] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df16] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df17] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df18] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df19] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df20] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df21] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df23] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df24] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_serialize_pandas[df25] PASSED [ 17%] distributed/protocol/tests/test_pandas.py::test_dumps_pandas_writable PASSED [ 17%] distributed/protocol/tests/test_pickle.py::test_pickle_data[4] PASSED [ 17%] distributed/protocol/tests/test_pickle.py::test_pickle_data[5] PASSED [ 18%] distributed/protocol/tests/test_pickle.py::test_pickle_out_of_band[4] PASSED [ 18%] distributed/protocol/tests/test_pickle.py::test_pickle_out_of_band[5] PASSED [ 18%] distributed/protocol/tests/test_pickle.py::test_pickle_empty[4] PASSED [ 18%] distributed/protocol/tests/test_pickle.py::test_pickle_empty[5] PASSED [ 18%] distributed/protocol/tests/test_pickle.py::test_pickle_numpy[4] PASSED [ 18%] distributed/protocol/tests/test_pickle.py::test_pickle_numpy[5] PASSED [ 18%] distributed/protocol/tests/test_pickle.py::test_pickle_functions[4] PASSED [ 18%] distributed/protocol/tests/test_pickle.py::test_pickle_functions[5] PASSED [ 18%] distributed/protocol/tests/test_pickle.py::test_pickle_by_value_when_registered PASSED [ 18%] distributed/protocol/tests/test_pickle.py::test_allow_pickle_if_registered_in_dask_serialize PASSED [ 18%] distributed/protocol/tests/test_pickle.py::test_nopickle_nested PASSED [ 18%] distributed/protocol/tests/test_pickle.py::test_pickle_zero_copy_read_only_flag[dask] PASSED [ 18%] distributed/protocol/tests/test_pickle.py::test_pickle_zero_copy_read_only_flag[pickle] PASSED [ 18%] distributed/protocol/tests/test_protocol.py::test_protocol PASSED [ 18%] distributed/protocol/tests/test_protocol.py::test_small PASSED [ 18%] distributed/protocol/tests/test_protocol.py::test_small_and_big PASSED [ 18%] distributed/protocol/tests/test_protocol.py::test_large_bytes[bytes] PASSED [ 18%] distributed/protocol/tests/test_protocol.py::test_large_bytes[memoryview] PASSED [ 18%] distributed/protocol/tests/test_protocol.py::test_large_messages_map PASSED [ 18%] distributed/protocol/tests/test_protocol.py::test_loads_deserialize_False PASSED [ 18%] distributed/protocol/tests/test_protocol.py::test_dumps_loads_Serialize PASSED [ 18%] distributed/protocol/tests/test_protocol.py::test_dumps_loads_Serialized PASSED [ 18%] distributed/protocol/tests/test_protocol.py::test_preserve_header[serializers0] PASSED [ 18%] distributed/protocol/tests/test_protocol.py::test_preserve_header[serializers1] PASSED [ 18%] distributed/protocol/tests/test_protocol.py::test_sizeof_serialize[Serialize-Serialized0] PASSED [ 18%] distributed/protocol/tests/test_protocol.py::test_sizeof_serialize[Serialize-Serialized1] PASSED [ 18%] distributed/protocol/tests/test_protocol.py::test_sizeof_serialize[ToPickle-Pickled] PASSED [ 18%] distributed/protocol/tests/test_protocol.py::test_deeply_nested_structures PASSED [ 18%] distributed/protocol/tests/test_protocol.py::test_fallback_to_pickle PASSED [ 18%] distributed/protocol/tests/test_protocol_utils.py::test_pack_frames PASSED [ 18%] distributed/protocol/tests/test_protocol_utils.py::test_unpack_frames_remainder[456] PASSED [ 18%] distributed/protocol/tests/test_protocol_utils.py::test_unpack_frames_remainder[] PASSED [ 18%] distributed/protocol/tests/test_protocol_utils.py::test_unpack_frames_partial PASSED [ 18%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_empty PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_one PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_parts[slices0] PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_parts[slices1] PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_parts[slices2] PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_parts[slices3] PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_parts[slices4] PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_parts[slices5] PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_parts[slices6] PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_parts[slices7] PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_parts[slices8] PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_parts[slices9] PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_readonly_buffer PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_catch_non_memoryview PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_catch_gaps[slices0] PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_catch_gaps[slices1] PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_catch_gaps[slices2] PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_catch_different_buffer PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_catch_different_non_contiguous PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_catch_multidimensional PASSED [ 19%] distributed/protocol/tests/test_protocol_utils.py::TestMergeMemoryviews::test_catch_different_formats PASSED [ 19%] distributed/protocol/tests/test_scipy.py::test_serialize_scipy_sparse[dtype0-bsr_matrix] PASSED [ 19%] distributed/protocol/tests/test_scipy.py::test_serialize_scipy_sparse[dtype0-coo_matrix] PASSED [ 19%] distributed/protocol/tests/test_scipy.py::test_serialize_scipy_sparse[dtype0-csc_matrix] PASSED [ 19%] distributed/protocol/tests/test_scipy.py::test_serialize_scipy_sparse[dtype0-csr_matrix] PASSED [ 19%] distributed/protocol/tests/test_scipy.py::test_serialize_scipy_sparse[dtype0-dia_matrix] PASSED [ 19%] distributed/protocol/tests/test_scipy.py::test_serialize_scipy_sparse[dtype0-dok_matrix] PASSED [ 19%] distributed/protocol/tests/test_scipy.py::test_serialize_scipy_sparse[dtype0-lil_matrix] PASSED [ 19%] distributed/protocol/tests/test_scipy.py::test_serialize_scipy_sparse[dtype1-bsr_matrix] PASSED [ 19%] distributed/protocol/tests/test_scipy.py::test_serialize_scipy_sparse[dtype1-coo_matrix] PASSED [ 19%] distributed/protocol/tests/test_scipy.py::test_serialize_scipy_sparse[dtype1-csc_matrix] PASSED [ 19%] distributed/protocol/tests/test_scipy.py::test_serialize_scipy_sparse[dtype1-csr_matrix] PASSED [ 19%] distributed/protocol/tests/test_scipy.py::test_serialize_scipy_sparse[dtype1-dia_matrix] PASSED [ 19%] distributed/protocol/tests/test_scipy.py::test_serialize_scipy_sparse[dtype1-dok_matrix] PASSED [ 19%] distributed/protocol/tests/test_scipy.py::test_serialize_scipy_sparse[dtype1-lil_matrix] PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_dumps_serialize PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_bytestrings PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_empty_array PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_arrays[b] PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_arrays[B] PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_arrays[h] PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_arrays[H] PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_arrays[i] PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_arrays[I] PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_arrays[l] PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_arrays[L] PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_arrays[q] PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_arrays[Q] PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_arrays[f] PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_arrays[d] PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_Serialize PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_Serialized PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_nested_deserialize PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_iterate_collection PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_object_in_graph PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_scatter PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_inter_worker_comms PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_empty PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_empty_loads PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_empty_loads_deep PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_bytes[kwargs0] PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_bytes[kwargs1] PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_bytes_numpy[kwargs0] PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_bytes_numpy[kwargs1] PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_deserialize_bytes_zero_copy_read_only PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_deserialize_bytes_zero_copy_writeable PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_serialize_list_compress PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_malicious_exception PASSED [ 20%] distributed/protocol/tests/test_serialize.py::test_errors PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_err_on_bad_deserializer PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_context_specific_serialization PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_context_specific_serialization_class PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_serialize_raises PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_deeply_nested_structures[100] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_deeply_nested_structures[150] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_deeply_nested_structures[200] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_deeply_nested_structures[250] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_deeply_nested_structures[300] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_deeply_nested_structures[350] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_deeply_nested_structures[400] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_deeply_nested_structures[450] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_deeply_nested_structures[500] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_deeply_nested_structures[550] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_different_compression_families PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_frame_split PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_check_dask_serializable[data0-False] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_check_dask_serializable[data1-False] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_check_dask_serializable[data2-False] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_check_dask_serializable[data3-False] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_check_dask_serializable[data4-False] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_check_dask_serializable[data5-True] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_check_dask_serializable[data6-True] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_check_dask_serializable[data7-True] XFAIL [ 21%] distributed/protocol/tests/test_serialize.py::test_check_dask_serializable[data8-True] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_check_dask_serializable[data9-True] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_check_dask_serializable[data10-True] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_check_dask_serializable[data11-True] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_check_dask_serializable[data12-True] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_serialize_lists[serializers0] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_serialize_lists[serializers1] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_serialize_lists[serializers2] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_serialize_lists[serializers3] PASSED [ 21%] distributed/protocol/tests/test_serialize.py::test_deserialize_memoryview[data_in0] PASSED [ 22%] distributed/protocol/tests/test_serialize.py::test_deserialize_memoryview[data_in1] PASSED [ 22%] distributed/protocol/tests/test_serialize.py::test_serialize_memoryview_object PASSED [ 22%] distributed/protocol/tests/test_serialize.py::test_serialize_empty_1d_memoryview PASSED [ 22%] distributed/protocol/tests/test_serialize.py::test_serialize_empty_nd_memoryview PASSED [ 22%] distributed/protocol/tests/test_serialize.py::test_large_pickled_object PASSED [ 22%] distributed/protocol/tests/test_serialize.py::test__is_msgpack_serializable PASSED [ 22%] distributed/protocol/tests/test_to_pickle.py::test_ToPickle PASSED [ 22%] distributed/protocol/tests/test_utils.py::test_host_array PASSED [ 22%] distributed/protocol/tests/test_utils.py::test_host_array_from_buffers PASSED [ 22%] distributed/protocol/tests/test_utils.py::test_host_array_from_buffers_numpy PASSED [ 22%] distributed/protocol/tests/test_utils_test.py::test_get_host_array PASSED [ 22%] distributed/shuffle/tests/test_buffer.py::test_memory_limit[big_payloads0] PASSED [ 22%] distributed/shuffle/tests/test_buffer.py::test_memory_limit[big_payloads1] PASSED [ 22%] distributed/shuffle/tests/test_buffer.py::test_memory_limit[big_payloads2] PASSED [ 22%] distributed/shuffle/tests/test_buffer.py::test_memory_limit[big_payloads3] PASSED [ 22%] distributed/shuffle/tests/test_buffer.py::test_memory_limit_blocked_exception PASSED [ 22%] distributed/shuffle/tests/test_comm_buffer.py::test_basic PASSED [ 22%] distributed/shuffle/tests/test_comm_buffer.py::test_exceptions PASSED [ 22%] distributed/shuffle/tests/test_comm_buffer.py::test_slow_send PASSED [ 22%] distributed/shuffle/tests/test_comm_buffer.py::test_concurrent_puts PASSED [ 22%] distributed/shuffle/tests/test_comm_buffer.py::test_concurrent_puts_error PASSED [ 22%] distributed/shuffle/tests/test_core.py::test_mean_shard_size PASSED [ 22%] distributed/shuffle/tests/test_core.py::test_mean_shard_size_numpy PASSED [ 22%] distributed/shuffle/tests/test_disk_buffer.py::test_basic PASSED [ 22%] distributed/shuffle/tests/test_disk_buffer.py::test_read_before_flush PASSED [ 22%] distributed/shuffle/tests/test_disk_buffer.py::test_many[2] PASSED [ 22%] distributed/shuffle/tests/test_disk_buffer.py::test_many[100] PASSED [ 22%] distributed/shuffle/tests/test_disk_buffer.py::test_many[1000] PASSED [ 22%] distributed/shuffle/tests/test_disk_buffer.py::test_exceptions PASSED [ 22%] distributed/shuffle/tests/test_disk_buffer.py::test_high_pressure_flush_with_exception PASSED [ 22%] distributed/shuffle/tests/test_graph.py::test_raise_on_complex_numbers[csingle] PASSED [ 22%] distributed/shuffle/tests/test_graph.py::test_raise_on_complex_numbers[cdouble] PASSED [ 22%] distributed/shuffle/tests/test_graph.py::test_raise_on_complex_numbers[clongdouble] PASSED [ 22%] distributed/shuffle/tests/test_graph.py::test_raise_on_custom_objects XFAIL from custom objects from meta alone.) [ 23%] distributed/shuffle/tests/test_graph.py::test_raise_on_sparse_data PASSED [ 23%] distributed/shuffle/tests/test_graph.py::test_raise_on_non_string_column_name PASSED [ 23%] distributed/shuffle/tests/test_graph.py::test_does_not_raise_on_stringified_numeric_column_name PASSED [ 23%] distributed/shuffle/tests/test_limiter.py::test_limiter_basic PASSED [ 23%] distributed/shuffle/tests/test_limiter.py::test_unlimited_limiter PASSED [ 23%] distributed/shuffle/tests/test_limiter.py::test_limiter_concurrent_decrease_releases_waiter PASSED [ 23%] distributed/shuffle/tests/test_limiter.py::test_limiter_statistics PASSED [ 23%] distributed/shuffle/tests/test_limiter.py::test_limiter_no_limit_no_statistics PASSED [ 23%] distributed/shuffle/tests/test_memory_buffer.py::test_basic PASSED [ 23%] distributed/shuffle/tests/test_memory_buffer.py::test_read_before_flush PASSED [ 23%] distributed/shuffle/tests/test_memory_buffer.py::test_many[2] PASSED [ 23%] distributed/shuffle/tests/test_memory_buffer.py::test_many[100] PASSED [ 23%] distributed/shuffle/tests/test_memory_buffer.py::test_many[1000] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_basic_merge[inner] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_basic_merge[left] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_basic_merge[right] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_basic_merge[outer] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_merge_p2p_shuffle_reused_dataframe_with_different_parameters PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_merge_p2p_shuffle_reused_dataframe_with_same_parameters PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_merge[True-inner] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_merge[True-outer] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_merge[True-left] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_merge[True-right] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_merge[False-inner] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_merge[False-outer] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_merge[False-left] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_merge[False-right] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_index_merge_p2p[inner] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_index_merge_p2p[left] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_index_merge_p2p[right] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_index_merge_p2p[outer] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_merge_with_npartitions[4] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_merge_with_npartitions[5] PASSED [ 23%] distributed/shuffle/tests/test_merge.py::test_merge_with_npartitions[10] PASSED [ 24%] distributed/shuffle/tests/test_merge.py::test_merge_with_npartitions[20] PASSED [ 24%] distributed/shuffle/tests/test_merge.py::test_merge_does_not_deadlock_if_worker_joins PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_known_to_unknown[idx-inner] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_known_to_unknown[idx-left] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_known_to_unknown[idx-right] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_known_to_unknown[idx-outer] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_known_to_unknown[on1-inner] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_known_to_unknown[on1-left] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_known_to_unknown[on1-right] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_known_to_unknown[on1-outer] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_known_to_unknown[on2-inner] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_known_to_unknown[on2-left] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_known_to_unknown[on2-right] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_known_to_unknown[on2-outer] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_known_to_unknown[on3-inner] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_known_to_unknown[on3-left] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_known_to_unknown[on3-right] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_known_to_unknown[on3-outer] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_known[idx-inner] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_known[idx-left] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_known[idx-right] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_known[idx-outer] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_known[on1-inner] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_known[on1-left] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_known[on1-right] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_known[on1-outer] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_known[on2-inner] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_known[on2-left] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_known[on2-right] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_known[on2-outer] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_known[on3-inner] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_known[on3-left] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_known[on3-right] PASSED [ 24%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_known[on3-outer] PASSED [ 25%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_unknown[idx-inner] PASSED [ 25%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_unknown[idx-left] PASSED [ 25%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_unknown[idx-right] PASSED [ 25%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_unknown[idx-outer] PASSED [ 25%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_unknown[on1-inner] PASSED [ 25%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_unknown[on1-left] PASSED [ 25%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_unknown[on1-right] PASSED [ 25%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_unknown[on1-outer] PASSED [ 25%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_unknown[on2-inner] PASSED [ 25%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_unknown[on2-left] PASSED [ 25%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_unknown[on2-right] PASSED [ 25%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_unknown[on2-outer] PASSED [ 25%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_unknown[on3-inner] PASSED [ 25%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_unknown[on3-left] PASSED [ 25%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_unknown[on3-right] PASSED [ 25%] distributed/shuffle/tests/test_merge_column_and_index.py::test_merge_unknown_to_unknown[on3-outer] PASSED [ 25%] distributed/shuffle/tests/test_metrics.py::test_rechunk PASSED [ 25%] distributed/shuffle/tests/test_pickle.py::test_pickle PASSED [ 25%] distributed/shuffle/tests/test_pickle.py::test_pickle_numpy PASSED [ 25%] distributed/shuffle/tests/test_pickle.py::test_pickle_zero_copy PASSED [ 25%] distributed/shuffle/tests/test_read_write_lock.py::test_basic PASSED [ 25%] distributed/shuffle/tests/test_read_write_lock.py::test_lock_timeout PASSED [ 25%] distributed/shuffle/tests/test_read_write_lock.py::test_concurrent_reads_are_allowed PASSED [ 25%] distributed/shuffle/tests/test_read_write_lock.py::test_read_blocks_write PASSED [ 25%] distributed/shuffle/tests/test_read_write_lock.py::test_write_blocks_read PASSED [ 25%] distributed/shuffle/tests/test_read_write_lock.py::test_write_blocks_write PASSED [ 25%] distributed/shuffle/tests/test_read_write_lock.py::test_write_preferred_over_read PASSED [ 25%] distributed/shuffle/tests/test_rechunk.py::test_lowlevel_rechunk[True-True-1] PASSED [ 25%] distributed/shuffle/tests/test_rechunk.py::test_lowlevel_rechunk[True-True-10] PASSED [ 25%] distributed/shuffle/tests/test_rechunk.py::test_lowlevel_rechunk[True-False-1] PASSED [ 25%] distributed/shuffle/tests/test_rechunk.py::test_lowlevel_rechunk[True-False-10] PASSED [ 25%] distributed/shuffle/tests/test_rechunk.py::test_lowlevel_rechunk[False-True-1] PASSED [ 25%] distributed/shuffle/tests/test_rechunk.py::test_lowlevel_rechunk[False-True-10] PASSED [ 25%] distributed/shuffle/tests/test_rechunk.py::test_lowlevel_rechunk[False-False-1] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_lowlevel_rechunk[False-False-10] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_configuration[tasks-tasks] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_configuration[tasks-p2p] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_configuration[tasks-None] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_configuration[p2p-tasks] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_configuration[p2p-p2p] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_configuration[p2p-None] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_configuration[None-tasks] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_configuration[None-p2p] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_configuration[None-None] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_heuristic[new0-p2p] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_heuristic[new1-tasks] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_heuristic[new2-tasks] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_heuristic[new3-p2p] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_cull_p2p_rechunk_independent_partitions PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_cull_p2p_rechunking_single_chunk PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_cull_p2p_rechunk_overlapping_partitions PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_2d[True] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_2d[False] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_4d[True] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_4d[False] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_expand PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_expand2 PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_method PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_blockshape PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_dtype PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_dict PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_empty_input PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_null_dimensions PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_integer PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_0d PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_empty_array[arr0] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_empty_array[arr1] PASSED [ 26%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_empty_array[arr2] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_empty PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_zero_dim_array PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_zero_dim_array_II PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_same PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_same_fully_unknown PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_same_fully_unknown_floats PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_same_partially_unknown PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_zero_placeholders PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_minus_one PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_warning PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_unknown_from_pandas PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_unknown_from_array PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension[x0-chunks0] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension[x1-chunks1] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension[x2-chunks2] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension[x3-chunks3] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension[x4-chunks4] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension[x5-chunks5] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension[x6-chunks6] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension[x7-chunks7] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension[x8-chunks8] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension[x9-chunks9] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension[x10-chunks10] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension[x11-chunks11] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x0-chunks0] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x1-chunks1] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x2-chunks2] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x3-chunks3] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x4-chunks4] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x5-chunks5] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x6-chunks6] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x7-chunks7] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x8-chunks8] PASSED [ 27%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x9-chunks9] PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x10-chunks10] PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x11-chunks11] PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension_explicit[new_chunks0] PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension_explicit[new_chunks1] PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension_explicit[new_chunks2] PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_unknown_raises PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_zero_dim PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_empty_chunks PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_avoid_needless_chunking PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_auto_1d[100-1-10-expected0] PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_auto_1d[100-50-10-expected1] PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_auto_1d[100-100-10-expected2] PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_auto_1d[20-7-10-expected3] PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_auto_1d[20-chunks4-5-expected4] PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_auto_2d PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_auto_3d PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_auto_image_stack[100] PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_auto_image_stack[1000] PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_down PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_zero PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_bad_keys PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_zero PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_split_axes_1 PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_split_axes_2 PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_split_axes_nan PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_split_axes_nan_single PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_split_axes_nan_long PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_split_axes_with_nonzero PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_split_axes_with_zero PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_preserve_writeable_flag PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_in_memory_shards_dont_share_buffer PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_pick_worker_homogeneous_distribution[1] PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_pick_worker_homogeneous_distribution[2] PASSED [ 28%] distributed/shuffle/tests/test_rechunk.py::test_pick_worker_homogeneous_distribution[41] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_pick_worker_homogeneous_distribution[50] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_partial_rechunk_taskgroups PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_1d[old0-new0-expected0] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_1d[old1-new1-expected1] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_1d[old2-new2-expected2] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_1d[old3-new3-expected3] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_1d[old4-new4-expected4] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_2d[old0-new0-expected0] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_2d[old1-new1-expected1] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_2d[old2-new2-expected2] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_2d[old3-new3-expected3] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_2d[old4-new4-expected4] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_2d[old5-new5-expected5] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_3d[old0-new0-expected0] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_3d[old1-new1-expected1] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_3d[old2-new2-expected2] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_3d[old3-new3-expected3] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_concatenation[1 B-expected0] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_concatenation[20 B-expected1] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_concatenation[40 B-expected2] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_concatenation[100 B-expected3] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_does_not_concatenate_object_type PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_splitting[old0-new0-expected0] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_splitting[old1-new1-expected1] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_splitting[old2-new2-expected2] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_homogeneously_schedule_unpack PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_datetime[tasks] PASSED [ 29%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_datetime[p2p] PASSED [ 29%] distributed/shuffle/tests/test_shuffle_plugins.py::test_installation_on_worker PASSED [ 29%] distributed/shuffle/tests/test_shuffle_plugins.py::test_installation_on_scheduler PASSED [ 29%] distributed/tests/test_active_memory_manager.py::test_no_policies PASSED [ 29%] distributed/tests/test_active_memory_manager.py::test_drop PASSED [ 29%] distributed/tests/test_active_memory_manager.py::test_start_stop PASSED [ 29%] distributed/tests/test_active_memory_manager.py::test_auto_start PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_add_policy PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_multi_start PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_not_registered PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_client_proxy_sync PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_client_proxy_async PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_drop_not_in_memory PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_drop_with_waiter PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_double_drop PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_double_drop_stress PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_drop_from_worker_with_least_free_memory PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_drop_with_candidates PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_drop_with_empty_candidates PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_drop_from_candidates_without_key PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_drop_with_bad_candidates PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_drop_prefers_paused_workers PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_drop_with_paused_workers_with_running_tasks_1 PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_drop_with_paused_workers_with_running_tasks_2 PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_drop_with_paused_workers_with_running_tasks_3_4[True] PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_drop_with_paused_workers_with_running_tasks_3_4[False] PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_drop_with_paused_workers_with_running_tasks_5 PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_replicate PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_replicate_not_in_memory PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_double_replicate_stress PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_replicate_to_worker_with_most_free_memory PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_replicate_with_candidates PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_replicate_with_empty_candidates PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_replicate_to_candidates_with_key PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_replicate_avoids_paused_workers_1 PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_replicate_avoids_paused_workers_2 PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_bad_measure PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_ReduceReplicas PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_ReduceReplicas_with_waiters[0-0-0-1] PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_ReduceReplicas_with_waiters[1-0-0-1] PASSED [ 30%] distributed/tests/test_active_memory_manager.py::test_ReduceReplicas_with_waiters[0-0-1-1] PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_ReduceReplicas_with_waiters[2-0-0-1] PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_ReduceReplicas_with_waiters[1-1-0-2] PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_ReduceReplicas_with_waiters[1-1-1-3] PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_ReduceReplicas_with_waiters[1-1-2-4] PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_ReduceReplicas_with_waiters[2-1-1-3] PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_ReduceReplicas_with_waiters[1-1-17-4] PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_ReduceReplicas_with_waiters[17-1-1-3] PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_ReduceReplicas_with_waiters[18-1-1-4] PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_RetireWorker_amm_on_off[False] PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_RetireWorker_amm_on_off[True] PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_RetireWorker_no_extension PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_RetireWorker_no_remove PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_RetireWorker_with_ReduceReplicas[False] PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_RetireWorker_with_ReduceReplicas[True] PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_RetireWorker_all_replicas_are_being_retired PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_RetireWorker_no_recipients PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_RetireWorker_all_recipients_are_paused PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_RetireWorker_new_keys_arrive_after_all_keys_moved_away XFAIL completed.) [ 31%] distributed/tests/test_active_memory_manager.py::test_RetireWorker_faulty_recipient PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_RetireWorker_mass PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_RetireWorker_incremental PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_dont_drop_actors PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_dont_replicate_actors PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_RetireWorker_with_actor[False] PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_RetireWorker_with_actor[True] PASSED [ 31%] distributed/tests/test_active_memory_manager.py::test_RetireWorker_with_actor_proxy PASSED [ 31%] distributed/tests/test_actor.py::test_client_actions[True] PASSED [ 31%] distributed/tests/test_actor.py::test_client_actions[False] PASSED [ 31%] distributed/tests/test_actor.py::test_worker_actions[False] PASSED [ 31%] distributed/tests/test_actor.py::test_worker_actions[True] PASSED [ 31%] distributed/tests/test_actor.py::test_Actor PASSED [ 31%] distributed/tests/test_actor.py::test_linear_access XFAIL (Tornado can pass things out of orderShould rely on sending small messages rather than rpc) [ 31%] distributed/tests/test_actor.py::test_exceptions_create PASSED [ 31%] distributed/tests/test_actor.py::test_exceptions_method PASSED [ 32%] distributed/tests/test_actor.py::test_gc PASSED [ 32%] distributed/tests/test_actor.py::test_track_dependencies PASSED [ 32%] distributed/tests/test_actor.py::test_future PASSED [ 32%] distributed/tests/test_actor.py::test_future_dependencies PASSED [ 32%] distributed/tests/test_actor.py::test_sync PASSED [ 32%] distributed/tests/test_actor.py::test_timeout PASSED [ 32%] distributed/tests/test_actor.py::test_failed_worker PASSED [ 32%] distributed/tests/test_actor.py::test_failed_worker_before_future_update PASSED [ 32%] distributed/tests/test_actor.py::test_numpy_roundtrip PASSED [ 32%] distributed/tests/test_actor.py::test_numpy_roundtrip_getattr PASSED [ 32%] distributed/tests/test_actor.py::test_repr PASSED [ 32%] distributed/tests/test_actor.py::test_dir PASSED [ 32%] distributed/tests/test_actor.py::test_many_computations PASSED [ 32%] distributed/tests/test_actor.py::test_thread_safety PASSED [ 32%] distributed/tests/test_actor.py::test_Actors_create_dependencies PASSED [ 32%] distributed/tests/test_actor.py::test_load_balance PASSED [ 32%] distributed/tests/test_actor.py::test_load_balance_map PASSED [ 32%] distributed/tests/test_actor.py::test_compute PASSED [ 32%] distributed/tests/test_actor.py::test_compute_sync PASSED [ 32%] distributed/tests/test_actor.py::test_actors_in_profile PASSED [ 32%] distributed/tests/test_actor.py::test_waiter PASSED [ 32%] distributed/tests/test_actor.py::test_worker_actor_handle_is_weakref PASSED [ 32%] distributed/tests/test_actor.py::test_worker_actor_handle_is_weakref_sync PASSED [ 32%] distributed/tests/test_actor.py::test_worker_actor_handle_is_weakref_from_compute_sync PASSED [ 32%] distributed/tests/test_actor.py::test_one_thread_deadlock PASSED [ 32%] distributed/tests/test_actor.py::test_one_thread_deadlock_timeout PASSED [ 32%] distributed/tests/test_actor.py::test_one_thread_deadlock_sync_client PASSED [ 32%] distributed/tests/test_actor.py::test_async_deadlock PASSED [ 32%] distributed/tests/test_actor.py::test_exception PASSED [ 32%] distributed/tests/test_actor.py::test_exception_async PASSED [ 32%] distributed/tests/test_actor.py::test_as_completed PASSED [ 32%] distributed/tests/test_actor.py::test_actor_future_awaitable PASSED [ 32%] distributed/tests/test_actor.py::test_actor_future_awaitable_deadlock PASSED [ 32%] distributed/tests/test_actor.py::test_serialize_with_pickle PASSED [ 33%] distributed/tests/test_actor.py::test_worker_client_async PASSED [ 33%] distributed/tests/test_actor.py::test_worker_client_separate_thread PASSED [ 33%] distributed/tests/test_actor.py::test_get_worker PASSED [ 33%] distributed/tests/test_actor.py::test_actor_worker_host_leaves_gracefully PASSED [ 33%] distributed/tests/test_actor.py::test_actor_worker_host_dies PASSED [ 33%] distributed/tests/test_as_completed.py::test_as_completed_async PASSED [ 33%] distributed/tests/test_as_completed.py::test_as_completed_timeout_async PASSED [ 33%] distributed/tests/test_as_completed.py::test_as_completed_timeout_sync PASSED [ 33%] distributed/tests/test_as_completed.py::test_as_completed_sync PASSED [ 33%] distributed/tests/test_as_completed.py::test_as_completed_with_non_futures PASSED [ 33%] distributed/tests/test_as_completed.py::test_as_completed_add PASSED [ 33%] distributed/tests/test_as_completed.py::test_as_completed_update PASSED [ 33%] distributed/tests/test_as_completed.py::test_as_completed_repeats PASSED [ 33%] distributed/tests/test_as_completed.py::test_as_completed_is_empty PASSED [ 33%] distributed/tests/test_as_completed.py::test_as_completed_cancel PASSED [ 33%] distributed/tests/test_as_completed.py::test_as_completed_cancel_last PASSED [ 33%] distributed/tests/test_as_completed.py::test_async_for_py2_equivalent PASSED [ 33%] distributed/tests/test_as_completed.py::test_as_completed_error_async PASSED [ 33%] distributed/tests/test_as_completed.py::test_as_completed_error PASSED [ 33%] distributed/tests/test_as_completed.py::test_as_completed_with_results PASSED [ 33%] distributed/tests/test_as_completed.py::test_as_completed_with_results_async PASSED [ 33%] distributed/tests/test_as_completed.py::test_as_completed_with_results_no_raise PASSED [ 33%] distributed/tests/test_as_completed.py::test_str PASSED [ 33%] distributed/tests/test_as_completed.py::test_as_completed_with_results_no_raise_async PASSED [ 33%] distributed/tests/test_as_completed.py::test_clear PASSED [ 33%] distributed/tests/test_async_task_group.py::test_async_task_group_initialization PASSED [ 33%] distributed/tests/test_async_task_group.py::test_async_task_group_call_soon_executes_task_in_background PASSED [ 33%] distributed/tests/test_async_task_group.py::test_async_task_group_call_later_executes_delayed_task_in_background PASSED [ 33%] distributed/tests/test_async_task_group.py::test_async_task_group_close_closes PASSED [ 33%] distributed/tests/test_async_task_group.py::test_async_task_group_close_does_not_cancel_existing_tasks PASSED [ 33%] distributed/tests/test_async_task_group.py::test_async_task_group_close_prohibits_new_tasks PASSED [ 33%] distributed/tests/test_async_task_group.py::test_async_task_group_stop_disallows_shutdown PASSED [ 33%] distributed/tests/test_async_task_group.py::test_async_task_group_stop_cancels_long_running PASSED [ 33%] distributed/tests/test_asyncio.py::test_rlock PASSED [ 33%] distributed/tests/test_asyncio.py::test_rlock_nonreentrant PASSED [ 34%] distributed/tests/test_asyncio.py::test_rlock_none PASSED [ 34%] distributed/tests/test_asyncio.py::test_rlock_release_on_raise PASSED [ 34%] distributed/tests/test_asyncprocess.py::test_simple PASSED [ 34%] distributed/tests/test_asyncprocess.py::test_exitcode PASSED [ 34%] distributed/tests/test_asyncprocess.py::test_sigint_from_same_process PASSED [ 34%] distributed/tests/test_asyncprocess.py::test_sigterm_from_parent_process PASSED [ 34%] distributed/tests/test_asyncprocess.py::test_terminate PASSED [ 34%] distributed/tests/test_asyncprocess.py::test_close PASSED [ 34%] distributed/tests/test_asyncprocess.py::test_exit_callback PASSED [ 34%] distributed/tests/test_asyncprocess.py::test_child_main_thread PASSED [ 34%] distributed/tests/test_asyncprocess.py::test_num_fds PASSED [ 34%] distributed/tests/test_asyncprocess.py::test_terminate_after_stop PASSED [ 34%] distributed/tests/test_asyncprocess.py::test_kill PASSED [ 34%] distributed/tests/test_asyncprocess.py::test_asyncprocess_child_teardown_on_parent_exit PASSED [ 34%] distributed/tests/test_batched.py::test_BatchedSend PASSED [ 34%] distributed/tests/test_batched.py::test_send_before_start PASSED [ 34%] distributed/tests/test_batched.py::test_send_after_stream_start PASSED [ 34%] distributed/tests/test_batched.py::test_send_before_close PASSED [ 34%] distributed/tests/test_batched.py::test_close_closed PASSED [ 34%] distributed/tests/test_batched.py::test_close_not_started PASSED [ 34%] distributed/tests/test_batched.py::test_close_twice PASSED [ 34%] distributed/tests/test_batched.py::test_sending_traffic_jam PASSED [ 34%] distributed/tests/test_batched.py::test_serializers PASSED [ 34%] distributed/tests/test_cancelled_state.py::test_abort_execution_release PASSED [ 34%] distributed/tests/test_cancelled_state.py::test_abort_execution_reschedule PASSED [ 34%] distributed/tests/test_cancelled_state.py::test_abort_execution_add_as_dependency PASSED [ 34%] distributed/tests/test_cancelled_state.py::test_abort_execution_to_fetch PASSED [ 34%] distributed/tests/test_cancelled_state.py::test_worker_stream_died_during_comm PASSED [ 34%] distributed/tests/test_cancelled_state.py::test_flight_to_executing_via_cancelled_resumed PASSED [ 34%] distributed/tests/test_cancelled_state.py::test_executing_cancelled_error PASSED [ 34%] distributed/tests/test_cancelled_state.py::test_flight_cancelled_error PASSED [ 34%] distributed/tests/test_cancelled_state.py::test_in_flight_lost_after_resumed PASSED [ 34%] distributed/tests/test_cancelled_state.py::test_cancelled_error PASSED [ 34%] distributed/tests/test_cancelled_state.py::test_cancelled_error_with_resources PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_cancelled_resumed_after_flight_with_dependencies PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_cancelled_resumed_after_flight_with_dependencies_workerstate PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_resumed_cancelled_handle_compute[True-True] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_resumed_cancelled_handle_compute[True-False] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_resumed_cancelled_handle_compute[False-True] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_resumed_cancelled_handle_compute[False-False] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_cancelled_handle_compute[True] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_cancelled_handle_compute[False] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_cancelled_task_error_rejected PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_deadlock_cancelled_after_inflight_before_gather_from_worker[True-resumed] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_deadlock_cancelled_after_inflight_before_gather_from_worker[True-cancelled] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_workerstate_executing_to_executing[executing] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_workerstate_executing_to_executing[long-running] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_workerstate_flight_to_flight PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_workerstate_executing_skips_fetch_on_success[executing] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_workerstate_executing_skips_fetch_on_success[long-running] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_workerstate_executing_failure_to_fetch[executing] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_workerstate_executing_failure_to_fetch[long-running] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_workerstate_flight_skips_executing_on_success PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_workerstate_flight_failure_to_executing[False] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_workerstate_flight_failure_to_executing[True] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_workerstate_resumed_fetch_to_cancelled_to_executing[executing] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_workerstate_resumed_fetch_to_cancelled_to_executing[long-running] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_workerstate_resumed_fetch_to_executing[executing] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_workerstate_resumed_fetch_to_executing[long-running] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_workerstate_resumed_waiting_to_flight PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_execute_preamble_early_cancel[executing-False] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_execute_preamble_early_cancel[executing-True] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_execute_preamble_early_cancel[resumed-False] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_execute_preamble_early_cancel[resumed-True] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_cancel_with_dependencies_in_memory[ExecuteSuccessEvent-False] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_cancel_with_dependencies_in_memory[ExecuteSuccessEvent-True] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_cancel_with_dependencies_in_memory[ExecuteFailureEvent-False] PASSED [ 35%] distributed/tests/test_cancelled_state.py::test_cancel_with_dependencies_in_memory[ExecuteFailureEvent-True] PASSED [ 36%] distributed/tests/test_cancelled_state.py::test_cancel_with_dependencies_in_memory[RescheduleEvent-False] PASSED [ 36%] distributed/tests/test_cancelled_state.py::test_cancel_with_dependencies_in_memory[RescheduleEvent-True] PASSED [ 36%] distributed/tests/test_cancelled_state.py::test_secede_cancelled_or_resumed_workerstate[ExecuteSuccessEvent-False-False] PASSED [ 36%] distributed/tests/test_cancelled_state.py::test_secede_cancelled_or_resumed_workerstate[ExecuteSuccessEvent-False-True] PASSED [ 36%] distributed/tests/test_cancelled_state.py::test_secede_cancelled_or_resumed_workerstate[ExecuteSuccessEvent-True-False] PASSED [ 36%] distributed/tests/test_cancelled_state.py::test_secede_cancelled_or_resumed_workerstate[ExecuteSuccessEvent-True-True] PASSED [ 36%] distributed/tests/test_cancelled_state.py::test_secede_cancelled_or_resumed_workerstate[ExecuteFailureEvent-False-False] PASSED [ 36%] distributed/tests/test_cancelled_state.py::test_secede_cancelled_or_resumed_workerstate[ExecuteFailureEvent-False-True] PASSED [ 36%] distributed/tests/test_cancelled_state.py::test_secede_cancelled_or_resumed_workerstate[ExecuteFailureEvent-True-False] PASSED [ 36%] distributed/tests/test_cancelled_state.py::test_secede_cancelled_or_resumed_workerstate[ExecuteFailureEvent-True-True] PASSED [ 36%] distributed/tests/test_cancelled_state.py::test_secede_cancelled_or_resumed_workerstate[RescheduleEvent-False-False] PASSED [ 36%] distributed/tests/test_cancelled_state.py::test_secede_cancelled_or_resumed_workerstate[RescheduleEvent-False-True] PASSED [ 36%] distributed/tests/test_cancelled_state.py::test_secede_cancelled_or_resumed_workerstate[RescheduleEvent-True-False] PASSED [ 36%] distributed/tests/test_cancelled_state.py::test_secede_cancelled_or_resumed_workerstate[RescheduleEvent-True-True] PASSED [ 36%] distributed/tests/test_cancelled_state.py::test_secede_racing_cancellation_and_scheduling_on_other_worker PASSED [ 36%] distributed/tests/test_cancelled_state.py::test_secede_racing_resuming_on_same_worker PASSED [ 36%] distributed/tests/test_cancelled_state.py::test_secede_cancelled_or_resumed_scheduler PASSED [ 36%] distributed/tests/test_cancelled_state.py::test_workerstate_remove_replica_of_cancelled_task_dependency PASSED [ 36%] distributed/tests/test_chaos.py::test_KillWorker[sys.exit] PASSED [ 36%] distributed/tests/test_chaos.py::test_KillWorker[graceful] PASSED [ 36%] distributed/tests/test_chaos.py::test_KillWorker[segfault] PASSED [ 36%] distributed/tests/test_client.py::test_submit PASSED [ 36%] distributed/tests/test_client.py::test_map PASSED [ 36%] distributed/tests/test_client.py::test_map_empty PASSED [ 36%] distributed/tests/test_client.py::test_map_keynames PASSED [ 36%] distributed/tests/test_client.py::test_map_retries PASSED [ 36%] distributed/tests/test_client.py::test_map_batch_size PASSED [ 36%] distributed/tests/test_client.py::test_custom_key_with_batches PASSED [ 36%] distributed/tests/test_client.py::test_compute_no_collection_or_future PASSED [ 36%] distributed/tests/test_client.py::test_compute_retries PASSED [ 36%] distributed/tests/test_client.py::test_compute_retries_annotations PASSED [ 36%] distributed/tests/test_client.py::test_retries_get PASSED [ 36%] distributed/tests/test_client.py::test_persist_retries PASSED [ 36%] distributed/tests/test_client.py::test_persist_retries_annotations PASSED [ 37%] distributed/tests/test_client.py::test_retries_dask_array PASSED [ 37%] distributed/tests/test_client.py::test_future_repr PASSED [ 37%] distributed/tests/test_client.py::test_future_tuple_repr PASSED [ 37%] distributed/tests/test_client.py::test_Future_exception PASSED [ 37%] distributed/tests/test_client.py::test_Future_exception_sync PASSED [ 37%] distributed/tests/test_client.py::test_Future_release PASSED [ 37%] distributed/tests/test_client.py::test_Future_release_sync PASSED [ 37%] distributed/tests/test_client.py::test_short_tracebacks[result] PASSED [ 37%] distributed/tests/test_client.py::test_short_tracebacks[gather] PASSED [ 37%] distributed/tests/test_client.py::test_short_tracebacks_async[await] PASSED [ 37%] distributed/tests/test_client.py::test_short_tracebacks_async[result] PASSED [ 37%] distributed/tests/test_client.py::test_short_tracebacks_async[gather] PASSED [ 37%] distributed/tests/test_client.py::test_map_naming PASSED [ 37%] distributed/tests/test_client.py::test_submit_naming PASSED [ 37%] distributed/tests/test_client.py::test_exceptions PASSED [ 37%] distributed/tests/test_client.py::test_gc PASSED [ 37%] distributed/tests/test_client.py::test_thread PASSED [ 37%] distributed/tests/test_client.py::test_sync_exceptions PASSED [ 37%] distributed/tests/test_client.py::test_gather PASSED [ 37%] distributed/tests/test_client.py::test_gather_mismatched_client PASSED [ 37%] distributed/tests/test_client.py::test_gather_lost PASSED [ 37%] distributed/tests/test_client.py::test_gather_sync PASSED [ 37%] distributed/tests/test_client.py::test_gather_strict PASSED [ 37%] distributed/tests/test_client.py::test_gather_skip PASSED [ 37%] distributed/tests/test_client.py::test_limit_concurrent_gathering PASSED [ 37%] distributed/tests/test_client.py::test_get PASSED [ 37%] distributed/tests/test_client.py::test_get_sync PASSED [ 37%] distributed/tests/test_client.py::test_no_future_references PASSED [ 37%] distributed/tests/test_client.py::test_get_sync_optimize_graph_passes_through PASSED [ 37%] distributed/tests/test_client.py::test_gather_errors PASSED [ 37%] distributed/tests/test_client.py::test_wait PASSED [ 37%] distributed/tests/test_client.py::test_wait_first_completed PASSED [ 37%] distributed/tests/test_client.py::test_wait_timeout PASSED [ 37%] distributed/tests/test_client.py::test_wait_sync PASSED [ 38%] distributed/tests/test_client.py::test_wait_informative_error_for_timeouts PASSED [ 38%] distributed/tests/test_client.py::test_garbage_collection PASSED [ 38%] distributed/tests/test_client.py::test_garbage_collection_with_scatter PASSED [ 38%] distributed/tests/test_client.py::test_recompute_released_key PASSED [ 38%] distributed/tests/test_client.py::test_tokenize_on_futures PASSED [ 38%] distributed/tests/test_client.py::test_restrictions_submit PASSED [ 38%] distributed/tests/test_client.py::test_restrictions_ip_port PASSED [ 38%] distributed/tests/test_client.py::test_restrictions_map PASSED [ 38%] distributed/tests/test_client.py::test_restrictions_get PASSED [ 38%] distributed/tests/test_client.py::test_restrictions_get_annotate PASSED [ 38%] distributed/tests/test_client.py::test_remove_worker PASSED [ 38%] distributed/tests/test_client.py::test_errors_dont_block PASSED [ 38%] distributed/tests/test_client.py::test_submit_quotes PASSED [ 38%] distributed/tests/test_client.py::test_map_quotes PASSED [ 38%] distributed/tests/test_client.py::test_two_consecutive_clients_share_results PASSED [ 38%] distributed/tests/test_client.py::test_submit_then_get_with_Future PASSED [ 38%] distributed/tests/test_client.py::test_aliases PASSED [ 38%] distributed/tests/test_client.py::test_aliases_2 PASSED [ 38%] distributed/tests/test_client.py::test_scatter PASSED [ 38%] distributed/tests/test_client.py::test_scatter_types PASSED [ 38%] distributed/tests/test_client.py::test_scatter_collection_subclass PASSED [ 38%] distributed/tests/test_client.py::test_scatter_namedtuple PASSED [ 38%] distributed/tests/test_client.py::test_scatter_non_list PASSED [ 38%] distributed/tests/test_client.py::test_scatter_tokenize_local PASSED [ 38%] distributed/tests/test_client.py::test_scatter_singletons PASSED [ 38%] distributed/tests/test_client.py::test_scatter_typename PASSED [ 38%] distributed/tests/test_client.py::test_scatter_hash PASSED [ 38%] distributed/tests/test_client.py::test_scatter_hash_2 PASSED [ 38%] distributed/tests/test_client.py::test_get_releases_data PASSED [ 38%] distributed/tests/test_client.py::test_current PASSED [ 38%] distributed/tests/test_client.py::test_current_nested PASSED [ 38%] distributed/tests/test_client.py::test_current_nested_async PASSED [ 38%] distributed/tests/test_client.py::test_current_concurrent PASSED [ 38%] distributed/tests/test_client.py::test_context_manager_used_from_different_tasks PASSED [ 39%] distributed/tests/test_client.py::test_context_manager_used_from_different_threads PASSED [ 39%] distributed/tests/test_client.py::test_global_clients PASSED [ 39%] distributed/tests/test_client.py::test_exception_on_exception PASSED [ 39%] distributed/tests/test_client.py::test_get_task_prefix_states PASSED [ 39%] distributed/tests/test_client.py::test_get_nbytes PASSED [ 39%] distributed/tests/test_client.py::test_nbytes_determines_worker PASSED [ 39%] distributed/tests/test_client.py::test_if_intermediates_clear_on_error PASSED [ 39%] distributed/tests/test_client.py::test_pragmatic_move_small_data_to_large_data PASSED [ 39%] distributed/tests/test_client.py::test_get_with_non_list_key PASSED [ 39%] distributed/tests/test_client.py::test_get_with_error PASSED [ 39%] distributed/tests/test_client.py::test_get_with_error_sync PASSED [ 39%] distributed/tests/test_client.py::test_directed_scatter PASSED [ 39%] distributed/tests/test_client.py::test_scatter_direct PASSED [ 39%] distributed/tests/test_client.py::test_scatter_direct_2 PASSED [ 39%] distributed/tests/test_client.py::test_scatter_direct_numpy PASSED [ 39%] distributed/tests/test_client.py::test_scatter_direct_broadcast PASSED [ 39%] distributed/tests/test_client.py::test_scatter_direct_balanced PASSED [ 39%] distributed/tests/test_client.py::test_scatter_direct_broadcast_target PASSED [ 39%] distributed/tests/test_client.py::test_scatter_direct_empty PASSED [ 39%] distributed/tests/test_client.py::test_scatter_direct_spread_evenly PASSED [ 39%] distributed/tests/test_client.py::test_scatter_gather_sync[True-True] PASSED [ 39%] distributed/tests/test_client.py::test_scatter_gather_sync[True-False] PASSED [ 39%] distributed/tests/test_client.py::test_scatter_gather_sync[False-True] PASSED [ 39%] distributed/tests/test_client.py::test_scatter_gather_sync[False-False] PASSED [ 39%] distributed/tests/test_client.py::test_gather_direct PASSED [ 39%] distributed/tests/test_client.py::test_many_submits_spread_evenly PASSED [ 39%] distributed/tests/test_client.py::test_traceback PASSED [ 39%] distributed/tests/test_client.py::test_get_traceback PASSED [ 39%] distributed/tests/test_client.py::test_gather_traceback PASSED [ 39%] distributed/tests/test_client.py::test_traceback_sync PASSED [ 39%] distributed/tests/test_client.py::test_upload_file PASSED [ 39%] distributed/tests/test_client.py::test_upload_file_refresh_delayed PASSED [ 39%] distributed/tests/test_client.py::test_upload_file_no_extension PASSED [ 39%] distributed/tests/test_client.py::test_upload_file_zip SKIPPED (Upstream issue in CPython. See https://github.com/dask/distributed/issues/8708 and https://github.com/python/cpython/issues/121342.) [ 40%] distributed/tests/test_client.py::test_upload_large_file PASSED [ 40%] distributed/tests/test_client.py::test_upload_file_sync PASSED [ 40%] distributed/tests/test_client.py::test_upload_file_exception PASSED [ 40%] distributed/tests/test_client.py::test_upload_file_exception_sync PASSED [ 40%] distributed/tests/test_client.py::test_upload_file_load PASSED [ 40%] distributed/tests/test_client.py::test_upload_file_new_worker PASSED [ 40%] distributed/tests/test_client.py::test_multiple_clients SKIPPED [ 40%] distributed/tests/test_client.py::test_async_compute PASSED [ 40%] distributed/tests/test_client.py::test_async_compute_with_scatter PASSED [ 40%] distributed/tests/test_client.py::test_sync_compute PASSED [ 40%] distributed/tests/test_client.py::test_remote_scatter_gather PASSED [ 40%] distributed/tests/test_client.py::test_remote_submit_on_Future PASSED [ 40%] distributed/tests/test_client.py::test_start_is_idempotent PASSED [ 40%] distributed/tests/test_client.py::test_client_with_scheduler PASSED [ 40%] distributed/tests/test_client.py::test_allow_restrictions PASSED [ 40%] distributed/tests/test_client.py::test_bad_address PASSED [ 40%] distributed/tests/test_client.py::test_informative_error_on_cluster_type PASSED [ 40%] distributed/tests/test_client.py::test_long_error PASSED [ 40%] distributed/tests/test_client.py::test_map_on_futures_with_kwargs PASSED [ 40%] distributed/tests/test_client.py::test_badly_serialized_input_stderr SKIPPED [ 40%] distributed/tests/test_client.py::test_repr[str] PASSED [ 40%] distributed/tests/test_client.py::test_repr[repr] PASSED [ 40%] distributed/tests/test_client.py::test_repr[func2] PASSED [ 40%] distributed/tests/test_client.py::test_repr_async PASSED [ 40%] distributed/tests/test_client.py::test_repr_no_memory_limit PASSED [ 40%] distributed/tests/test_client.py::test_repr_localcluster PASSED [ 40%] distributed/tests/test_client.py::test_forget_simple PASSED [ 40%] distributed/tests/test_client.py::test_forget_complex PASSED [ 40%] distributed/tests/test_client.py::test_forget_in_flight PASSED [ 40%] distributed/tests/test_client.py::test_forget_errors PASSED [ 40%] distributed/tests/test_client.py::test_repr_sync PASSED [ 40%] distributed/tests/test_client.py::test_multi_client PASSED [ 40%] distributed/tests/test_client.py::test_cleanup_after_broken_client_connection PASSED [ 40%] distributed/tests/test_client.py::test_multi_garbage_collection PASSED [ 41%] distributed/tests/test_client.py::test__broadcast PASSED [ 41%] distributed/tests/test_client.py::test__broadcast_integer PASSED [ 41%] distributed/tests/test_client.py::test__broadcast_dict PASSED [ 41%] distributed/tests/test_client.py::test_proxy PASSED [ 41%] distributed/tests/test_client.py::test_cancel PASSED [ 41%] distributed/tests/test_client.py::test_cancel_tuple_key PASSED [ 41%] distributed/tests/test_client.py::test_cancel_multi_client PASSED [ 41%] distributed/tests/test_client.py::test_cancel_before_known_to_scheduler PASSED [ 41%] distributed/tests/test_client.py::test_cancel_collection PASSED [ 41%] distributed/tests/test_client.py::test_cancel_sync PASSED [ 41%] distributed/tests/test_client.py::test_future_type PASSED [ 41%] distributed/tests/test_client.py::test_traceback_clean PASSED [ 41%] distributed/tests/test_client.py::test_map_different_lengths PASSED [ 41%] distributed/tests/test_client.py::test_Future_exception_sync_2 PASSED [ 41%] distributed/tests/test_client.py::test_async_persist PASSED [ 41%] distributed/tests/test_client.py::test_persist_async PASSED [ 41%] distributed/tests/test_client.py::test_persist PASSED [ 41%] distributed/tests/test_client.py::test_long_traceback PASSED [ 41%] distributed/tests/test_client.py::test_wait_on_collections PASSED [ 41%] distributed/tests/test_client.py::test_futures_of_get PASSED [ 41%] distributed/tests/test_client.py::test_futures_of_class PASSED [ 41%] distributed/tests/test_client.py::test_futures_of_cancelled_raises PASSED [ 41%] distributed/tests/test_client.py::test_balance_tasks_by_stacks SKIPPED [ 41%] distributed/tests/test_client.py::test_run PASSED [ 41%] distributed/tests/test_client.py::test_run_handles_picklable_data PASSED [ 41%] distributed/tests/test_client.py::test_run_sync PASSED [ 41%] distributed/tests/test_client.py::test_run_coroutine PASSED [ 41%] distributed/tests/test_client.py::test_run_coroutine_sync PASSED [ 41%] distributed/tests/test_client.py::test_run_exception PASSED [ 41%] distributed/tests/test_client.py::test_run_rpc_error PASSED [ 41%] distributed/tests/test_client.py::test_diagnostic_ui PASSED [ 41%] distributed/tests/test_client.py::test_diagnostic_nbytes_sync PASSED [ 41%] distributed/tests/test_client.py::test_diagnostic_nbytes PASSED [ 41%] distributed/tests/test_client.py::test_worker_aliases PASSED [ 42%] distributed/tests/test_client.py::test_persist_get_sync PASSED [ 42%] distributed/tests/test_client.py::test_persist_get[True] PASSED [ 42%] distributed/tests/test_client.py::test_persist_get[False] PASSED [ 42%] distributed/tests/test_client.py::test_client_num_fds PASSED [ 42%] distributed/tests/test_client.py::test_startup_close_startup PASSED [ 42%] distributed/tests/test_client.py::test_startup_close_startup_sync PASSED [ 42%] distributed/tests/test_client.py::test_badly_serialized_exceptions PASSED [ 42%] distributed/tests/test_client.py::test_rebalance PASSED [ 42%] distributed/tests/test_client.py::test_rebalance_workers_and_keys PASSED [ 42%] distributed/tests/test_client.py::test_rebalance_sync PASSED [ 42%] distributed/tests/test_client.py::test_rebalance_unprepared PASSED [ 42%] distributed/tests/test_client.py::test_receive_lost_key PASSED [ 42%] distributed/tests/test_client.py::test_unrunnable_task_runs PASSED [ 42%] distributed/tests/test_client.py::test_add_worker_after_tasks PASSED [ 42%] distributed/tests/test_client.py::test_workers_register_indirect_data PASSED [ 42%] distributed/tests/test_client.py::test_submit_on_cancelled_future PASSED [ 42%] distributed/tests/test_client.py::test_compute_partially_forgotten[True-True] PASSED [ 42%] distributed/tests/test_client.py::test_compute_partially_forgotten[True-False] PASSED [ 42%] distributed/tests/test_client.py::test_compute_partially_forgotten[False-True] PASSED [ 42%] distributed/tests/test_client.py::test_compute_partially_forgotten[False-False] PASSED [ 42%] distributed/tests/test_client.py::test_replicate PASSED [ 42%] distributed/tests/test_client.py::test_replicate_tuple_keys PASSED [ 42%] distributed/tests/test_client.py::test_replicate_workers PASSED [ 42%] distributed/tests/test_client.py::test_replicate_tree_branching PASSED [ 42%] distributed/tests/test_client.py::test_client_replicate PASSED [ 42%] distributed/tests/test_client.py::test_client_replicate_host PASSED [ 42%] distributed/tests/test_client.py::test_client_replicate_sync PASSED [ 42%] distributed/tests/test_client.py::test_task_load_adapts_quickly PASSED [ 42%] distributed/tests/test_client.py::test_even_load_after_fast_functions PASSED [ 42%] distributed/tests/test_client.py::test_even_load_on_startup PASSED [ 42%] distributed/tests/test_client.py::test_contiguous_load SKIPPED [ 42%] distributed/tests/test_client.py::test_balanced_with_submit PASSED [ 42%] distributed/tests/test_client.py::test_balanced_with_submit_and_resident_data PASSED [ 42%] distributed/tests/test_client.py::test_scheduler_saturates_cores PASSED [ 43%] distributed/tests/test_client.py::test_scheduler_saturates_cores_random PASSED [ 43%] distributed/tests/test_client.py::test_cancel_clears_processing PASSED [ 43%] distributed/tests/test_client.py::test_default_get PASSED [ 43%] distributed/tests/test_client.py::test_ensure_default_client PASSED [ 43%] distributed/tests/test_client.py::test_set_as_default PASSED [ 43%] distributed/tests/test_client.py::test_get_foo PASSED [ 43%] distributed/tests/test_client.py::test_get_foo_lost_keys PASSED [ 43%] distributed/tests/test_client.py::test_get_processing_sync PASSED [ 43%] distributed/tests/test_client.py::test_close_idempotent PASSED [ 43%] distributed/tests/test_client.py::test_get_returns_early PASSED [ 43%] distributed/tests/test_client.py::test_Client_clears_references_after_restart PASSED [ 43%] distributed/tests/test_client.py::test_get_stops_work_after_error PASSED [ 43%] distributed/tests/test_client.py::test_as_completed_list PASSED [ 43%] distributed/tests/test_client.py::test_as_completed_results PASSED [ 43%] distributed/tests/test_client.py::test_as_completed_batches[True] PASSED [ 43%] distributed/tests/test_client.py::test_as_completed_batches[False] PASSED [ 43%] distributed/tests/test_client.py::test_as_completed_next_batch PASSED [ 43%] distributed/tests/test_client.py::test_status PASSED [ 43%] distributed/tests/test_client.py::test_persist_optimize_graph PASSED [ 43%] distributed/tests/test_client.py::test_scatter_raises_if_no_workers PASSED [ 43%] distributed/tests/test_client.py::test_reconnect_timeout PASSED [ 43%] distributed/tests/test_client.py::test_idempotence PASSED [ 43%] distributed/tests/test_client.py::test_scheduler_info PASSED [ 43%] distributed/tests/test_client.py::test_scheduler_info_async PASSED [ 43%] distributed/tests/test_client.py::test_write_scheduler_file PASSED [ 43%] distributed/tests/test_client.py::test_get_versions_sync PASSED [ 43%] distributed/tests/test_client.py::test_get_versions_async PASSED [ 43%] distributed/tests/test_client.py::test_get_versions_rpc_error PASSED [ 43%] distributed/tests/test_client.py::test_threaded_get_within_distributed PASSED [ 43%] distributed/tests/test_client.py::test_lose_scattered_data PASSED [ 43%] distributed/tests/test_client.py::test_partially_lose_scattered_data PASSED [ 43%] distributed/tests/test_client.py::test_scatter_compute_lose PASSED [ 43%] distributed/tests/test_client.py::test_scatter_compute_store_lose PASSED [ 43%] distributed/tests/test_client.py::test_scatter_compute_store_lose_processing PASSED [ 44%] distributed/tests/test_client.py::test_temp_default_client PASSED [ 44%] distributed/tests/test_client.py::test_as_current PASSED [ 44%] distributed/tests/test_client.py::test_as_current_is_thread_local PASSED [ 44%] distributed/tests/test_client.py::test_as_current_is_task_local PASSED [ 44%] distributed/tests/test_client.py::test_persist_workers_annotate PASSED [ 44%] distributed/tests/test_client.py::test_persist_workers_annotate2 PASSED [ 44%] distributed/tests/test_client.py::test_persist_workers PASSED [ 44%] distributed/tests/test_client.py::test_compute_workers_annotate PASSED [ 44%] distributed/tests/test_client.py::test_compute_workers PASSED [ 44%] distributed/tests/test_client.py::test_compute_nested_containers PASSED [ 44%] distributed/tests/test_client.py::test_scatter_type PASSED [ 44%] distributed/tests/test_client.py::test_retire_workers_2 PASSED [ 44%] distributed/tests/test_client.py::test_retire_many_workers PASSED [ 44%] distributed/tests/test_client.py::test_weight_occupancy_against_data_movement PASSED [ 44%] distributed/tests/test_client.py::test_distribute_tasks_by_nthreads PASSED [ 44%] distributed/tests/test_client.py::test_add_done_callback PASSED [ 44%] distributed/tests/test_client.py::test_normalize_collection PASSED [ 44%] distributed/tests/test_client.py::test_normalize_collection_dask_array PASSED [ 44%] distributed/tests/test_client.py::test_interleave_computations PASSED [ 44%] distributed/tests/test_client.py::test_interleave_computations_map SKIPPED [ 44%] distributed/tests/test_client.py::test_scatter_dict_workers PASSED [ 44%] distributed/tests/test_client.py::test_submit_list_kwargs PASSED [ 44%] distributed/tests/test_client.py::test_map_list_kwargs PASSED [ 44%] distributed/tests/test_client.py::test_recreate_error_sync PASSED [ 44%] distributed/tests/test_client.py::test_recreate_error_not_error PASSED [ 44%] distributed/tests/test_client.py::test_recreate_task_sync PASSED [ 44%] distributed/tests/test_client.py::test_retire_workers PASSED [ 44%] distributed/tests/test_client.py::test_restart_workers_no_nanny_raises PASSED [ 44%] distributed/tests/test_client.py::test_restart_workers_no_nanny_raises_many_workers PASSED [ 44%] distributed/tests/test_client.py::test_robust_unserializable PASSED [ 44%] distributed/tests/test_client.py::test_robust_undeserializable PASSED [ 44%] distributed/tests/test_client.py::test_robust_undeserializable_function PASSED [ 44%] distributed/tests/test_client.py::test_fire_and_forget PASSED [ 44%] distributed/tests/test_client.py::test_fire_and_forget_err PASSED [ 45%] distributed/tests/test_client.py::test_quiet_client_close FAILED [ 45%] distributed/tests/test_client.py::test_close PASSED [ 45%] distributed/tests/test_client.py::test_threadsafe PASSED [ 45%] distributed/tests/test_client.py::test_identity PASSED [ 45%] distributed/tests/test_client.py::test_get_client PASSED [ 45%] distributed/tests/test_client.py::test_get_client_no_cluster PASSED [ 45%] distributed/tests/test_client.py::test_serialize_collections PASSED [ 45%] distributed/tests/test_client.py::test_secede_simple PASSED [ 45%] distributed/tests/test_client.py::test_secede_balances PASSED [ 45%] distributed/tests/test_client.py::test_long_running_not_in_occupancy[True] PASSED [ 45%] distributed/tests/test_client.py::test_long_running_not_in_occupancy[False] PASSED [ 45%] distributed/tests/test_client.py::test_long_running_removal_clean[True] PASSED [ 45%] distributed/tests/test_client.py::test_long_running_removal_clean[False] PASSED [ 45%] distributed/tests/test_client.py::test_sub_submit_priority PASSED [ 45%] distributed/tests/test_client.py::test_get_client_sync PASSED [ 45%] distributed/tests/test_client.py::test_serialize_collections_of_futures PASSED [ 45%] distributed/tests/test_client.py::test_serialize_collections_of_futures_sync PASSED [ 45%] distributed/tests/test_client.py::test_dynamic_workloads_sync PASSED [ 45%] distributed/tests/test_client.py::test_unicode_ascii_keys PASSED [ 45%] distributed/tests/test_client.py::test_unicode_keys PASSED [ 45%] distributed/tests/test_client.py::test_use_synchronous_client_in_async_context PASSED [ 45%] distributed/tests/test_client.py::test_quiet_quit_when_cluster_leaves PASSED [ 45%] distributed/tests/test_client.py::test_call_stack PASSED [ 45%] distributed/tests/test_client.py::test_client_with_name PASSED [ 45%] distributed/tests/test_client.py::test_client_async_before_loop_starts PASSED [ 45%] distributed/tests/test_client.py::test_task_metadata PASSED [ 45%] distributed/tests/test_client.py::test_logs PASSED [ 45%] distributed/tests/test_client.py::test_logs_from_worker_submodules PASSED [ 45%] distributed/tests/test_client.py::test_avoid_delayed_finalize PASSED [ 45%] distributed/tests/test_client.py::test_config_scheduler_address PASSED [ 45%] distributed/tests/test_client.py::test_warn_when_submitting_large_values PASSED [ 45%] distributed/tests/test_client.py::test_warn_when_submitting_large_values_memoryview PASSED [ 45%] distributed/tests/test_client.py::test_closed_client_send_message PASSED [ 45%] distributed/tests/test_client.py::test_unhashable_function PASSED [ 46%] distributed/tests/test_client.py::test_client_name PASSED [ 46%] distributed/tests/test_client.py::test_client_doesnt_close_given_loop PASSED [ 46%] distributed/tests/test_client.py::test_dashboard_link PASSED [ 46%] distributed/tests/test_client.py::test_dashboard_link_inproc PASSED [ 46%] distributed/tests/test_client.py::test_client_timeout_2 PASSED [ 46%] distributed/tests/test_client.py::test_turn_off_pickle[True] PASSED [ 46%] distributed/tests/test_client.py::test_turn_off_pickle[False] PASSED [ 46%] distributed/tests/test_client.py::test_de_serialization PASSED [ 46%] distributed/tests/test_client.py::test_de_serialization_none PASSED [ 46%] distributed/tests/test_client.py::test_client_repr_closed PASSED [ 46%] distributed/tests/test_client.py::test_client_repr_closed_sync SKIPPED [ 46%] distributed/tests/test_client.py::test_nested_prioritization XPASS [ 46%] distributed/tests/test_client.py::test_scatter_error_cancel PASSED [ 46%] distributed/tests/test_client.py::test_scatter_and_replicate_avoid_paused_workers[False-False-False] PASSED [ 46%] distributed/tests/test_client.py::test_scatter_and_replicate_avoid_paused_workers[False-False-True] PASSED [ 46%] distributed/tests/test_client.py::test_scatter_and_replicate_avoid_paused_workers[False-True-False] PASSED [ 46%] distributed/tests/test_client.py::test_scatter_and_replicate_avoid_paused_workers[False-True-True] PASSED [ 46%] distributed/tests/test_client.py::test_scatter_and_replicate_avoid_paused_workers[True-False-False] PASSED [ 46%] distributed/tests/test_client.py::test_scatter_and_replicate_avoid_paused_workers[True-False-True] PASSED [ 46%] distributed/tests/test_client.py::test_scatter_and_replicate_avoid_paused_workers[True-True-False] PASSED [ 46%] distributed/tests/test_client.py::test_scatter_and_replicate_avoid_paused_workers[True-True-True] PASSED [ 46%] distributed/tests/test_client.py::test_scatter_and_replicate_avoid_paused_workers[10-False-False] PASSED [ 46%] distributed/tests/test_client.py::test_scatter_and_replicate_avoid_paused_workers[10-False-True] PASSED [ 46%] distributed/tests/test_client.py::test_scatter_and_replicate_avoid_paused_workers[10-True-False] PASSED [ 46%] distributed/tests/test_client.py::test_scatter_and_replicate_avoid_paused_workers[10-True-True] PASSED [ 46%] distributed/tests/test_client.py::test_no_threads_lingering XPASS [ 46%] distributed/tests/test_client.py::test_direct_async PASSED [ 46%] distributed/tests/test_client.py::test_direct_sync PASSED [ 46%] distributed/tests/test_client.py::test_mixing_clients_same_scheduler PASSED [ 46%] distributed/tests/test_client.py::test_mixing_clients_different_scheduler PASSED [ 46%] distributed/tests/test_client.py::test_tuple_keys PASSED [ 46%] distributed/tests/test_client.py::test_multiple_scatter PASSED [ 46%] distributed/tests/test_client.py::test_map_large_kwargs_in_graph PASSED [ 46%] distributed/tests/test_client.py::test_retry PASSED [ 47%] distributed/tests/test_client.py::test_retry_dependencies PASSED [ 47%] distributed/tests/test_client.py::test_released_dependencies PASSED [ 47%] distributed/tests/test_client.py::test_get_mix_futures_and_SubgraphCallable_dask_dataframe PASSED [ 47%] distributed/tests/test_client.py::test_direct_to_workers PASSED [ 47%] distributed/tests/test_client.py::test_instances PASSED [ 47%] distributed/tests/test_client.py::test_wait_for_workers PASSED [ 47%] distributed/tests/test_client.py::test_file_descriptors_dont_leak[Worker] PASSED [ 47%] distributed/tests/test_client.py::test_dashboard_link_cluster PASSED [ 47%] distributed/tests/test_client.py::test_shutdown PASSED [ 47%] distributed/tests/test_client.py::test_shutdown_localcluster PASSED [ 47%] distributed/tests/test_client.py::test_shutdown_stops_callbacks PASSED [ 47%] distributed/tests/test_client.py::test_shutdown_is_quiet_with_cluster PASSED [ 47%] distributed/tests/test_client.py::test_client_is_quiet_cluster_close PASSED [ 47%] distributed/tests/test_client.py::test_config_inherited_by_subprocess PASSED [ 47%] distributed/tests/test_client.py::test_futures_of_sorted PASSED [ 47%] distributed/tests/test_client.py::test_await_future PASSED [ 47%] distributed/tests/test_client.py::test_as_completed_async_for PASSED [ 47%] distributed/tests/test_client.py::test_as_completed_async_for_results PASSED [ 47%] distributed/tests/test_client.py::test_as_completed_async_for_cancel PASSED [ 47%] distributed/tests/test_client.py::test_async_with PASSED [ 47%] distributed/tests/test_client.py::test_client_sync_with_async_def PASSED [ 47%] distributed/tests/test_client.py::test_dont_hold_on_to_large_messages SKIPPED [ 47%] distributed/tests/test_client.py::test_run_on_scheduler_async_def PASSED [ 47%] distributed/tests/test_client.py::test_run_on_scheduler_async_def_wait PASSED [ 47%] distributed/tests/test_client.py::test_as_completed_condition_loop PASSED [ 47%] distributed/tests/test_client.py::test_get_task_metadata PASSED [ 47%] distributed/tests/test_client.py::test_get_task_metadata_multiple PASSED [ 47%] distributed/tests/test_client.py::test_register_worker_plugin_instance_required PASSED [ 47%] distributed/tests/test_client.py::test_register_worker_plugin_exception PASSED [ 47%] distributed/tests/test_client.py::test_annotations_task_state PASSED [ 47%] distributed/tests/test_client.py::test_annotations_compute_time[compute] PASSED [ 47%] distributed/tests/test_client.py::test_annotations_compute_time[persist] PASSED [ 47%] distributed/tests/test_client.py::test_annotations_survive_optimization XFAIL [ 47%] distributed/tests/test_client.py::test_annotations_priorities PASSED [ 48%] distributed/tests/test_client.py::test_annotations_workers PASSED [ 48%] distributed/tests/test_client.py::test_annotations_retries PASSED [ 48%] distributed/tests/test_client.py::test_annotations_blockwise_unpack PASSED [ 48%] distributed/tests/test_client.py::test_annotations_resources PASSED [ 48%] distributed/tests/test_client.py::test_annotations_resources_culled PASSED [ 48%] distributed/tests/test_client.py::test_annotations_loose_restrictions PASSED [ 48%] distributed/tests/test_client.py::test_annotations_submit_map PASSED [ 48%] distributed/tests/test_client.py::test_annotations_global_vs_local PASSED [ 48%] distributed/tests/test_client.py::test_workers_collection_restriction PASSED [ 48%] distributed/tests/test_client.py::test_get_client_functions_spawn_clusters PASSED [ 48%] distributed/tests/test_client.py::test_computation_code_walk_frames PASSED [ 48%] distributed/tests/test_client.py::test_computation_store_annotations PASSED [ 48%] distributed/tests/test_client.py::test_computation_object_code_dask_compute PASSED [ 48%] distributed/tests/test_client.py::test_computation_object_code_dask_compute_no_frames_default PASSED [ 48%] distributed/tests/test_client.py::test_computation_object_code_not_available SKIPPED [ 48%] distributed/tests/test_client.py::test_computation_object_code_dask_persist PASSED [ 48%] distributed/tests/test_client.py::test_computation_object_code_client_submit_simple PASSED [ 48%] distributed/tests/test_client.py::test_computation_object_code_client_submit_list_comp PASSED [ 48%] distributed/tests/test_client.py::test_computation_object_code_client_submit_dict_comp PASSED [ 48%] distributed/tests/test_client.py::test_computation_object_code_client_map PASSED [ 48%] distributed/tests/test_client.py::test_computation_object_code_client_compute PASSED [ 48%] distributed/tests/test_client.py::test_upload_directory PASSED [ 48%] distributed/tests/test_client.py::test_upload_directory_invalid_mode PASSED [ 48%] distributed/tests/test_client.py::test_upload_directory_to_scheduler[all] PASSED [ 48%] distributed/tests/test_client.py::test_upload_directory_to_scheduler[scheduler] PASSED [ 48%] distributed/tests/test_client.py::test_duck_typed_register_plugin_raises PASSED [ 48%] distributed/tests/test_client.py::test_exception_text PASSED [ 48%] distributed/tests/test_client.py::test_async_task PASSED [ 48%] distributed/tests/test_client.py::test_async_task_with_partial PASSED [ 48%] distributed/tests/test_client.py::test_warn_manual PASSED [ 48%] distributed/tests/test_client.py::test_warn_remote PASSED [ 48%] distributed/tests/test_client.py::test_print_remote PASSED [ 48%] distributed/tests/test_client.py::test_print_manual PASSED [ 48%] distributed/tests/test_client.py::test_print_manual_bad_args PASSED [ 49%] distributed/tests/test_client.py::test_print_non_msgpack_serializable PASSED [ 49%] distributed/tests/test_client.py::test_print_local PASSED [ 49%] distributed/tests/test_client.py::test_forward_logging PASSED [ 49%] distributed/tests/test_client.py::test_dump_cluster_state_write_from_scheduler PASSED [ 49%] distributed/tests/test_client.py::test_dump_cluster_state_sync[msgpack-True] PASSED [ 49%] distributed/tests/test_client.py::test_dump_cluster_state_sync[msgpack-False] PASSED [ 49%] distributed/tests/test_client.py::test_dump_cluster_state_sync[yaml-True] PASSED [ 49%] distributed/tests/test_client.py::test_dump_cluster_state_sync[yaml-False] PASSED [ 49%] distributed/tests/test_client.py::test_dump_cluster_state_async[msgpack-True] PASSED [ 49%] distributed/tests/test_client.py::test_dump_cluster_state_async[msgpack-False] PASSED [ 49%] distributed/tests/test_client.py::test_dump_cluster_state_async[yaml-True] PASSED [ 49%] distributed/tests/test_client.py::test_dump_cluster_state_async[yaml-False] PASSED [ 49%] distributed/tests/test_client.py::test_dump_cluster_state_json[True] PASSED [ 49%] distributed/tests/test_client.py::test_dump_cluster_state_json[False] PASSED [ 49%] distributed/tests/test_client.py::test_dump_cluster_state_exclude_default PASSED [ 49%] distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader PASSED [ 49%] distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_ignored_if_explicit_security_provided PASSED [ 49%] distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_ignored_if_returns_none PASSED [ 49%] distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_import_failed PASSED [ 49%] distributed/tests/test_client.py::test_benchmark_hardware_no_workers PASSED [ 49%] distributed/tests/test_client.py::test_wait_for_workers_updates_info PASSED [ 49%] distributed/tests/test_client.py::test_fast_close_on_aexit_failure PASSED [ 49%] distributed/tests/test_client.py::test_wait_for_workers_n_workers_value_check[None-ValueError] PASSED [ 49%] distributed/tests/test_client.py::test_wait_for_workers_n_workers_value_check[0-ValueError] PASSED [ 49%] distributed/tests/test_client.py::test_wait_for_workers_n_workers_value_check[1.0-ValueError] PASSED [ 49%] distributed/tests/test_client.py::test_wait_for_workers_n_workers_value_check[1-None] PASSED [ 49%] distributed/tests/test_client.py::test_wait_for_workers_n_workers_value_check[2-None] PASSED [ 49%] distributed/tests/test_client.py::test_unpacks_remotedata_namedtuple[PlainNamedTuple-args0-kwargs0] PASSED [ 49%] distributed/tests/test_client.py::test_unpacks_remotedata_namedtuple[NewArgsNamedTuple-args1-kwargs1] PASSED [ 49%] distributed/tests/test_client.py::test_unpacks_remotedata_namedtuple[NewArgsExNamedTuple-args2-kwargs2] PASSED [ 49%] distributed/tests/test_client.py::test_resolves_future_in_namedtuple[PlainNamedTuple-args0-kwargs0] PASSED [ 49%] distributed/tests/test_client.py::test_resolves_future_in_namedtuple[NewArgsNamedTuple-args1-kwargs1] PASSED [ 49%] distributed/tests/test_client.py::test_resolves_future_in_namedtuple[NewArgsExNamedTuple-args2-kwargs2] PASSED [ 49%] distributed/tests/test_client.py::test_resolves_future_in_dict PASSED [ 50%] distributed/tests/test_client.py::test_gather_race_vs_AMM[False] PASSED [ 50%] distributed/tests/test_client.py::test_gather_race_vs_AMM[True] PASSED [ 50%] distributed/tests/test_client.py::test_client_disconnect_exception_on_cancelled_futures PASSED [ 50%] distributed/tests/test_client.py::test_release_persisted_collection PASSED [ 50%] distributed/tests/test_client.py::test_release_persisted_collection_sync PASSED [ 50%] distributed/tests/test_client.py::test_adjust_heartbeat PASSED [ 50%] distributed/tests/test_client.py::test_count_serialization[True] PASSED [ 50%] distributed/tests/test_client.py::test_count_serialization[False] PASSED [ 50%] distributed/tests/test_client.py::test_map_accepts_nested_futures[False] PASSED [ 50%] distributed/tests/test_client.py::test_map_accepts_nested_futures[simple] PASSED [ 50%] distributed/tests/test_client.py::test_map_accepts_nested_futures[future] PASSED [ 50%] distributed/tests/test_client_executor.py::test_submit PASSED [ 50%] distributed/tests/test_client_executor.py::test_as_completed PASSED [ 50%] distributed/tests/test_client_executor.py::test_wait PASSED [ 50%] distributed/tests/test_client_executor.py::test_cancellation PASSED [ 50%] distributed/tests/test_client_executor.py::test_cancellation_wait PASSED [ 50%] distributed/tests/test_client_executor.py::test_cancellation_as_completed PASSED [ 50%] distributed/tests/test_client_executor.py::test_pure PASSED [ 50%] distributed/tests/test_client_executor.py::test_workers PASSED [ 50%] distributed/tests/test_client_executor.py::test_unsupported_arguments PASSED [ 50%] distributed/tests/test_client_executor.py::test_retries PASSED [ 50%] distributed/tests/test_client_executor.py::test_shutdown_wait PASSED [ 50%] distributed/tests/test_client_executor.py::test_shutdown_nowait PASSED [ 50%] distributed/tests/test_client_loop.py::test_close_loop_sync_start_new_loop PASSED [ 50%] distributed/tests/test_client_loop.py::test_close_loop_sync_use_running_loop PASSED [ 50%] distributed/tests/test_cluster_dump.py::test_tuple_to_list[input0-expected0] PASSED [ 50%] distributed/tests/test_cluster_dump.py::test_tuple_to_list[input1-expected1] PASSED [ 50%] distributed/tests/test_cluster_dump.py::test_tuple_to_list[input2-expected2] PASSED [ 50%] distributed/tests/test_cluster_dump.py::test_tuple_to_list[foo-foo] PASSED [ 50%] distributed/tests/test_cluster_dump.py::test_write_state_msgpack PASSED [ 50%] distributed/tests/test_cluster_dump.py::test_write_state_yaml PASSED [ 50%] distributed/tests/test_cluster_dump.py::test_cluster_dump_state PASSED [ 50%] distributed/tests/test_cluster_dump.py::test_cluster_dump_story PASSED [ 50%] distributed/tests/test_cluster_dump.py::test_cluster_dump_to_yamls PASSED [ 50%] distributed/tests/test_collections.py::test_lru PASSED [ 51%] distributed/tests/test_collections.py::test_heapset PASSED [ 51%] distributed/tests/test_collections.py::test_heapset_sorted_flag_left PASSED [ 51%] distributed/tests/test_collections.py::test_heapset_sorted_flag_right PASSED [ 51%] distributed/tests/test_collections.py::test_heapset_popright[False] PASSED [ 51%] distributed/tests/test_collections.py::test_heapset_popright[True] PASSED [ 51%] distributed/tests/test_collections.py::test_heapset_pickle PASSED [ 51%] distributed/tests/test_collections.py::test_heapset_sort_duplicate PASSED [ 51%] distributed/tests/test_collections.py::test_sum_mappings PASSED [ 51%] distributed/tests/test_compatibility.py::test_deprecated_periodic_callback PASSED [ 51%] distributed/tests/test_computations.py::test_computations PASSED [ 51%] distributed/tests/test_computations.py::test_computations_futures PASSED [ 51%] distributed/tests/test_computations.py::test_computations_no_workers PASSED [ 51%] distributed/tests/test_computations.py::test_computations_no_resources PASSED [ 51%] distributed/tests/test_computations.py::test_computations_long_running PASSED [ 51%] distributed/tests/test_config.py::test_logging_default[config0] PASSED [ 51%] distributed/tests/test_config.py::test_logging_default[None] PASSED [ 51%] distributed/tests/test_config.py::test_logging_simple_under_distributed PASSED [ 51%] distributed/tests/test_config.py::test_logging_simple PASSED [ 51%] distributed/tests/test_config.py::test_logging_extended PASSED [ 51%] distributed/tests/test_config.py::test_default_logging_does_not_override_basic_config PASSED [ 51%] distributed/tests/test_config.py::test_basic_config_does_not_override_default_logging PASSED [ 51%] distributed/tests/test_config.py::test_logging_mutual_exclusive PASSED [ 51%] distributed/tests/test_config.py::test_logging_file_config PASSED [ 51%] distributed/tests/test_config.py::test_schema PASSED [ 51%] distributed/tests/test_config.py::test_schema_is_complete PASSED [ 51%] distributed/tests/test_config.py::test_uvloop SKIPPED (could not import 'uvloop': No module named 'uvloop') [ 51%] distributed/tests/test_config.py::test_deprecations_on_set[args0-kwargs0] PASSED [ 51%] distributed/tests/test_config.py::test_deprecations_on_set[args1-kwargs1] PASSED [ 51%] distributed/tests/test_config.py::test_deprecations_on_set[args2-kwargs2] PASSED [ 51%] distributed/tests/test_config.py::test_deprecations_on_env_variables PASSED [ 51%] distributed/tests/test_config.py::test_deprecations_on_yaml[allowed-failures] PASSED [ 51%] distributed/tests/test_config.py::test_deprecations_on_yaml[allowed_failures] PASSED [ 51%] distributed/tests/test_core.py::test_server_status_is_always_enum PASSED [ 51%] distributed/tests/test_core.py::test_server_assign_assign_enum_is_quiet PASSED [ 52%] distributed/tests/test_core.py::test_server_status_compare_enum_is_quiet PASSED [ 52%] distributed/tests/test_core.py::test_server_close_stops_gil_monitoring SKIPPED [ 52%] distributed/tests/test_core.py::test_server PASSED [ 52%] distributed/tests/test_core.py::test_server_raises_on_blocked_handlers PASSED [ 52%] distributed/tests/test_core.py::test_rpc_default PASSED [ 52%] distributed/tests/test_core.py::test_rpc_tcp PASSED [ 52%] distributed/tests/test_core.py::test_rpc_tls PASSED [ 52%] distributed/tests/test_core.py::test_rpc_inproc PASSED [ 52%] distributed/tests/test_core.py::test_rpc_inputs PASSED [ 52%] distributed/tests/test_core.py::test_rpc_message_lifetime_default PASSED [ 52%] distributed/tests/test_core.py::test_rpc_message_lifetime_tcp PASSED [ 52%] distributed/tests/test_core.py::test_rpc_message_lifetime_inproc PASSED [ 52%] distributed/tests/test_core.py::test_rpc_with_many_connections_tcp PASSED [ 52%] distributed/tests/test_core.py::test_rpc_with_many_connections_inproc PASSED [ 52%] distributed/tests/test_core.py::test_large_packets_inproc PASSED [ 52%] distributed/tests/test_core.py::test_identity_tcp PASSED [ 52%] distributed/tests/test_core.py::test_identity_inproc PASSED [ 52%] distributed/tests/test_core.py::test_ports PASSED [ 52%] distributed/tests/test_core.py::test_errors PASSED [ 52%] distributed/tests/test_core.py::test_connect_raises PASSED [ 52%] distributed/tests/test_core.py::test_send_recv_args PASSED [ 52%] distributed/tests/test_core.py::test_send_recv_cancelled PASSED [ 52%] distributed/tests/test_core.py::test_coerce_to_address PASSED [ 52%] distributed/tests/test_core.py::test_connection_pool PASSED [ 52%] distributed/tests/test_core.py::test_connection_pool_close_while_connecting PASSED [ 52%] distributed/tests/test_core.py::test_connection_pool_outside_cancellation PASSED [ 52%] distributed/tests/test_core.py::test_remove_cancels_connect_attempts PASSED [ 52%] distributed/tests/test_core.py::test_remove_cancels_connect_before_task_running PASSED [ 52%] distributed/tests/test_core.py::test_connection_pool_respects_limit PASSED [ 52%] distributed/tests/test_core.py::test_connection_pool_tls PASSED [ 52%] distributed/tests/test_core.py::test_connection_pool_remove PASSED [ 52%] distributed/tests/test_core.py::test_counters PASSED [ 52%] distributed/tests/test_core.py::test_ticks SKIPPED (could not import 'crick': No module named 'crick') [ 52%] distributed/tests/test_core.py::test_tick_logging SKIPPED (could not import 'crick': No module named 'crick') [ 53%] distributed/tests/test_core.py::test_compression[echo_serialize-None] PASSED [ 53%] distributed/tests/test_core.py::test_compression[echo_serialize-False] PASSED [ 53%] distributed/tests/test_core.py::test_compression[echo_serialize-auto] PASSED [ 53%] distributed/tests/test_core.py::test_compression[echo_serialize-zlib] PASSED [ 53%] distributed/tests/test_core.py::test_compression[echo_serialize-snappy] PASSED [ 53%] distributed/tests/test_core.py::test_compression[echo_serialize-lz4] PASSED [ 53%] distributed/tests/test_core.py::test_compression[echo_serialize-zstd] PASSED [ 53%] distributed/tests/test_core.py::test_compression[echo_no_serialize-None] PASSED [ 53%] distributed/tests/test_core.py::test_compression[echo_no_serialize-False] PASSED [ 53%] distributed/tests/test_core.py::test_compression[echo_no_serialize-auto] PASSED [ 53%] distributed/tests/test_core.py::test_compression[echo_no_serialize-zlib] PASSED [ 53%] distributed/tests/test_core.py::test_compression[echo_no_serialize-snappy] PASSED [ 53%] distributed/tests/test_core.py::test_compression[echo_no_serialize-lz4] PASSED [ 53%] distributed/tests/test_core.py::test_compression[echo_no_serialize-zstd] PASSED [ 53%] distributed/tests/test_core.py::test_rpc_serialization PASSED [ 53%] distributed/tests/test_core.py::test_rpc_closed_exception PASSED [ 53%] distributed/tests/test_core.py::test_thread_id PASSED [ 53%] distributed/tests/test_core.py::test_deserialize_error PASSED [ 53%] distributed/tests/test_core.py::test_connection_pool_detects_remote_close PASSED [ 53%] distributed/tests/test_core.py::test_close_properly PASSED [ 53%] distributed/tests/test_core.py::test_server_redundant_kwarg PASSED [ 53%] distributed/tests/test_core.py::test_server_comms_mark_active_handlers PASSED [ 53%] distributed/tests/test_core.py::test_server_sys_path_local_directory_cleanup PASSED [ 53%] distributed/tests/test_core.py::test_close_fast_without_active_handlers[True] PASSED [ 53%] distributed/tests/test_core.py::test_close_fast_without_active_handlers[False] PASSED [ 53%] distributed/tests/test_core.py::test_close_grace_period_for_handlers PASSED [ 53%] distributed/tests/test_core.py::test_expects_comm PASSED [ 53%] distributed/tests/test_core.py::test_messages_are_ordered_bsend PASSED [ 53%] distributed/tests/test_core.py::test_messages_are_ordered_raw PASSED [ 53%] distributed/tests/test_counter.py::test_digest[Counter-] PASSED [ 53%] distributed/tests/test_counter.py::test_digest[None-] SKIPPED [ 53%] distributed/tests/test_counter.py::test_counter PASSED [ 53%] distributed/tests/test_dask_collections.py::test_persist PASSED [ 53%] distributed/tests/test_dask_collections.py::test_dataframes PASSED [ 54%] distributed/tests/test_dask_collections.py::test_dask_array_collections PASSED [ 54%] distributed/tests/test_dask_collections.py::test_bag_groupby_tasks_default PASSED [ 54%] distributed/tests/test_dask_collections.py::test_bag_groupby_key_hashing PASSED [ 54%] distributed/tests/test_dask_collections.py::test_dataframe_set_index_sync[wait] PASSED [ 54%] distributed/tests/test_dask_collections.py::test_dataframe_set_index_sync[] PASSED [ 54%] distributed/tests/test_dask_collections.py::test_loc_sync PASSED [ 54%] distributed/tests/test_dask_collections.py::test_rolling_sync PASSED [ 54%] distributed/tests/test_dask_collections.py::test_loc PASSED [ 54%] distributed/tests/test_dask_collections.py::test_dataframe_groupby_tasks PASSED [ 54%] distributed/tests/test_dask_collections.py::test_sparse_arrays SKIPPED [ 54%] distributed/tests/test_dask_collections.py::test_delayed_none PASSED [ 54%] distributed/tests/test_dask_collections.py::test_tuple_futures_arg[tuple] PASSED [ 54%] distributed/tests/test_dask_collections.py::test_tuple_futures_arg[list] PASSED [ 54%] distributed/tests/test_deadline.py::test_deadline PASSED [ 54%] distributed/tests/test_deadline.py::test_infinite_deadline PASSED [ 54%] distributed/tests/test_deadline.py::test_deadline_progress PASSED [ 54%] distributed/tests/test_deadline.py::test_deadline_expiration PASSED [ 54%] distributed/tests/test_deadline.py::test_deadline_expiration_async PASSED [ 54%] distributed/tests/test_diskutils.py::test_workdir_simple PASSED [ 54%] distributed/tests/test_diskutils.py::test_two_workspaces_in_same_directory PASSED [ 54%] distributed/tests/test_diskutils.py::test_workspace_process_crash PASSED [ 54%] distributed/tests/test_diskutils.py::test_workspace_rmtree_failure PASSED [ 54%] distributed/tests/test_diskutils.py::test_locking_disabled PASSED [ 54%] distributed/tests/test_diskutils.py::test_unwritable_base_dir PASSED [ 54%] distributed/tests/test_event_logging.py::test_log_event PASSED [ 54%] distributed/tests/test_event_logging.py::test_log_events PASSED [ 54%] distributed/tests/test_event_logging.py::test_log_event_e2e PASSED [ 54%] distributed/tests/test_event_logging.py::test_log_event_multiple_clients PASSED [ 54%] distributed/tests/test_event_logging.py::test_events_subscribe_topic PASSED [ 54%] distributed/tests/test_event_logging.py::test_events_subscribe_topic_cancelled PASSED [ 54%] distributed/tests/test_event_logging.py::test_topic_subscribe_unsubscribe PASSED [ 54%] distributed/tests/test_event_logging.py::test_events_all_servers_use_same_channel PASSED [ 54%] distributed/tests/test_event_logging.py::test_events_unsubscribe_raises_if_unknown PASSED [ 54%] distributed/tests/test_event_logging.py::test_log_event_msgpack PASSED [ 55%] distributed/tests/test_event_logging.py::test_configurable_events_log_length PASSED [ 55%] distributed/tests/test_event_logging.py::test_log_event_on_nanny PASSED [ 55%] distributed/tests/test_event_logging.py::test_log_event_on_worker PASSED [ 55%] distributed/tests/test_event_logging.py::test_log_event_plugin PASSED [ 55%] distributed/tests/test_event_logging.py::test_log_event_plugin_multiple_topics PASSED [ 55%] distributed/tests/test_events.py::test_event_on_workers PASSED [ 55%] distributed/tests/test_events.py::test_default_event PASSED [ 55%] distributed/tests/test_events.py::test_set_not_set PASSED [ 55%] distributed/tests/test_events.py::test_set_not_set_many_events PASSED [ 55%] distributed/tests/test_events.py::test_timeout PASSED [ 55%] distributed/tests/test_events.py::test_event_sync PASSED [ 55%] distributed/tests/test_events.py::test_event_types PASSED [ 55%] distributed/tests/test_events.py::test_serializable PASSED [ 55%] distributed/tests/test_events.py::test_two_events_on_workers PASSED [ 55%] distributed/tests/test_events.py::test_unpickle_without_client PASSED [ 55%] distributed/tests/test_failed_workers.py::test_submit_after_failed_worker_async[False-False-closing] PASSED [ 55%] distributed/tests/test_failed_workers.py::test_submit_after_failed_worker_async[False-False-closed] PASSED [ 55%] distributed/tests/test_failed_workers.py::test_submit_after_failed_worker_async[False-True-closing] PASSED [ 55%] distributed/tests/test_failed_workers.py::test_submit_after_failed_worker_async[False-True-closed] PASSED [ 55%] distributed/tests/test_failed_workers.py::test_submit_after_failed_worker_async[True-False-closing] PASSED [ 55%] distributed/tests/test_failed_workers.py::test_submit_after_failed_worker_async[True-False-closed] PASSED [ 55%] distributed/tests/test_failed_workers.py::test_submit_after_failed_worker_async[True-True-closing] PASSED [ 55%] distributed/tests/test_failed_workers.py::test_submit_after_failed_worker_async[True-True-closed] PASSED [ 55%] distributed/tests/test_failed_workers.py::test_submit_after_failed_worker PASSED [ 55%] distributed/tests/test_failed_workers.py::test_restart FAILED [ 55%] distributed/tests/test_failed_workers.py::test_restart_cleared FAILED [ 55%] distributed/tests/test_failed_workers.py::test_restart_sync PASSED [ 55%] distributed/tests/test_failed_workers.py::test_worker_doesnt_await_task_completion PASSED [ 55%] distributed/tests/test_failed_workers.py::test_multiple_clients_restart FAILED [ 55%] distributed/tests/test_failed_workers.py::test_restart_scheduler FAILED [ 55%] distributed/tests/test_failed_workers.py::test_forgotten_futures_dont_clean_up_new_futures FAILED [ 55%] distributed/tests/test_failed_workers.py::test_restart_during_computation PASSED [ 55%] distributed/tests/test_failed_workers.py::test_worker_same_host_replicas_missing PASSED [ 55%] distributed/tests/test_failed_workers.py::test_forget_data_not_supposed_to_have PASSED [ 56%] distributed/tests/test_failed_workers.py::test_failing_worker_with_additional_replicas_on_cluster PASSED [ 56%] distributed/tests/test_gc.py::test_fractional_timer PASSED [ 56%] distributed/tests/test_gc.py::test_gc_diagnosis_cpu_time PASSED [ 56%] distributed/tests/test_gc.py::test_gc_diagnosis_rss_win XFAIL (flaky and re-fails on rerun) [ 56%] distributed/tests/test_imports.py::test_can_import_distributed_in_background_thread PASSED [ 56%] distributed/tests/test_init.py::test_version PASSED [ 56%] distributed/tests/test_itertools.py::test_ffill PASSED [ 56%] distributed/tests/test_locks.py::test_lock PASSED [ 56%] distributed/tests/test_locks.py::test_timeout PASSED [ 56%] distributed/tests/test_locks.py::test_acquires_with_zero_timeout PASSED [ 56%] distributed/tests/test_locks.py::test_acquires_blocking PASSED [ 56%] distributed/tests/test_locks.py::test_timeout_sync PASSED [ 56%] distributed/tests/test_locks.py::test_errors PASSED [ 56%] distributed/tests/test_locks.py::test_lock_sync PASSED [ 56%] distributed/tests/test_locks.py::test_lock_types PASSED [ 56%] distributed/tests/test_locks.py::test_serializable PASSED [ 56%] distributed/tests/test_locks.py::test_serializable_no_ctx PASSED [ 56%] distributed/tests/test_locks.py::test_locks PASSED [ 56%] distributed/tests/test_locks.py::test_locks_inf_lease_timeout PASSED [ 56%] distributed/tests/test_metrics.py::test_wall_clock[time] PASSED [ 56%] distributed/tests/test_metrics.py::test_wall_clock[monotonic] PASSED [ 56%] distributed/tests/test_metrics.py::test_meter PASSED [ 56%] distributed/tests/test_metrics.py::test_meter_raise PASSED [ 56%] distributed/tests/test_metrics.py::test_meter_floor[kwargs0-0] PASSED [ 56%] distributed/tests/test_metrics.py::test_meter_floor[kwargs1-0.1] PASSED [ 56%] distributed/tests/test_metrics.py::test_meter_floor[kwargs2--1] PASSED [ 56%] distributed/tests/test_metrics.py::test_context_meter PASSED [ 56%] distributed/tests/test_metrics.py::test_context_meter_raise PASSED [ 56%] distributed/tests/test_metrics.py::test_context_meter_nested PASSED [ 56%] distributed/tests/test_metrics.py::test_context_meter_decorator PASSED [ 56%] distributed/tests/test_metrics.py::test_context_meter_nested_floor PASSED [ 56%] distributed/tests/test_metrics.py::test_context_meter_pickle PASSED [ 56%] distributed/tests/test_metrics.py::test_context_meter_clear_callbacks PASSED [ 56%] distributed/tests/test_metrics.py::test_context_meter_clear_callbacks_raises PASSED [ 57%] distributed/tests/test_metrics.py::test_context_meter_allow_offload PASSED [ 57%] distributed/tests/test_metrics.py::test_delayed_metrics_ledger PASSED [ 57%] distributed/tests/test_metrics.py::test_context_meter_keyed PASSED [ 57%] distributed/tests/test_metrics.py::test_delayed_metrics_ledger_keyed PASSED [ 57%] distributed/tests/test_multi_locks.py::test_single_lock PASSED [ 57%] distributed/tests/test_multi_locks.py::test_timeout PASSED [ 57%] distributed/tests/test_multi_locks.py::test_timeout_wake_waiter PASSED [ 57%] distributed/tests/test_multi_locks.py::test_multiple_locks PASSED [ 57%] distributed/tests/test_multi_locks.py::test_num_locks PASSED [ 57%] distributed/tests/test_preload.py::test_worker_preload_file PASSED [ 57%] distributed/tests/test_preload.py::test_worker_preload_text PASSED [ 57%] distributed/tests/test_preload.py::test_preload_manager_sequence PASSED [ 57%] distributed/tests/test_preload.py::test_worker_preload_config PASSED [ 57%] distributed/tests/test_preload.py::test_worker_preload_module PASSED [ 57%] distributed/tests/test_preload.py::test_worker_preload_click PASSED [ 57%] distributed/tests/test_preload.py::test_worker_preload_click_async PASSED [ 57%] distributed/tests/test_preload.py::test_preload_import_time PASSED [ 57%] distributed/tests/test_preload.py::test_web_preload PASSED [ 57%] distributed/tests/test_preload.py::test_scheduler_startup PASSED [ 57%] distributed/tests/test_preload.py::test_scheduler_startup_nanny PASSED [ 57%] distributed/tests/test_preload.py::test_web_preload_worker PASSED [ 57%] distributed/tests/test_preload.py::test_client_preload_text XFAIL (The preload argument to the client isn't supported yet) [ 57%] distributed/tests/test_preload.py::test_client_preload_config PASSED [ 57%] distributed/tests/test_preload.py::test_client_preload_click XFAIL (The preload argument to the client isn't supported yet) [ 57%] distributed/tests/test_preload.py::test_failure_doesnt_crash_scheduler PASSED [ 57%] distributed/tests/test_preload.py::test_failure_doesnt_crash_worker PASSED [ 57%] distributed/tests/test_preload.py::test_failure_doesnt_crash_nanny PASSED [ 57%] distributed/tests/test_preload.py::test_failure_doesnt_crash_client PASSED [ 57%] distributed/tests/test_preload.py::test_client_preload_config_click PASSED [ 57%] distributed/tests/test_priorities.py::test_submit[queue on worker] PASSED [ 57%] distributed/tests/test_priorities.py::test_submit[queue on scheduler] PASSED [ 57%] distributed/tests/test_priorities.py::test_map[queue on worker] PASSED [ 57%] distributed/tests/test_priorities.py::test_map[queue on scheduler] PASSED [ 57%] distributed/tests/test_priorities.py::test_compute[queue on worker] PASSED [ 58%] distributed/tests/test_priorities.py::test_compute[queue on scheduler] PASSED [ 58%] distributed/tests/test_priorities.py::test_persist[queue on worker] PASSED [ 58%] distributed/tests/test_priorities.py::test_persist[queue on scheduler] PASSED [ 58%] distributed/tests/test_priorities.py::test_annotate_compute[queue on worker] PASSED [ 58%] distributed/tests/test_priorities.py::test_annotate_compute[queue on scheduler] PASSED [ 58%] distributed/tests/test_priorities.py::test_annotate_persist[queue on worker] PASSED [ 58%] distributed/tests/test_priorities.py::test_annotate_persist[queue on scheduler] PASSED [ 58%] distributed/tests/test_priorities.py::test_repeated_persists_same_priority[queue on worker] PASSED [ 58%] distributed/tests/test_priorities.py::test_repeated_persists_same_priority[queue on scheduler] PASSED [ 58%] distributed/tests/test_priorities.py::test_last_in_first_out[queue on worker] PASSED [ 58%] distributed/tests/test_priorities.py::test_last_in_first_out[queue on scheduler] PASSED [ 58%] distributed/tests/test_profile.py::test_basic PASSED [ 58%] distributed/tests/test_profile.py::test_basic_low_level SKIPPED (could not import 'stacktrace': No module named 'stacktrace') [ 58%] distributed/tests/test_profile.py::test_merge PASSED [ 58%] distributed/tests/test_profile.py::test_merge_empty PASSED [ 58%] distributed/tests/test_profile.py::test_call_stack PASSED [ 58%] distributed/tests/test_profile.py::test_identifier PASSED [ 58%] distributed/tests/test_profile.py::test_watch PASSED [ 58%] distributed/tests/test_profile.py::test_watch_requires_lock_to_run PASSED [ 58%] distributed/tests/test_profile.py::test_info_frame_f_lineno[-1-1] PASSED [ 58%] distributed/tests/test_profile.py::test_info_frame_f_lineno[0-2] PASSED [ 58%] distributed/tests/test_profile.py::test_info_frame_f_lineno[1-2] PASSED [ 58%] distributed/tests/test_profile.py::test_info_frame_f_lineno[11-2] PASSED [ 58%] distributed/tests/test_profile.py::test_info_frame_f_lineno[12-3] PASSED [ 58%] distributed/tests/test_profile.py::test_info_frame_f_lineno[21-4] PASSED [ 58%] distributed/tests/test_profile.py::test_info_frame_f_lineno[22-4] PASSED [ 58%] distributed/tests/test_profile.py::test_info_frame_f_lineno[23-4] PASSED [ 58%] distributed/tests/test_profile.py::test_info_frame_f_lineno[24-2] PASSED [ 58%] distributed/tests/test_profile.py::test_info_frame_f_lineno[25-2] PASSED [ 58%] distributed/tests/test_profile.py::test_info_frame_f_lineno[26-2] PASSED [ 58%] distributed/tests/test_profile.py::test_info_frame_f_lineno[27-2] PASSED [ 58%] distributed/tests/test_profile.py::test_info_frame_f_lineno[100-2] PASSED [ 58%] distributed/tests/test_profile.py::test_call_stack_f_lineno[-1-1] PASSED [ 58%] distributed/tests/test_profile.py::test_call_stack_f_lineno[0-2] PASSED [ 59%] distributed/tests/test_profile.py::test_call_stack_f_lineno[1-2] PASSED [ 59%] distributed/tests/test_profile.py::test_call_stack_f_lineno[11-2] PASSED [ 59%] distributed/tests/test_profile.py::test_call_stack_f_lineno[12-3] PASSED [ 59%] distributed/tests/test_profile.py::test_call_stack_f_lineno[21-4] PASSED [ 59%] distributed/tests/test_profile.py::test_call_stack_f_lineno[22-4] PASSED [ 59%] distributed/tests/test_profile.py::test_call_stack_f_lineno[23-4] PASSED [ 59%] distributed/tests/test_profile.py::test_call_stack_f_lineno[24-2] PASSED [ 59%] distributed/tests/test_profile.py::test_call_stack_f_lineno[25-2] PASSED [ 59%] distributed/tests/test_profile.py::test_call_stack_f_lineno[26-2] PASSED [ 59%] distributed/tests/test_profile.py::test_call_stack_f_lineno[27-2] PASSED [ 59%] distributed/tests/test_profile.py::test_call_stack_f_lineno[100-2] PASSED [ 59%] distributed/tests/test_profile.py::test_stack_overflow PASSED [ 59%] distributed/tests/test_profile.py::test_builtin PASSED [ 59%] distributed/tests/test_publish.py::test_publish_simple PASSED [ 59%] distributed/tests/test_publish.py::test_publish_non_string_names[name0] PASSED [ 59%] distributed/tests/test_publish.py::test_publish_non_string_names[9.0] PASSED [ 59%] distributed/tests/test_publish.py::test_publish_non_string_names[8] PASSED [ 59%] distributed/tests/test_publish.py::test_publish_multiple_names PASSED [ 59%] distributed/tests/test_publish.py::test_publish_roundtrip PASSED [ 59%] distributed/tests/test_publish.py::test_unpublish PASSED [ 59%] distributed/tests/test_publish.py::test_unpublish_sync PASSED [ 59%] distributed/tests/test_publish.py::test_unpublish_nonexistent PASSED [ 59%] distributed/tests/test_publish.py::test_publish_multiple_datasets PASSED [ 59%] distributed/tests/test_publish.py::test_publish_multiple_collections_one_name PASSED [ 59%] distributed/tests/test_publish.py::test_unpublish_multiple_datasets_sync PASSED [ 59%] distributed/tests/test_publish.py::test_publish_bag PASSED [ 59%] distributed/tests/test_publish.py::test_datasets_setitem PASSED [ 59%] distributed/tests/test_publish.py::test_datasets_getitem PASSED [ 59%] distributed/tests/test_publish.py::test_datasets_getitem_default PASSED [ 59%] distributed/tests/test_publish.py::test_datasets_delitem PASSED [ 59%] distributed/tests/test_publish.py::test_datasets_keys PASSED [ 59%] distributed/tests/test_publish.py::test_datasets_contains PASSED [ 59%] distributed/tests/test_publish.py::test_publish_dataset_override PASSED [ 59%] distributed/tests/test_publish.py::test_publish_dataset_override_releases_old_keys PASSED [ 60%] distributed/tests/test_publish.py::test_publish_dataset_override_partial PASSED [ 60%] distributed/tests/test_publish.py::test_datasets_iter PASSED [ 60%] distributed/tests/test_publish.py::test_datasets_async PASSED [ 60%] distributed/tests/test_publish.py::test_pickle_safe PASSED [ 60%] distributed/tests/test_publish.py::test_deserialize_client PASSED [ 60%] distributed/tests/test_publish.py::test_publish_unpublish_wait_for_batched_comms PASSED [ 60%] distributed/tests/test_publish.py::test_publish_bad_syntax PASSED [ 60%] distributed/tests/test_publish.py::test_publish_unpersisted PASSED [ 60%] distributed/tests/test_queues.py::test_queue PASSED [ 60%] distributed/tests/test_queues.py::test_queue_with_data PASSED [ 60%] distributed/tests/test_queues.py::test_sync PASSED [ 60%] distributed/tests/test_queues.py::test_hold_futures PASSED [ 60%] distributed/tests/test_queues.py::test_picklability PASSED [ 60%] distributed/tests/test_queues.py::test_picklability_sync PASSED [ 60%] distributed/tests/test_queues.py::test_same_futures PASSED [ 60%] distributed/tests/test_queues.py::test_get_many PASSED [ 60%] distributed/tests/test_queues.py::test_Future_knows_status_immediately PASSED [ 60%] distributed/tests/test_queues.py::test_erred_future PASSED [ 60%] distributed/tests/test_queues.py::test_close PASSED [ 60%] distributed/tests/test_queues.py::test_timeout PASSED [ 60%] distributed/tests/test_queues.py::test_2220 PASSED [ 60%] distributed/tests/test_queues.py::test_queue_in_task PASSED [ 60%] distributed/tests/test_queues.py::test_unpickle_without_client PASSED [ 60%] distributed/tests/test_queues.py::test_set_cancelled_future PASSED [ 60%] distributed/tests/test_reschedule.py::test_scheduler_reschedule_warns PASSED [ 60%] distributed/tests/test_reschedule.py::test_raise_reschedule[executing] PASSED [ 60%] distributed/tests/test_reschedule.py::test_raise_reschedule[long-running] PASSED [ 60%] distributed/tests/test_reschedule.py::test_cancelled_reschedule[executing] PASSED [ 60%] distributed/tests/test_reschedule.py::test_cancelled_reschedule[long-running] PASSED [ 60%] distributed/tests/test_reschedule.py::test_cancelled_reschedule_worker_state[executing] PASSED [ 60%] distributed/tests/test_reschedule.py::test_cancelled_reschedule_worker_state[long-running] PASSED [ 60%] distributed/tests/test_reschedule.py::test_reschedule_releases[executing] PASSED [ 60%] distributed/tests/test_reschedule.py::test_reschedule_releases[long-running] PASSED [ 60%] distributed/tests/test_reschedule.py::test_reschedule_cancelled[executing] PASSED [ 61%] distributed/tests/test_reschedule.py::test_reschedule_cancelled[long-running] PASSED [ 61%] distributed/tests/test_reschedule.py::test_reschedule_resumed[executing] PASSED [ 61%] distributed/tests/test_reschedule.py::test_reschedule_resumed[long-running] PASSED [ 61%] distributed/tests/test_resources.py::test_resource_submit PASSED [ 61%] distributed/tests/test_resources.py::test_submit_many_non_overlapping PASSED [ 61%] distributed/tests/test_resources.py::test_submit_many_non_overlapping_2 PASSED [ 61%] distributed/tests/test_resources.py::test_move PASSED [ 61%] distributed/tests/test_resources.py::test_dont_work_steal PASSED [ 61%] distributed/tests/test_resources.py::test_map PASSED [ 61%] distributed/tests/test_resources.py::test_persist PASSED [ 61%] distributed/tests/test_resources.py::test_compute PASSED [ 61%] distributed/tests/test_resources.py::test_get PASSED [ 61%] distributed/tests/test_resources.py::test_persist_multiple_collections PASSED [ 61%] distributed/tests/test_resources.py::test_resources_str PASSED [ 61%] distributed/tests/test_resources.py::test_minimum_resource PASSED [ 61%] distributed/tests/test_resources.py::test_constrained_vs_ready_priority_1[1-0-y-False] PASSED [ 61%] distributed/tests/test_resources.py::test_constrained_vs_ready_priority_1[1-0-y-True] PASSED [ 61%] distributed/tests/test_resources.py::test_constrained_vs_ready_priority_1[0-1-x-False] PASSED [ 61%] distributed/tests/test_resources.py::test_constrained_vs_ready_priority_1[0-1-x-True] PASSED [ 61%] distributed/tests/test_resources.py::test_constrained_vs_ready_priority_2[1-0-y-False] PASSED [ 61%] distributed/tests/test_resources.py::test_constrained_vs_ready_priority_2[1-0-y-True] PASSED [ 61%] distributed/tests/test_resources.py::test_constrained_vs_ready_priority_2[0-1-x-False] PASSED [ 61%] distributed/tests/test_resources.py::test_constrained_vs_ready_priority_2[0-1-x-True] PASSED [ 61%] distributed/tests/test_resources.py::test_constrained_tasks_respect_priority PASSED [ 61%] distributed/tests/test_resources.py::test_task_cancelled_and_readded_with_resources PASSED [ 61%] distributed/tests/test_resources.py::test_balance_resources SKIPPED [ 61%] distributed/tests/test_resources.py::test_set_resources PASSED [ 61%] distributed/tests/test_resources.py::test_persist_collections PASSED [ 61%] distributed/tests/test_resources.py::test_dont_optimize_out SKIPPED [ 61%] distributed/tests/test_resources.py::test_full_collections SKIPPED (atop fusion seemed to break this) [ 61%] distributed/tests/test_resources.py::test_resources_from_config PASSED [ 61%] distributed/tests/test_resources.py::test_resources_from_python_override_config PASSED [ 61%] distributed/tests/test_resources.py::test_cancelled_with_resources[executing-ExecuteSuccessEvent] PASSED [ 61%] distributed/tests/test_resources.py::test_cancelled_with_resources[executing-ExecuteFailureEvent] PASSED [ 62%] distributed/tests/test_resources.py::test_cancelled_with_resources[executing-RescheduleEvent] PASSED [ 62%] distributed/tests/test_resources.py::test_cancelled_with_resources[long-running-ExecuteSuccessEvent] PASSED [ 62%] distributed/tests/test_resources.py::test_cancelled_with_resources[long-running-ExecuteFailureEvent] PASSED [ 62%] distributed/tests/test_resources.py::test_cancelled_with_resources[long-running-RescheduleEvent] PASSED [ 62%] distributed/tests/test_resources.py::test_resumed_with_resources[executing-ExecuteSuccessEvent] PASSED [ 62%] distributed/tests/test_resources.py::test_resumed_with_resources[executing-ExecuteFailureEvent] PASSED [ 62%] distributed/tests/test_resources.py::test_resumed_with_resources[executing-RescheduleEvent] PASSED [ 62%] distributed/tests/test_resources.py::test_resumed_with_resources[long-running-ExecuteSuccessEvent] PASSED [ 62%] distributed/tests/test_resources.py::test_resumed_with_resources[long-running-ExecuteFailureEvent] PASSED [ 62%] distributed/tests/test_resources.py::test_resumed_with_resources[long-running-RescheduleEvent] PASSED [ 62%] distributed/tests/test_resources.py::test_resumed_with_different_resources[executing-ExecuteSuccessEvent] PASSED [ 62%] distributed/tests/test_resources.py::test_resumed_with_different_resources[executing-ExecuteFailureEvent] PASSED [ 62%] distributed/tests/test_resources.py::test_resumed_with_different_resources[executing-RescheduleEvent] PASSED [ 62%] distributed/tests/test_resources.py::test_resumed_with_different_resources[long-running-ExecuteSuccessEvent] PASSED [ 62%] distributed/tests/test_resources.py::test_resumed_with_different_resources[long-running-ExecuteFailureEvent] PASSED [ 62%] distributed/tests/test_resources.py::test_resumed_with_different_resources[long-running-RescheduleEvent] PASSED [ 62%] distributed/tests/test_scheduler.py::test_administration PASSED [ 62%] distributed/tests/test_scheduler.py::test_respect_data_in_memory PASSED [ 62%] distributed/tests/test_scheduler.py::test_recompute_released_results PASSED [ 62%] distributed/tests/test_scheduler.py::test_decide_worker_with_many_independent_leaves PASSED [ 62%] distributed/tests/test_scheduler.py::test_decide_worker_with_restrictions PASSED [ 62%] distributed/tests/test_scheduler.py::test_decide_worker_coschedule_order_neighbors[nthreads0-0] PASSED [ 62%] distributed/tests/test_scheduler.py::test_decide_worker_coschedule_order_neighbors[nthreads0-1] PASSED [ 62%] distributed/tests/test_scheduler.py::test_decide_worker_coschedule_order_neighbors[nthreads0-4] PASSED [ 62%] distributed/tests/test_scheduler.py::test_decide_worker_coschedule_order_neighbors[nthreads1-0] PASSED [ 62%] distributed/tests/test_scheduler.py::test_decide_worker_coschedule_order_neighbors[nthreads1-1] PASSED [ 62%] distributed/tests/test_scheduler.py::test_decide_worker_coschedule_order_neighbors[nthreads1-4] PASSED [ 62%] distributed/tests/test_scheduler.py::test_decide_worker_rootish_while_last_worker_is_retiring SKIPPEDissues/7204) [ 62%] distributed/tests/test_scheduler.py::test_graph_execution_width PASSED [ 62%] distributed/tests/test_scheduler.py::test_forget_tasks_while_processing PASSED [ 62%] distributed/tests/test_scheduler.py::test_queued_release_multiple_workers PASSED [ 62%] distributed/tests/test_scheduler.py::test_queued_paused_new_worker PASSED [ 62%] distributed/tests/test_scheduler.py::test_queued_paused_unpaused[True] PASSED [ 62%] distributed/tests/test_scheduler.py::test_queued_paused_unpaused[False] PASSED [ 63%] distributed/tests/test_scheduler.py::test_queued_remove_add_worker PASSED [ 63%] distributed/tests/test_scheduler.py::test_secede_opens_slot PASSED [ 63%] distributed/tests/test_scheduler.py::test_saturation_factor[2.5-expected_task_counts0] PASSED [ 63%] distributed/tests/test_scheduler.py::test_saturation_factor[2.0-expected_task_counts1] PASSED [ 63%] distributed/tests/test_scheduler.py::test_saturation_factor[1.1-expected_task_counts2] PASSED [ 63%] distributed/tests/test_scheduler.py::test_saturation_factor[1.0-expected_task_counts3] PASSED [ 63%] distributed/tests/test_scheduler.py::test_saturation_factor[0.1-expected_task_counts4] PASSED [ 63%] distributed/tests/test_scheduler.py::test_saturation_factor[inf-expected_task_counts5] PASSED [ 63%] distributed/tests/test_scheduler.py::test_saturation_factor[inf-expected_task_counts6] PASSED [ 63%] distributed/tests/test_scheduler.py::test_bad_saturation_factor PASSED [ 63%] distributed/tests/test_scheduler.py::test_move_data_over_break_restrictions PASSED [ 63%] distributed/tests/test_scheduler.py::test_balance_with_restrictions PASSED [ 63%] distributed/tests/test_scheduler.py::test_no_valid_workers PASSED [ 63%] distributed/tests/test_scheduler.py::test_no_valid_workers_loose_restrictions PASSED [ 63%] distributed/tests/test_scheduler.py::test_no_workers[False] PASSED [ 63%] distributed/tests/test_scheduler.py::test_no_workers[True] PASSED [ 63%] distributed/tests/test_scheduler.py::test_retire_workers_empty PASSED [ 63%] distributed/tests/test_scheduler.py::test_retire_workers_lock PASSED [ 63%] distributed/tests/test_scheduler.py::test_retire_workers_concurrently PASSED [ 63%] distributed/tests/test_scheduler.py::test_server_listens_to_other_ops PASSED [ 63%] distributed/tests/test_scheduler.py::test_remove_worker_from_scheduler PASSED [ 63%] distributed/tests/test_scheduler.py::test_remove_worker_by_name_from_scheduler PASSED [ 63%] distributed/tests/test_scheduler.py::test_clear_events_worker_removal PASSED [ 63%] distributed/tests/test_scheduler.py::test_clear_events_client_removal PASSED [ 63%] distributed/tests/test_scheduler.py::test_add_worker PASSED [ 63%] distributed/tests/test_scheduler.py::test_blocked_handlers_are_respected PASSED [ 63%] distributed/tests/test_scheduler.py::test_scheduler_init_pulls_blocked_handlers_from_config PASSED [ 63%] distributed/tests/test_scheduler.py::test_feed PASSED [ 63%] distributed/tests/test_scheduler.py::test_feed_setup_teardown PASSED [ 63%] distributed/tests/test_scheduler.py::test_feed_large_bytestring PASSED [ 63%] distributed/tests/test_scheduler.py::test_delete_data PASSED [ 63%] distributed/tests/test_scheduler.py::test_delete PASSED [ 63%] distributed/tests/test_scheduler.py::test_dumps_function PASSED [ 63%] distributed/tests/test_scheduler.py::test_ready_remove_worker[1.0] PASSED [ 64%] distributed/tests/test_scheduler.py::test_ready_remove_worker[inf] PASSED [ 64%] distributed/tests/test_scheduler.py::test_restart_not_all_workers_return PASSED [ 64%] distributed/tests/test_scheduler.py::test_restart_worker_rejoins_after_timeout_expired PASSED [ 64%] distributed/tests/test_scheduler.py::test_restart_no_wait_for_workers PASSED [ 64%] distributed/tests/test_scheduler.py::test_broadcast PASSED [ 64%] distributed/tests/test_scheduler.py::test_broadcast_tls PASSED [ 64%] distributed/tests/test_scheduler.py::test_broadcast_nanny PASSED [ 64%] distributed/tests/test_scheduler.py::test_broadcast_on_error PASSED [ 64%] distributed/tests/test_scheduler.py::test_broadcast_deprecation PASSED [ 64%] distributed/tests/test_scheduler.py::test_broadcast_reuse_comm[True] PASSED [ 64%] distributed/tests/test_scheduler.py::test_broadcast_reuse_comm[False] PASSED [ 64%] distributed/tests/test_scheduler.py::test_worker_name PASSED [ 64%] distributed/tests/test_scheduler.py::test_coerce_address PASSED [ 64%] distributed/tests/test_scheduler.py::test_config_stealing PASSED [ 64%] distributed/tests/test_scheduler.py::test_config_no_stealing PASSED [ 64%] distributed/tests/test_scheduler.py::test_file_descriptors_dont_leak PASSED [ 64%] distributed/tests/test_scheduler.py::test_update_graph_culls PASSED [ 64%] distributed/tests/test_scheduler.py::test_story PASSED [ 64%] distributed/tests/test_scheduler.py::test_scatter_no_workers[False] PASSED [ 64%] distributed/tests/test_scheduler.py::test_scatter_no_workers[True] PASSED [ 64%] distributed/tests/test_scheduler.py::test_scheduler_sees_memory_limits PASSED [ 64%] distributed/tests/test_scheduler.py::test_retire_workers PASSED [ 64%] distributed/tests/test_scheduler.py::test_retire_workers_n PASSED [ 64%] distributed/tests/test_scheduler.py::test_workers_to_close PASSED [ 64%] distributed/tests/test_scheduler.py::test_workers_to_close_grouped PASSED [ 64%] distributed/tests/test_scheduler.py::test_workers_to_close_never_close_long_running[True] PASSED [ 64%] distributed/tests/test_scheduler.py::test_workers_to_close_never_close_long_running[False] PASSED [ 64%] distributed/tests/test_scheduler.py::test_retire_workers_no_suspicious_tasks PASSED [ 64%] distributed/tests/test_scheduler.py::test_balance_many_workers PASSED [ 64%] distributed/tests/test_scheduler.py::test_balance_many_workers_2 PASSED [ 64%] distributed/tests/test_scheduler.py::test_include_communication_in_occupancy PASSED [ 64%] distributed/tests/test_scheduler.py::test_nonempty_data_is_rejected PASSED [ 64%] distributed/tests/test_scheduler.py::test_worker_arrives_with_data_is_rejected PASSED [ 64%] distributed/tests/test_scheduler.py::test_run_on_scheduler_sync PASSED [ 65%] distributed/tests/test_scheduler.py::test_run_on_scheduler PASSED [ 65%] distributed/tests/test_scheduler.py::test_allow_pickle_false PASSED [ 65%] distributed/tests/test_scheduler.py::test_close_worker PASSED [ 65%] distributed/tests/test_scheduler.py::test_retire_workers_close PASSED [ 65%] distributed/tests/test_scheduler.py::test_fifo_submission PASSED [ 65%] distributed/tests/test_scheduler.py::test_scheduler_file PASSED [ 65%] distributed/tests/test_scheduler.py::test_dashboard_host[None-expect0-tcp://0.0.0.0] PASSED [ 65%] distributed/tests/test_scheduler.py::test_dashboard_host[None-expect0-tcp://127.0.0.1] PASSED [ 65%] distributed/tests/test_scheduler.py::test_dashboard_host[None-expect0-tcp://127.0.0.1:38275] PASSED [ 65%] distributed/tests/test_scheduler.py::test_dashboard_host[127.0.0.1:0-expect1-tcp://0.0.0.0] PASSED [ 65%] distributed/tests/test_scheduler.py::test_dashboard_host[127.0.0.1:0-expect1-tcp://127.0.0.1] PASSED [ 65%] distributed/tests/test_scheduler.py::test_dashboard_host[127.0.0.1:0-expect1-tcp://127.0.0.1:38275] PASSED [ 65%] distributed/tests/test_scheduler.py::test_profile_metadata PASSED [ 65%] distributed/tests/test_scheduler.py::test_profile_metadata_timeout PASSED [ 65%] distributed/tests/test_scheduler.py::test_profile_metadata_keys PASSED [ 65%] distributed/tests/test_scheduler.py::test_statistical_profiling PASSED [ 65%] distributed/tests/test_scheduler.py::test_statistical_profiling_failure PASSED [ 65%] distributed/tests/test_scheduler.py::test_cancel_fire_and_forget PASSED [ 65%] distributed/tests/test_scheduler.py::test_get_task_status PASSED [ 65%] distributed/tests/test_scheduler.py::test_deque_handler PASSED [ 65%] distributed/tests/test_scheduler.py::test_retries PASSED [ 65%] distributed/tests/test_scheduler.py::test_missing_data_errant_worker PASSED [ 65%] distributed/tests/test_scheduler.py::test_dont_recompute_if_persisted PASSED [ 65%] distributed/tests/test_scheduler.py::test_dont_recompute_if_persisted_2 PASSED [ 65%] distributed/tests/test_scheduler.py::test_dont_recompute_if_persisted_3 PASSED [ 65%] distributed/tests/test_scheduler.py::test_dont_recompute_if_persisted_4 PASSED [ 65%] distributed/tests/test_scheduler.py::test_dont_forget_released_keys PASSED [ 65%] distributed/tests/test_scheduler.py::test_dont_recompute_if_erred_transition_log PASSED [ 65%] distributed/tests/test_scheduler.py::test_dont_recompute_if_erred PASSED [ 65%] distributed/tests/test_scheduler.py::test_closing_scheduler_closes_workers PASSED [ 65%] distributed/tests/test_scheduler.py::test_resources_reset_after_cancelled_task PASSED [ 65%] distributed/tests/test_scheduler.py::test_gh2187 PASSED [ 65%] distributed/tests/test_scheduler.py::test_collect_versions PASSED [ 65%] distributed/tests/test_scheduler.py::test_idle_during_update_graph PASSED [ 66%] distributed/tests/test_scheduler.py::test_idle_timeout_no_workers PASSED [ 66%] distributed/tests/test_scheduler.py::test_no_workers_timeout_disabled PASSED [ 66%] distributed/tests/test_scheduler.py::test_no_workers_timeout_queued PASSED [ 66%] distributed/tests/test_scheduler.py::test_bandwidth PASSED [ 66%] distributed/tests/test_scheduler.py::test_workerstate_clean PASSED [ 66%] distributed/tests/test_scheduler.py::test_result_type PASSED [ 66%] distributed/tests/test_scheduler.py::test_close_workers PASSED [ 66%] distributed/tests/test_scheduler.py::test_host_address PASSED [ 66%] distributed/tests/test_scheduler.py::test_dashboard_address SKIPPED [ 66%] distributed/tests/test_scheduler.py::test_stop_services SKIPPED (could not import 'bokeh': No module named 'bokeh') [ 66%] distributed/tests/test_scheduler.py::test_adaptive_target PASSED [ 66%] distributed/tests/test_scheduler.py::test_adaptive_target_empty_cluster[True] PASSED [ 66%] distributed/tests/test_scheduler.py::test_adaptive_target_empty_cluster[False] PASSED [ 66%] distributed/tests/test_scheduler.py::test_async_context_manager PASSED [ 66%] distributed/tests/test_scheduler.py::test_allowed_failures_config PASSED [ 66%] distributed/tests/test_scheduler.py::test_finished PASSED [ 66%] distributed/tests/test_scheduler.py::test_retire_names_str PASSED [ 66%] distributed/tests/test_scheduler.py::test_retire_workers_bad_params PASSED [ 66%] distributed/tests/test_scheduler.py::test_get_prefix_duration PASSED [ 66%] distributed/tests/test_scheduler.py::test_default_task_duration_splits PASSED [ 66%] distributed/tests/test_scheduler.py::test_no_dangling_asyncio_tasks PASSED [ 66%] distributed/tests/test_scheduler.py::test_task_group_and_prefix_statistics PASSED [ 66%] distributed/tests/test_scheduler.py::test_task_groups_update_start_stop PASSED [ 66%] distributed/tests/test_scheduler.py::test_task_group_done PASSED [ 66%] distributed/tests/test_scheduler.py::test_task_group_not_done_waiting PASSED [ 66%] distributed/tests/test_scheduler.py::test_task_group_not_done_noworker PASSED [ 66%] distributed/tests/test_scheduler.py::test_task_group_not_done_queued PASSED [ 66%] distributed/tests/test_scheduler.py::test_task_group_not_done_processing PASSED [ 66%] distributed/tests/test_scheduler.py::test_task_prefix PASSED [ 66%] distributed/tests/test_scheduler.py::test_task_group_non_tuple_key PASSED [ 66%] distributed/tests/test_scheduler.py::test_task_unique_groups PASSED [ 66%] distributed/tests/test_scheduler.py::test_task_group_on_fire_and_forget PASSED [ 66%] distributed/tests/test_scheduler.py::test_gather_failing_can_recover PASSED [ 66%] distributed/tests/test_scheduler.py::test_gather_failing_cnn_error PASSED [ 66%] distributed/tests/test_scheduler.py::test_gather_no_workers PASSED [ 67%] distributed/tests/test_scheduler.py::test_gather_bad_worker[False] PASSED [ 67%] distributed/tests/test_scheduler.py::test_gather_bad_worker[True] PASSED [ 67%] distributed/tests/test_scheduler.py::test_too_many_groups PASSED [ 67%] distributed/tests/test_scheduler.py::test_multiple_listeners[{scheme}://{host}:{port}/status-dashboard at:\\s*http://] PASSED [ 67%] distributed/tests/test_scheduler.py::test_multiple_listeners[{ENV_VAR_MISSING}-dashboard at:\\s*:\\d*] PASSED [ 67%] distributed/tests/test_scheduler.py::test_worker_name_collision PASSED [ 67%] distributed/tests/test_scheduler.py::test_unknown_task_duration_config PASSED [ 67%] distributed/tests/test_scheduler.py::test_unknown_task_duration_config_2 PASSED [ 67%] distributed/tests/test_scheduler.py::test_retire_state_change PASSED [ 67%] distributed/tests/test_scheduler.py::test_get_worker_monitor_info PASSED [ 67%] distributed/tests/test_scheduler.py::test_quiet_cluster_round_robin PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_sum PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-0-0-0] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-0-0-1] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-0-0-2] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-0-0-3] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-0-1-0] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-0-1-1] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-0-1-2] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-0-1-3] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-0-2-0] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-0-2-1] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-0-2-2] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-0-2-3] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-0-3-0] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-0-3-1] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-0-3-2] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-0-3-3] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-1-0-0] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-1-0-1] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-1-0-2] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-1-0-3] PASSED [ 67%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-1-1-0] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-1-1-1] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-1-1-2] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-1-1-3] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-1-2-0] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-1-2-1] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-1-2-2] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-1-2-3] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-1-3-0] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-1-3-1] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-1-3-2] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-1-3-3] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-2-0-0] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-2-0-1] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-2-0-2] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-2-0-3] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-2-1-0] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-2-1-1] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-2-1-2] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-2-1-3] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-2-2-0] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-2-2-1] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-2-2-2] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-2-2-3] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-2-3-0] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-2-3-1] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-2-3-2] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-2-3-3] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-3-0-0] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-3-0-1] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-3-0-2] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-3-0-3] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-3-1-0] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-3-1-1] PASSED [ 68%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-3-1-2] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-3-1-3] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-3-2-0] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-3-2-1] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-3-2-2] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-3-2-3] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-3-3-0] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-3-3-1] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-3-3-2] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[0-3-3-3] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-0-0-0] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-0-0-1] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-0-0-2] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-0-0-3] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-0-1-0] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-0-1-1] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-0-1-2] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-0-1-3] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-0-2-0] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-0-2-1] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-0-2-2] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-0-2-3] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-0-3-0] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-0-3-1] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-0-3-2] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-0-3-3] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-1-0-0] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-1-0-1] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-1-0-2] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-1-0-3] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-1-1-0] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-1-1-1] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-1-1-2] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-1-1-3] PASSED [ 69%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-1-2-0] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-1-2-1] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-1-2-2] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-1-2-3] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-1-3-0] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-1-3-1] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-1-3-2] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-1-3-3] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-2-0-0] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-2-0-1] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-2-0-2] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-2-0-3] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-2-1-0] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-2-1-1] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-2-1-2] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-2-1-3] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-2-2-0] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-2-2-1] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-2-2-2] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-2-2-3] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-2-3-0] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-2-3-1] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-2-3-2] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-2-3-3] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-3-0-0] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-3-0-1] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-3-0-2] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-3-0-3] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-3-1-0] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-3-1-1] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-3-1-2] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-3-1-3] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-3-2-0] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-3-2-1] PASSED [ 70%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-3-2-2] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-3-2-3] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-3-3-0] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-3-3-1] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-3-3-2] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[1-3-3-3] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-0-0-0] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-0-0-1] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-0-0-2] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-0-0-3] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-0-1-0] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-0-1-1] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-0-1-2] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-0-1-3] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-0-2-0] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-0-2-1] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-0-2-2] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-0-2-3] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-0-3-0] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-0-3-1] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-0-3-2] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-0-3-3] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-1-0-0] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-1-0-1] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-1-0-2] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-1-0-3] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-1-1-0] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-1-1-1] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-1-1-2] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-1-1-3] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-1-2-0] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-1-2-1] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-1-2-2] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-1-2-3] PASSED [ 71%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-1-3-0] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-1-3-1] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-1-3-2] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-1-3-3] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-2-0-0] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-2-0-1] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-2-0-2] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-2-0-3] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-2-1-0] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-2-1-1] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-2-1-2] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-2-1-3] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-2-2-0] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-2-2-1] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-2-2-2] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-2-2-3] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-2-3-0] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-2-3-1] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-2-3-2] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-2-3-3] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-3-0-0] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-3-0-1] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-3-0-2] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-3-0-3] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-3-1-0] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-3-1-1] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-3-1-2] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-3-1-3] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-3-2-0] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-3-2-1] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-3-2-2] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-3-2-3] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-3-3-0] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-3-3-1] PASSED [ 72%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-3-3-2] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[2-3-3-3] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-0-0-0] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-0-0-1] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-0-0-2] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-0-0-3] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-0-1-0] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-0-1-1] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-0-1-2] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-0-1-3] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-0-2-0] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-0-2-1] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-0-2-2] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-0-2-3] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-0-3-0] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-0-3-1] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-0-3-2] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-0-3-3] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-1-0-0] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-1-0-1] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-1-0-2] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-1-0-3] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-1-1-0] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-1-1-1] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-1-1-2] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-1-1-3] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-1-2-0] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-1-2-1] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-1-2-2] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-1-2-3] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-1-3-0] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-1-3-1] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-1-3-2] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-1-3-3] PASSED [ 73%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-2-0-0] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-2-0-1] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-2-0-2] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-2-0-3] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-2-1-0] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-2-1-1] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-2-1-2] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-2-1-3] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-2-2-0] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-2-2-1] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-2-2-2] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-2-2-3] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-2-3-0] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-2-3-1] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-2-3-2] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-2-3-3] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-3-0-0] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-3-0-1] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-3-0-2] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-3-0-3] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-3-1-0] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-3-1-1] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-3-1-2] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-3-1-3] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-3-2-0] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-3-2-1] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-3-2-2] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-3-2-3] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-3-3-0] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-3-3-1] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-3-3-2] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate_adds_up[3-3-3-3] PASSED [ 74%] distributed/tests/test_scheduler.py::test_memorystate__to_dict PASSED [ 74%] distributed/tests/test_scheduler.py::test_memory_no_zict PASSED [ 74%] distributed/tests/test_scheduler.py::test_memory_no_workers PASSED [ 75%] distributed/tests/test_scheduler.py::test_infrequent_sysmon PASSED [ 75%] distributed/tests/test_scheduler.py::test_close_scheduler__close_workers_Worker PASSED [ 75%] distributed/tests/test_scheduler.py::test_rebalance_managed_memory PASSED [ 75%] distributed/tests/test_scheduler.py::test_rebalance_workers_and_keys PASSED [ 75%] distributed/tests/test_scheduler.py::test_rebalance_missing_data1 PASSED [ 75%] distributed/tests/test_scheduler.py::test_rebalance_missing_data2 PASSED [ 75%] distributed/tests/test_scheduler.py::test_rebalance_raises_missing_data3[False] PASSED [ 75%] distributed/tests/test_scheduler.py::test_rebalance_raises_missing_data3[True] XFAIL explicit RPC). Therefore, the partial-fail is very timing sensitive and subject to a race condition. This test assumes that the data is freed before the rebalance get_data requests come in but merely deleting the futures is not sufficient to guarantee this) [ 75%] distributed/tests/test_scheduler.py::test_rebalance_no_workers PASSED [ 75%] distributed/tests/test_scheduler.py::test_rebalance_no_limit PASSED [ 75%] distributed/tests/test_scheduler.py::test_rebalance_skip_recipient PASSED [ 75%] distributed/tests/test_scheduler.py::test_rebalance_skip_all_recipients PASSED [ 75%] distributed/tests/test_scheduler.py::test_gather_on_worker PASSED [ 75%] distributed/tests/test_scheduler.py::test_gather_on_worker_bad_recipient PASSED [ 75%] distributed/tests/test_scheduler.py::test_gather_on_worker_bad_sender PASSED [ 75%] distributed/tests/test_scheduler.py::test_gather_on_worker_bad_sender_replicated[False] PASSED [ 75%] distributed/tests/test_scheduler.py::test_gather_on_worker_bad_sender_replicated[True] PASSED [ 75%] distributed/tests/test_scheduler.py::test_gather_on_worker_key_not_on_sender PASSED [ 75%] distributed/tests/test_scheduler.py::test_gather_on_worker_key_not_on_sender_replicated[False] PASSED [ 75%] distributed/tests/test_scheduler.py::test_gather_on_worker_key_not_on_sender_replicated[True] PASSED [ 75%] distributed/tests/test_scheduler.py::test_gather_on_worker_duplicate_task PASSED [ 75%] distributed/tests/test_scheduler.py::test_rebalance_dead_recipient PASSED [ 75%] distributed/tests/test_scheduler.py::test_delete_worker_data PASSED [ 75%] distributed/tests/test_scheduler.py::test_delete_worker_data_double_delete PASSED [ 75%] distributed/tests/test_scheduler.py::test_delete_worker_data_bad_worker PASSED [ 75%] distributed/tests/test_scheduler.py::test_delete_worker_data_bad_task[False] PASSED [ 75%] distributed/tests/test_scheduler.py::test_delete_worker_data_bad_task[True] PASSED [ 75%] distributed/tests/test_scheduler.py::test_transition_counter PASSED [ 75%] distributed/tests/test_scheduler.py::test_transition_counter_max_scheduler PASSED [ 75%] distributed/tests/test_scheduler.py::test_transition_counter_max_worker PASSED [ 75%] distributed/tests/test_scheduler.py::test_disable_transition_counter_max PASSED [ 75%] distributed/tests/test_scheduler.py::test_worker_heartbeat_after_cancel PASSED [ 75%] distributed/tests/test_scheduler.py::test_set_restrictions PASSED [ 75%] distributed/tests/test_scheduler.py::test_avoid_paused_workers PASSED [ 76%] distributed/tests/test_scheduler.py::test_Scheduler__to_dict PASSED [ 76%] distributed/tests/test_scheduler.py::test_TaskState__to_dict PASSED [ 76%] distributed/tests/test_scheduler.py::test_get_cluster_state PASSED [ 76%] distributed/tests/test_scheduler.py::test_get_cluster_state_worker_error PASSED [ 76%] distributed/tests/test_scheduler.py::test_dump_cluster_state[msgpack] PASSED [ 76%] distributed/tests/test_scheduler.py::test_dump_cluster_state[yaml] PASSED [ 76%] distributed/tests/test_scheduler.py::test_repr PASSED [ 76%] distributed/tests/test_scheduler.py::test_ensure_events_dont_include_taskstate_objects PASSED [ 76%] distributed/tests/test_scheduler.py::test_worker_state_unique_regardless_of_address PASSED [ 76%] distributed/tests/test_scheduler.py::test_runspec_regression_sync PASSED [ 76%] distributed/tests/test_scheduler.py::test_KilledWorker_informative_message PASSED [ 76%] distributed/tests/test_scheduler.py::test_count_task_prefix PASSED [ 76%] distributed/tests/test_scheduler.py::test_transition_waiting_memory PASSED [ 76%] distributed/tests/test_scheduler.py::test_deadlock_resubmit_queued_tasks_fast[True] PASSED [ 76%] distributed/tests/test_scheduler.py::test_deadlock_resubmit_queued_tasks_fast[False] PASSED [ 76%] distributed/tests/test_scheduler.py::test_transition_failure_triggers_log_event PASSED [ 76%] distributed/tests/test_scheduler.py::test_deadlock_dependency_of_queued_released_when_worker_replaced[True] PASSED [ 76%] distributed/tests/test_scheduler.py::test_deadlock_dependency_of_queued_released_when_worker_replaced[False] PASSED [ 76%] distributed/tests/test_scheduler.py::test_deadlock_dependency_of_queued_released_when_worker_removed[True] PASSED [ 76%] distributed/tests/test_scheduler.py::test_deadlock_dependency_of_queued_released_when_worker_removed[False] PASSED [ 76%] distributed/tests/test_scheduler.py::test_submit_dependency_of_erred_task PASSED [ 76%] distributed/tests/test_scheduler.py::test_tell_workers_when_peers_have_left PASSED [ 76%] distributed/tests/test_scheduler.py::test_scatter_creates_ts PASSED [ 76%] distributed/tests/test_scheduler.py::test_refuse_to_schedule_huge_task[False] PASSED [ 76%] distributed/tests/test_scheduler.py::test_refuse_to_schedule_huge_task[True] PASSED [ 76%] distributed/tests/test_scheduler.py::test_html_repr PASSED [ 76%] distributed/tests/test_scheduler.py::test_resubmit_nondeterministic_task_different_deps[False] PASSED [ 76%] distributed/tests/test_scheduler.py::test_resubmit_nondeterministic_task_different_deps[True] PASSED [ 76%] distributed/tests/test_scheduler.py::test_fan_out_pattern_deadlock PASSED [ 76%] distributed/tests/test_scheduler.py::test_stimulus_from_erred_task PASSED [ 76%] distributed/tests/test_scheduler.py::test_concurrent_close_requests PASSED [ 76%] distributed/tests/test_scheduler.py::test_rootish_taskgroup_configuration PASSED [ 76%] distributed/tests/test_scheduler.py::test_alias_resolving_break_queuing PASSED [ 76%] distributed/tests/test_scheduler.py::test_data_producers PASSED [ 77%] distributed/tests/test_security.py::test_defaults PASSED [ 77%] distributed/tests/test_security.py::test_constructor_errors PASSED [ 77%] distributed/tests/test_security.py::test_attribute_error PASSED [ 77%] distributed/tests/test_security.py::test_from_config PASSED [ 77%] distributed/tests/test_security.py::test_min_max_version_from_config[None-None] PASSED [ 77%] distributed/tests/test_security.py::test_min_max_version_from_config[None-1.2] PASSED [ 77%] distributed/tests/test_security.py::test_min_max_version_from_config[None-1.3] PASSED [ 77%] distributed/tests/test_security.py::test_min_max_version_from_config[1.2-None] PASSED [ 77%] distributed/tests/test_security.py::test_min_max_version_from_config[1.2-1.2] PASSED [ 77%] distributed/tests/test_security.py::test_min_max_version_from_config[1.2-1.3] PASSED [ 77%] distributed/tests/test_security.py::test_min_max_version_from_config[1.3-None] PASSED [ 77%] distributed/tests/test_security.py::test_min_max_version_from_config[1.3-1.2] PASSED [ 77%] distributed/tests/test_security.py::test_min_max_version_from_config[1.3-1.3] PASSED [ 77%] distributed/tests/test_security.py::test_min_max_version_config_errors[min-version] PASSED [ 77%] distributed/tests/test_security.py::test_min_max_version_config_errors[max-version] PASSED [ 77%] distributed/tests/test_security.py::test_invalid_min_version_from_config_errors PASSED [ 77%] distributed/tests/test_security.py::test_kwargs PASSED [ 77%] distributed/tests/test_security.py::test_min_max_version_kwarg_errors[tls_min_version] PASSED [ 77%] distributed/tests/test_security.py::test_min_max_version_kwarg_errors[tls_max_version] PASSED [ 77%] distributed/tests/test_security.py::test_repr_temp_keys PASSED [ 77%] distributed/tests/test_security.py::test_repr_local_keys PASSED [ 77%] distributed/tests/test_security.py::test_tls_config_for_role PASSED [ 77%] distributed/tests/test_security.py::test_connection_args PASSED [ 77%] distributed/tests/test_security.py::test_extra_conn_args_connection_args PASSED [ 77%] distributed/tests/test_security.py::test_listen_args PASSED [ 77%] distributed/tests/test_security.py::test_tls_listen_connect PASSED [ 77%] distributed/tests/test_security.py::test_require_encryption PASSED [ 77%] distributed/tests/test_security.py::test_temporary_credentials PASSED [ 77%] distributed/tests/test_security.py::test_extra_conn_args_in_temporary_credentials PASSED [ 77%] distributed/tests/test_security.py::test_tls_temporary_credentials_functional PASSED [ 77%] distributed/tests/test_semaphore.py::test_semaphore_trivial PASSED [ 77%] distributed/tests/test_semaphore.py::test_serializable PASSED [ 77%] distributed/tests/test_semaphore.py::test_release_simple PASSED [ 77%] distributed/tests/test_semaphore.py::test_acquires_with_timeout PASSED [ 78%] distributed/tests/test_semaphore.py::test_timeout_sync PASSED [ 78%] distributed/tests/test_semaphore.py::test_release_semaphore_after_timeout PASSED [ 78%] distributed/tests/test_semaphore.py::test_async_ctx PASSED [ 78%] distributed/tests/test_semaphore.py::test_worker_dies PASSED [ 78%] distributed/tests/test_semaphore.py::test_access_semaphore_by_name PASSED [ 78%] distributed/tests/test_semaphore.py::test_close_async PASSED [ 78%] distributed/tests/test_semaphore.py::test_release_once_too_many PASSED [ 78%] distributed/tests/test_semaphore.py::test_release_once_too_many_resilience PASSED [ 78%] distributed/tests/test_semaphore.py::test_retry_acquire PASSED [ 78%] distributed/tests/test_semaphore.py::test_oversubscribing_leases PASSED [ 78%] distributed/tests/test_semaphore.py::test_timeout_zero PASSED [ 78%] distributed/tests/test_semaphore.py::test_getvalue PASSED [ 78%] distributed/tests/test_semaphore.py::test_metrics PASSED [ 78%] distributed/tests/test_semaphore.py::test_threadpoolworkers_pick_correct_ioloop PASSED [ 78%] distributed/tests/test_semaphore.py::test_release_retry PASSED [ 78%] distributed/tests/test_semaphore.py::test_release_failure PASSED [ 78%] distributed/tests/test_semaphore.py::test_unpickle_without_client PASSED [ 78%] distributed/tests/test_sizeof.py::test_safe_sizeof[obj0] PASSED [ 78%] distributed/tests/test_sizeof.py::test_safe_sizeof[obj1] PASSED [ 78%] distributed/tests/test_sizeof.py::test_safe_sizeof[obj2] PASSED [ 78%] distributed/tests/test_sizeof.py::test_safe_sizeof_logs_on_failure PASSED [ 78%] distributed/tests/test_spans.py::test_spans PASSED [ 78%] distributed/tests/test_spans.py::test_submit PASSED [ 78%] distributed/tests/test_spans.py::test_multiple_tags PASSED [ 78%] distributed/tests/test_spans.py::test_repeat_span PASSED [ 78%] distributed/tests/test_spans.py::test_default_span PASSED [ 78%] distributed/tests/test_spans.py::test_no_extension PASSED [ 78%] distributed/tests/test_spans.py::test_task_groups[False] PASSED [ 78%] distributed/tests/test_spans.py::test_task_groups[True] PASSED [ 78%] distributed/tests/test_spans.py::test_before_first_task_finished PASSED [ 78%] distributed/tests/test_spans.py::test_duplicate_task_group PASSED [ 78%] distributed/tests/test_spans.py::test_mismatched_span[False] PASSED [ 78%] distributed/tests/test_spans.py::test_mismatched_span[True] PASSED [ 78%] distributed/tests/test_spans.py::test_no_tags PASSED [ 79%] distributed/tests/test_spans.py::test_scatter_creates_ts PASSED [ 79%] distributed/tests/test_spans.py::test_scatter_creates_tg PASSED [ 79%] distributed/tests/test_spans.py::test_worker_metrics PASSED [ 79%] distributed/tests/test_spans.py::test_merge_by_tags PASSED [ 79%] distributed/tests/test_spans.py::test_merge_by_tags_metrics PASSED [ 79%] distributed/tests/test_spans.py::test_code PASSED [ 79%] distributed/tests/test_spans.py::test_no_code_by_default PASSED [ 79%] distributed/tests/test_spans.py::test_merge_all PASSED [ 79%] distributed/tests/test_spans.py::test_merge_nothing PASSED [ 79%] distributed/tests/test_spans.py::test_active_cpu_seconds_trivial PASSED [ 79%] distributed/tests/test_spans.py::test_active_cpu_seconds_not_done[False] PASSED [ 79%] distributed/tests/test_spans.py::test_active_cpu_seconds_not_done[True] PASSED [ 79%] distributed/tests/test_spans.py::test_active_cpu_seconds_change_nthreads PASSED [ 79%] distributed/tests/test_spans.py::test_active_cpu_seconds_merged PASSED [ 79%] distributed/tests/test_spans.py::test_spans_are_visible_from_tasks PASSED [ 79%] distributed/tests/test_spans.py::test_span_on_persist PASSED [ 79%] distributed/tests/test_spans.py::test_collections_metadata PASSED [ 79%] distributed/tests/test_spec.py::test_address_default_none PASSED [ 79%] distributed/tests/test_spec.py::test_child_address_persists PASSED [ 79%] distributed/tests/test_spill.py::test_psize PASSED [ 79%] distributed/tests/test_spill.py::test_spillbuffer PASSED [ 79%] distributed/tests/test_spill.py::test_disk_size_calculation PASSED [ 79%] distributed/tests/test_spill.py::test_spillbuffer_maxlim PASSED [ 79%] distributed/tests/test_spill.py::test_spillbuffer_fail_to_serialize PASSED [ 79%] distributed/tests/test_spill.py::test_spillbuffer_oserror PASSED [ 79%] distributed/tests/test_spill.py::test_spillbuffer_evict PASSED [ 79%] distributed/tests/test_spill.py::test_no_pop PASSED [ 79%] distributed/tests/test_spill.py::test_weakref_cache[60-SupportsWeakRef-True] PASSED [ 79%] distributed/tests/test_spill.py::test_weakref_cache[60-NoWeakRef-False] PASSED [ 79%] distributed/tests/test_spill.py::test_weakref_cache[110-SupportsWeakRef-True] PASSED [ 79%] distributed/tests/test_spill.py::test_weakref_cache[110-NoWeakRef-False] PASSED [ 79%] distributed/tests/test_spill.py::test_metrics PASSED [ 79%] distributed/tests/test_spill.py::test_compression_settings[zlib-100-500] PASSED [ 79%] distributed/tests/test_spill.py::test_compression_settings[None-20000-20500] PASSED [ 80%] distributed/tests/test_spill.py::test_str_collision PASSED [ 80%] distributed/tests/test_steal.py::test_work_stealing PASSED [ 80%] distributed/tests/test_steal.py::test_dont_steal_expensive_data_fast_computation PASSED [ 80%] distributed/tests/test_steal.py::test_steal_cheap_data_slow_computation PASSED [ 80%] distributed/tests/test_steal.py::test_steal_expensive_data_slow_computation PASSED [ 80%] distributed/tests/test_steal.py::test_worksteal_many_thieves PASSED [ 80%] distributed/tests/test_steal.py::test_stop_plugin PASSED [ 80%] distributed/tests/test_steal.py::test_stop_in_flight PASSED [ 80%] distributed/tests/test_steal.py::test_eventually_steal_unknown_functions PASSED [ 80%] distributed/tests/test_steal.py::test_steal_related_tasks SKIPPED [ 80%] distributed/tests/test_steal.py::test_dont_steal_fast_tasks_compute_time FAILED [ 80%] distributed/tests/test_steal.py::test_dont_steal_fast_tasks_blocklist PASSED [ 80%] distributed/tests/test_steal.py::test_dont_steal_worker_restrictions PASSED [ 80%] distributed/tests/test_steal.py::test_steal_worker_restrictions PASSED [ 80%] distributed/tests/test_steal.py::test_dont_steal_host_restrictions PASSED [ 80%] distributed/tests/test_steal.py::test_steal_host_restrictions FAILED [ 80%] distributed/tests/test_steal.py::test_dont_steal_resource_restrictions PASSED [ 80%] distributed/tests/test_steal.py::test_steal_resource_restrictions PASSED [ 80%] distributed/tests/test_steal.py::test_steal_resource_restrictions_asym_diff PASSED [ 80%] distributed/tests/test_steal.py::test_balance_without_dependencies PASSED [ 80%] distributed/tests/test_steal.py::test_dont_steal_executing_tasks PASSED [ 80%] distributed/tests/test_steal.py::test_dont_steal_executing_tasks_2 PASSED [ 80%] distributed/tests/test_steal.py::test_dont_steal_few_saturated_tasks_many_workers PASSED [ 80%] distributed/tests/test_steal.py::test_steal_large_dependency PASSED [ 80%] distributed/tests/test_steal.py::test_steal_more_attractive_tasks PASSED [ 80%] distributed/tests/test_steal.py::test_balance[don't move unnecessarily] PASSED [ 80%] distributed/tests/test_steal.py::test_balance[balance] PASSED [ 80%] distributed/tests/test_steal.py::test_balance[balance even if results in even] PASSED [ 80%] distributed/tests/test_steal.py::test_balance[don't over balance] PASSED [ 80%] distributed/tests/test_steal.py::test_balance[move from larger] PASSED [ 80%] distributed/tests/test_steal.py::test_balance[move to smaller] PASSED [ 80%] distributed/tests/test_steal.py::test_balance[choose easier first] PASSED [ 80%] distributed/tests/test_steal.py::test_balance[spread evenly] PASSED [ 80%] distributed/tests/test_steal.py::test_balance[move easier] PASSED [ 81%] distributed/tests/test_steal.py::test_balance[be willing to move costly items] PASSED [ 81%] distributed/tests/test_steal.py::test_balance[but don't move too many] PASSED [ 81%] distributed/tests/test_steal.py::test_balance[no one clearly saturated] PASSED [ 81%] distributed/tests/test_steal.py::test_balance[balance multiple saturated workers] PASSED [ 81%] distributed/tests/test_steal.py::test_restart PASSED [ 81%] distributed/tests/test_steal.py::test_paused_workers_must_not_steal PASSED [ 81%] distributed/tests/test_steal.py::test_dont_steal_already_released PASSED [ 81%] distributed/tests/test_steal.py::test_dont_steal_long_running_tasks PASSED [ 81%] distributed/tests/test_steal.py::test_cleanup_repeated_tasks PASSED [ 81%] distributed/tests/test_steal.py::test_lose_task PASSED [ 81%] distributed/tests/test_steal.py::test_parse_stealing_interval[None-1000] PASSED [ 81%] distributed/tests/test_steal.py::test_parse_stealing_interval[500ms-500] PASSED [ 81%] distributed/tests/test_steal.py::test_parse_stealing_interval[2-2] PASSED [ 81%] distributed/tests/test_steal.py::test_blocklist_shuffle_split PASSED [ 81%] distributed/tests/test_steal.py::test_steal_concurrent_simple PASSED [ 81%] distributed/tests/test_steal.py::test_steal_reschedule_reset_in_flight_occupancy PASSED [ 81%] distributed/tests/test_steal.py::test_get_story PASSED [ 81%] distributed/tests/test_steal.py::test_steal_worker_dies_same_ip PASSED [ 81%] distributed/tests/test_steal.py::test_reschedule_concurrent_requests_deadlock PASSED [ 81%] distributed/tests/test_steal.py::test_correct_bad_time_estimate SKIPPED [ 81%] distributed/tests/test_steal.py::test_steal_stimulus_id_unique PASSED [ 81%] distributed/tests/test_steal.py::test_steal_worker_state[executing] PASSED [ 81%] distributed/tests/test_steal.py::test_steal_worker_state[long-running] PASSED [ 81%] distributed/tests/test_steal.py::test_steal_very_fast_tasks PASSED [ 81%] distributed/tests/test_steal.py::test_balance_expensive_tasks[not enough work to steal] PASSED [ 81%] distributed/tests/test_steal.py::test_balance_expensive_tasks[enough work to steal] PASSED [ 81%] distributed/tests/test_steal.py::test_balance_expensive_tasks[not enough work for increased cost] PASSED [ 81%] distributed/tests/test_steal.py::test_balance_uneven_without_replica PASSED [ 81%] distributed/tests/test_steal.py::test_balance_eventually_steals_large_dependency_without_replica PASSED [ 81%] distributed/tests/test_steal.py::test_balance_even_with_replica PASSED [ 81%] distributed/tests/test_steal.py::test_balance_to_replica PASSED [ 81%] distributed/tests/test_steal.py::test_balance_multiple_to_replica PASSED [ 81%] distributed/tests/test_steal.py::test_balance_to_larger_dependency PASSED [ 81%] distributed/tests/test_steal.py::test_balance_prefers_busier_with_dependency PASSED [ 82%] distributed/tests/test_steal.py::test_trivial_workload_should_not_cause_work_stealing PASSED [ 82%] distributed/tests/test_steal.py::test_stealing_objective_accounts_for_in_flight PASSED [ 82%] distributed/tests/test_steal.py::test_do_not_ping_pong PASSED [ 82%] distributed/tests/test_stories.py::test_scheduler_story_stimulus_success PASSED [ 82%] distributed/tests/test_stories.py::test_scheduler_story_stimulus_retry PASSED [ 82%] distributed/tests/test_stories.py::test_client_story PASSED [ 82%] distributed/tests/test_stories.py::test_client_story_failed_worker[ignore] PASSED [ 82%] distributed/tests/test_stories.py::test_client_story_failed_worker[raise] PASSED [ 82%] distributed/tests/test_stories.py::test_worker_story_with_deps PASSED [ 82%] distributed/tests/test_stress.py::test_stress_1 PASSED [ 82%] distributed/tests/test_stress.py::test_cancel_stress PASSED [ 82%] distributed/tests/test_stress.py::test_cancel_stress_sync PASSED [ 82%] distributed/tests/test_stress.py::test_stress_scatter_death PASSED [ 82%] distributed/tests/test_stress.py::test_stress_steal SKIPPED [ 82%] distributed/tests/test_system.py::test_memory_limit PASSED [ 82%] distributed/tests/test_system.py::test_hard_memory_limit_cgroups PASSED [ 82%] distributed/tests/test_system.py::test_soft_memory_limit_cgroups PASSED [ 82%] distributed/tests/test_system.py::test_hard_memory_limit_cgroups2 PASSED [ 82%] distributed/tests/test_system.py::test_soft_memory_limit_cgroups2 PASSED [ 82%] distributed/tests/test_system.py::test_softest_memory_limit_cgroups2 PASSED [ 82%] distributed/tests/test_system.py::test_rlimit PASSED [ 82%] distributed/tests/test_system_monitor.py::test_SystemMonitor PASSED [ 82%] distributed/tests/test_system_monitor.py::test_maxlen_zero PASSED [ 82%] distributed/tests/test_system_monitor.py::test_count PASSED [ 82%] distributed/tests/test_system_monitor.py::test_range_query PASSED [ 82%] distributed/tests/test_system_monitor.py::test_disk_config PASSED [ 82%] distributed/tests/test_system_monitor.py::test_host_cpu PASSED [ 82%] distributed/tests/test_system_monitor.py::test_gil_contention SKIPPED [ 82%] distributed/tests/test_threadpoolexecutor.py::test_tpe PASSED [ 82%] distributed/tests/test_threadpoolexecutor.py::test_shutdown_timeout PASSED [ 82%] distributed/tests/test_threadpoolexecutor.py::test_shutdown_timeout_raises PASSED [ 82%] distributed/tests/test_threadpoolexecutor.py::test_shutdown_wait PASSED [ 82%] distributed/tests/test_threadpoolexecutor.py::test_secede_rejoin_busy PASSED [ 82%] distributed/tests/test_threadpoolexecutor.py::test_secede_rejoin_quiet PASSED [ 83%] distributed/tests/test_threadpoolexecutor.py::test_rejoin_idempotent PASSED [ 83%] distributed/tests/test_threadpoolexecutor.py::test_thread_name PASSED [ 83%] distributed/tests/test_tls_functional.py::test_basic PASSED [ 83%] distributed/tests/test_tls_functional.py::test_Queue PASSED [ 83%] distributed/tests/test_tls_functional.py::test_client_submit PASSED [ 83%] distributed/tests/test_tls_functional.py::test_gather PASSED [ 83%] distributed/tests/test_tls_functional.py::test_scatter PASSED [ 83%] distributed/tests/test_tls_functional.py::test_nanny PASSED [ 83%] distributed/tests/test_tls_functional.py::test_rebalance PASSED [ 83%] distributed/tests/test_tls_functional.py::test_work_stealing PASSED [ 83%] distributed/tests/test_tls_functional.py::test_worker_client PASSED [ 83%] distributed/tests/test_tls_functional.py::test_worker_client_gather PASSED [ 83%] distributed/tests/test_tls_functional.py::test_worker_client_executor PASSED [ 83%] distributed/tests/test_tls_functional.py::test_retire_workers PASSED [ 83%] distributed/tests/test_tls_functional.py::test_security_dict_input_no_security PASSED [ 83%] distributed/tests/test_tls_functional.py::test_security_bool_input_disabled_security PASSED [ 83%] distributed/tests/test_tls_functional.py::test_security_dict_input PASSED [ 83%] distributed/tests/test_utils.py::test_All PASSED [ 83%] distributed/tests/test_utils.py::test_sync PASSED [ 83%] distributed/tests/test_utils.py::test_sync_error PASSED [ 83%] distributed/tests/test_utils.py::test_sync_timeout PASSED [ 83%] distributed/tests/test_utils.py::test_sync_closed_loop PASSED [ 83%] distributed/tests/test_utils.py::test_sync_contextvars PASSED [ 83%] distributed/tests/test_utils.py::test_is_kernel PASSED [ 83%] distributed/tests/test_utils.py::test_ensure_ip PASSED [ 83%] distributed/tests/test_utils.py::test_get_ip_interface PASSED [ 83%] distributed/tests/test_utils.py::test_get_mp_context PASSED [ 83%] distributed/tests/test_utils.py::test_truncate_exception PASSED [ 83%] distributed/tests/test_utils.py::test_get_traceback PASSED [ 83%] distributed/tests/test_utils.py::test_read_block PASSED [ 83%] distributed/tests/test_utils.py::test_seek_delimiter_endline PASSED [ 83%] distributed/tests/test_utils.py::test_ensure_memoryview[] PASSED [ 83%] distributed/tests/test_utils.py::test_ensure_memoryview[data1] PASSED [ 83%] distributed/tests/test_utils.py::test_ensure_memoryview[1] PASSED [ 83%] distributed/tests/test_utils.py::test_ensure_memoryview[data3] PASSED [ 84%] distributed/tests/test_utils.py::test_ensure_memoryview[data4] PASSED [ 84%] distributed/tests/test_utils.py::test_ensure_memoryview[data5] PASSED [ 84%] distributed/tests/test_utils.py::test_ensure_memoryview[data6] PASSED [ 84%] distributed/tests/test_utils.py::test_ensure_memoryview[data7] PASSED [ 84%] distributed/tests/test_utils.py::test_ensure_memoryview[data8] PASSED [ 84%] distributed/tests/test_utils.py::test_ensure_memoryview[data9] PASSED [ 84%] distributed/tests/test_utils.py::test_ensure_memoryview[data10] PASSED [ 84%] distributed/tests/test_utils.py::test_ensure_memoryview[data11] PASSED [ 84%] distributed/tests/test_utils.py::test_ensure_memoryview[data12] PASSED [ 84%] distributed/tests/test_utils.py::test_ensure_memoryview[data13] PASSED [ 84%] distributed/tests/test_utils.py::test_ensure_memoryview[data14] PASSED [ 84%] distributed/tests/test_utils.py::test_ensure_memoryview_ndarray[i8-12-shape0-strides0] PASSED [ 84%] distributed/tests/test_utils.py::test_ensure_memoryview_ndarray[i8-12-shape1-strides1] PASSED [ 84%] distributed/tests/test_utils.py::test_ensure_memoryview_ndarray[i8-12-shape2-strides2] PASSED [ 84%] distributed/tests/test_utils.py::test_ensure_memoryview_ndarray[i8-12-shape3-strides3] PASSED [ 84%] distributed/tests/test_utils.py::test_ensure_memoryview_ndarray[i8-12-shape4-strides4] PASSED [ 84%] distributed/tests/test_utils.py::test_ensure_memoryview_pyarrow_buffer PASSED [ 84%] distributed/tests/test_utils.py::test_nbytes PASSED [ 84%] distributed/tests/test_utils.py::test_open_port PASSED [ 84%] distributed/tests/test_utils.py::test_set_thread_state PASSED [ 84%] distributed/tests/test_utils.py::test_loop_runner PASSED [ 84%] distributed/tests/test_utils.py::test_two_loop_runners PASSED [ 84%] distributed/tests/test_utils.py::test_loop_runner_gen PASSED [ 84%] distributed/tests/test_utils.py::test_loop_runner_exception_in_start PASSED [ 84%] distributed/tests/test_utils.py::test_loop_runner_exception_in_teardown PASSED [ 84%] distributed/tests/test_utils.py::test_all_quiet_exceptions PASSED [ 84%] distributed/tests/test_utils.py::test_warn_on_duration PASSED [ 84%] distributed/tests/test_utils.py::test_logs PASSED [ 84%] distributed/tests/test_utils.py::test_is_valid_xml PASSED [ 84%] distributed/tests/test_utils.py::test_format_dashboard_link PASSED [ 84%] distributed/tests/test_utils.py::test_parse_ports PASSED [ 84%] distributed/tests/test_utils.py::test_run_in_executor_with_context PASSED [ 84%] distributed/tests/test_utils.py::test_run_in_executor_with_context_preserves_contextvars PASSED [ 84%] distributed/tests/test_utils.py::test_offload PASSED [ 85%] distributed/tests/test_utils.py::test_offload_preserves_contextvars PASSED [ 85%] distributed/tests/test_utils.py::test_iscoroutinefunction_unhashable_input PASSED [ 85%] distributed/tests/test_utils.py::test_iscoroutinefunction_nested_partial PASSED [ 85%] distributed/tests/test_utils.py::test_recursive_to_dict PASSED [ 85%] distributed/tests/test_utils.py::test_recursive_to_dict_no_nest PASSED [ 85%] distributed/tests/test_utils.py::test_log_errors PASSED [ 85%] distributed/tests/test_utils.py::test_log_errors_unroll_stack[0-test_utils] PASSED [ 85%] distributed/tests/test_utils.py::test_log_errors_unroll_stack[1-a.b.c] PASSED [ 85%] distributed/tests/test_utils.py::test_load_json_robust_timeout PASSED [ 85%] distributed/tests/test_utils.py::test_rate_limiter_filter PASSED [ 85%] distributed/tests/test_utils.py::test_tuple_comparable_lt[obj10-obj20-False] PASSED [ 85%] distributed/tests/test_utils.py::test_tuple_comparable_lt[obj11-obj21-True] PASSED [ 85%] distributed/tests/test_utils.py::test_tuple_comparable_lt[1-1-False] PASSED [ 85%] distributed/tests/test_utils.py::test_tuple_comparable_lt[1-2-True] PASSED [ 85%] distributed/tests/test_utils.py::test_tuple_comparable_lt[None-0-False] PASSED [ 85%] distributed/tests/test_utils.py::test_tuple_comparable_lt[None-obj25-True] PASSED [ 85%] distributed/tests/test_utils.py::test_tuple_comparable_eq[obj10-obj20-True] PASSED [ 85%] distributed/tests/test_utils.py::test_tuple_comparable_eq[obj11-obj21-False] PASSED [ 85%] distributed/tests/test_utils.py::test_tuple_comparable_eq[1-1-True] PASSED [ 85%] distributed/tests/test_utils.py::test_tuple_comparable_eq[1-2-False] PASSED [ 85%] distributed/tests/test_utils.py::test_tuple_comparable_eq[None-0-True] PASSED [ 85%] distributed/tests/test_utils.py::test_tuple_comparable_eq[None-obj25-False] PASSED [ 85%] distributed/tests/test_utils.py::test_tuple_comparable_error PASSED [ 85%] distributed/tests/test_utils_comm.py::test_pack_data PASSED [ 85%] distributed/tests/test_utils_comm.py::test_subs_multiple PASSED [ 85%] distributed/tests/test_utils_comm.py::test_gather_from_workers_missing_replicas PASSED [ 85%] distributed/tests/test_utils_comm.py::test_gather_from_workers_permissive PASSED [ 85%] distributed/tests/test_utils_comm.py::test_gather_from_workers_permissive_flaky PASSED [ 85%] distributed/tests/test_utils_comm.py::test_gather_from_workers_busy PASSED [ 85%] distributed/tests/test_utils_comm.py::test_gather_from_workers_serialization_error[pickle] PASSED [ 85%] distributed/tests/test_utils_comm.py::test_gather_from_workers_serialization_error[unpickle] PASSED [ 85%] distributed/tests/test_utils_comm.py::test_retry_no_exception PASSED [ 85%] distributed/tests/test_utils_comm.py::test_retry0_raises_immediately PASSED [ 85%] distributed/tests/test_utils_comm.py::test_retry_does_retry_and_sleep PASSED [ 86%] distributed/tests/test_utils_comm.py::test_retry_truncates_large_coro_repr[1] PASSED [ 86%] distributed/tests/test_utils_comm.py::test_retry_truncates_large_coro_repr[2] PASSED [ 86%] distributed/tests/test_utils_comm.py::test_unpack_remotedata PASSED [ 86%] distributed/tests/test_utils_test.py::test_bare_cluster PASSED [ 86%] distributed/tests/test_utils_test.py::test_cluster PASSED [ 86%] distributed/tests/test_utils_test.py::test_gen_cluster PASSED [ 86%] distributed/tests/test_utils_test.py::test_gen_cluster_pytest_fixture PASSED [ 86%] distributed/tests/test_utils_test.py::test_gen_cluster_parametrized[True] PASSED [ 86%] distributed/tests/test_utils_test.py::test_gen_cluster_multi_parametrized[a-True] PASSED [ 86%] distributed/tests/test_utils_test.py::test_gen_cluster_multi_parametrized[b-True] PASSED [ 86%] distributed/tests/test_utils_test.py::test_gen_cluster_parametrized_variadic_workers[True] PASSED [ 86%] distributed/tests/test_utils_test.py::test_gen_cluster_set_config_nanny PASSED [ 86%] distributed/tests/test_utils_test.py::test_gen_cluster_cleans_up_client SKIPPED [ 86%] distributed/tests/test_utils_test.py::test_gen_cluster_without_client PASSED [ 86%] distributed/tests/test_utils_test.py::test_gen_cluster_tls PASSED [ 86%] distributed/tests/test_utils_test.py::test_gen_cluster_validate PASSED [ 86%] distributed/tests/test_utils_test.py::test_gen_cluster_validate_override PASSED [ 86%] distributed/tests/test_utils_test.py::test_gen_test XFAIL (Test should always fail to ensure the body of the test function was run) [ 86%] distributed/tests/test_utils_test.py::test_gen_test_legacy_implicit XFAILn) [ 86%] distributed/tests/test_utils_test.py::test_gen_test_legacy_explicit XFAILn) [ 86%] distributed/tests/test_utils_test.py::test_gen_test_parametrized[True] PASSED [ 86%] distributed/tests/test_utils_test.py::test_gen_test_double_parametrized[False-True] PASSED [ 86%] distributed/tests/test_utils_test.py::test_gen_test_pytest_fixture PASSED [ 86%] distributed/tests/test_utils_test.py::test_gen_test_config PASSED [ 86%] distributed/tests/test_utils_test.py::test_gen_test_config_default PASSED [ 86%] distributed/tests/test_utils_test.py::test_new_config PASSED [ 86%] distributed/tests/test_utils_test.py::test_lingering_client PASSED [ 86%] distributed/tests/test_utils_test.py::test_lingering_client_2 PASSED [ 86%] distributed/tests/test_utils_test.py::test_tls_cluster PASSED [ 86%] distributed/tests/test_utils_test.py::test_tls_scheduler PASSED [ 86%] distributed/tests/test_utils_test.py::test__UnhashableCallable PASSED [ 86%] distributed/tests/test_utils_test.py::test_locked_comm_drop_in_replacement PASSED [ 86%] distributed/tests/test_utils_test.py::test_locked_comm_intercept_read PASSED [ 86%] distributed/tests/test_utils_test.py::test_locked_comm_intercept_write PASSED [ 87%] distributed/tests/test_utils_test.py::test_assert_story PASSED [ 87%] distributed/tests/test_utils_test.py::test_assert_story_malformed_story[Missing payload, stimulus_id, ts] PASSED [ 87%] distributed/tests/test_utils_test.py::test_assert_story_malformed_story[Missing (stimulus_id, ts)] PASSED [ 87%] distributed/tests/test_utils_test.py::test_assert_story_malformed_story[Missing ts] PASSED [ 87%] distributed/tests/test_utils_test.py::test_assert_story_malformed_story[ts is not a float] PASSED [ 87%] distributed/tests/test_utils_test.py::test_assert_story_malformed_story[ts is in the future] PASSED [ 87%] distributed/tests/test_utils_test.py::test_assert_story_malformed_story[ts is too old] PASSED [ 87%] distributed/tests/test_utils_test.py::test_assert_story_malformed_story[stimulus_id is not a str] PASSED [ 87%] distributed/tests/test_utils_test.py::test_assert_story_malformed_story[stimulus_id is an empty str] PASSED [ 87%] distributed/tests/test_utils_test.py::test_assert_story_malformed_story[no payload] PASSED [ 87%] distributed/tests/test_utils_test.py::test_assert_story_malformed_story[timestamps out of order] PASSED [ 87%] distributed/tests/test_utils_test.py::test_assert_story_identity[True] PASSED [ 87%] distributed/tests/test_utils_test.py::test_assert_story_identity[False] PASSED [ 87%] distributed/tests/test_utils_test.py::test_check_process_leak PASSED [ 87%] distributed/tests/test_utils_test.py::test_check_process_leak_slow_cleanup PASSED [ 87%] distributed/tests/test_utils_test.py::test_check_process_leak_pre_cleanup[False] PASSED [ 87%] distributed/tests/test_utils_test.py::test_check_process_leak_pre_cleanup[True] PASSED [ 87%] distributed/tests/test_utils_test.py::test_check_process_leak_post_cleanup[False] PASSED [ 87%] distributed/tests/test_utils_test.py::test_check_process_leak_post_cleanup[True] PASSED [ 87%] distributed/tests/test_utils_test.py::test_start_failure_worker[True] PASSED [ 87%] distributed/tests/test_utils_test.py::test_start_failure_worker[False] PASSED [ 87%] distributed/tests/test_utils_test.py::test_start_failure_scheduler PASSED [ 87%] distributed/tests/test_utils_test.py::test_invalid_transitions PASSED [ 87%] distributed/tests/test_utils_test.py::test_invalid_worker_state PASSED [ 87%] distributed/tests/test_utils_test.py::test_raises_with_cause PASSED [ 87%] distributed/tests/test_utils_test.py::test_fail_hard[True] PASSED [ 87%] distributed/tests/test_utils_test.py::test_fail_hard[False] PASSED [ 87%] distributed/tests/test_utils_test.py::test_popen_write_during_terminate_deadlock PASSED [ 87%] distributed/tests/test_utils_test.py::test_popen_timeout PASSED [ 87%] distributed/tests/test_utils_test.py::test_popen_always_prints_output PASSED [ 87%] distributed/tests/test_utils_test.py::test_freeze_batched_send PASSED [ 87%] distributed/tests/test_utils_test.py::test_wait_for_state PASSED [ 87%] distributed/tests/test_utils_test.py::test_wait_for_stimulus PASSED [ 87%] distributed/tests/test_utils_test.py::test_ws_with_running_task[executing] PASSED [ 88%] distributed/tests/test_utils_test.py::test_ws_with_running_task[long-running] PASSED [ 88%] distributed/tests/test_utils_test.py::test_sizeof PASSED [ 88%] distributed/tests/test_utils_test.py::test_sizeof_error[12345.0-TypeError-Expected integer] PASSED [ 88%] distributed/tests/test_utils_test.py::test_sizeof_error[-1-ValueError-larger than] PASSED [ 88%] distributed/tests/test_utils_test.py::test_sizeof_error[0-ValueError-larger than] PASSED [ 88%] distributed/tests/test_utils_test.py::test_sizeof_error[10-ValueError-larger than] PASSED [ 88%] distributed/tests/test_utils_test.py::test_ensure_no_new_clients PASSED [ 88%] distributed/tests/test_utils_test.py::test_cluster_uses_config_for_test[False] PASSED [ 88%] distributed/tests/test_utils_test.py::test_cluster_uses_config_for_test[True] PASSED [ 88%] distributed/tests/test_utils_test.py::test_captured_logger PASSED [ 88%] distributed/tests/test_utils_test.py::test_captured_context_meter PASSED [ 88%] distributed/tests/test_utils_test.py::test_popen_file_not_found PASSED [ 88%] distributed/tests/test_variable.py::test_variable PASSED [ 88%] distributed/tests/test_variable.py::test_variable_in_task PASSED [ 88%] distributed/tests/test_variable.py::test_delete_unset_variable PASSED [ 88%] distributed/tests/test_variable.py::test_queue_with_data PASSED [ 88%] distributed/tests/test_variable.py::test_sync PASSED [ 88%] distributed/tests/test_variable.py::test_hold_futures PASSED [ 88%] distributed/tests/test_variable.py::test_timeout PASSED [ 88%] distributed/tests/test_variable.py::test_timeout_sync PASSED [ 88%] distributed/tests/test_variable.py::test_cleanup PASSED [ 88%] distributed/tests/test_variable.py::test_pickleable PASSED [ 88%] distributed/tests/test_variable.py::test_timeout_get PASSED [ 88%] distributed/tests/test_variable.py::test_Future_knows_status_immediately PASSED [ 88%] distributed/tests/test_variable.py::test_erred_future PASSED [ 88%] distributed/tests/test_variable.py::test_future_erred_sync PASSED [ 88%] distributed/tests/test_variable.py::test_variables_do_not_leak_client PASSED [ 88%] distributed/tests/test_variable.py::test_unpickle_without_client PASSED [ 88%] distributed/tests/test_variable.py::test_set_cancelled_future PASSED [ 88%] distributed/tests/test_versions.py::test_versions_match PASSED [ 88%] distributed/tests/test_versions.py::test_version_mismatch[source-MISMATCHED] PASSED [ 88%] distributed/tests/test_versions.py::test_version_mismatch[source-MISSING] PASSED [ 88%] distributed/tests/test_versions.py::test_version_mismatch[source-KEY_ERROR] PASSED [ 88%] distributed/tests/test_versions.py::test_version_mismatch[source-NONE] PASSED [ 89%] distributed/tests/test_versions.py::test_version_mismatch[scheduler-MISMATCHED] PASSED [ 89%] distributed/tests/test_versions.py::test_version_mismatch[scheduler-MISSING] PASSED [ 89%] distributed/tests/test_versions.py::test_version_mismatch[scheduler-KEY_ERROR] PASSED [ 89%] distributed/tests/test_versions.py::test_version_mismatch[scheduler-NONE] PASSED [ 89%] distributed/tests/test_versions.py::test_version_mismatch[worker-1-MISMATCHED] PASSED [ 89%] distributed/tests/test_versions.py::test_version_mismatch[worker-1-MISSING] PASSED [ 89%] distributed/tests/test_versions.py::test_version_mismatch[worker-1-KEY_ERROR] PASSED [ 89%] distributed/tests/test_versions.py::test_version_mismatch[worker-1-NONE] PASSED [ 89%] distributed/tests/test_versions.py::test_scheduler_mismatched_irrelevant_package PASSED [ 89%] distributed/tests/test_versions.py::test_scheduler_additional_irrelevant_package PASSED [ 89%] distributed/tests/test_versions.py::test_python_mismatch[source] PASSED [ 89%] distributed/tests/test_versions.py::test_python_mismatch[scheduler] PASSED [ 89%] distributed/tests/test_versions.py::test_python_mismatch[worker-1] PASSED [ 89%] distributed/tests/test_versions.py::test_version_warning_in_cluster PASSED [ 89%] distributed/tests/test_versions.py::test_python_version PASSED [ 89%] distributed/tests/test_versions.py::test_version_custom_pkgs PASSED [ 89%] distributed/tests/test_worker.py::test_worker_nthreads PASSED [ 89%] distributed/tests/test_worker.py::test_str PASSED [ 89%] distributed/tests/test_worker.py::test_identity PASSED [ 89%] distributed/tests/test_worker.py::test_worker_bad_args PASSED [ 89%] distributed/tests/test_worker.py::test_upload_file PASSED [ 89%] distributed/tests/test_worker.py::test_upload_file_pyc SKIPPED (don't yet support uploading pyc files) [ 89%] distributed/tests/test_worker.py::test_upload_egg PASSED [ 89%] distributed/tests/test_worker.py::test_upload_pyz PASSED [ 89%] distributed/tests/test_worker.py::test_upload_large_file SKIPPED (could not import 'crick': No module named 'crick') [ 89%] distributed/tests/test_worker.py::test_broadcast PASSED [ 89%] distributed/tests/test_worker.py::test_worker_with_port_zero PASSED [ 89%] distributed/tests/test_worker.py::test_worker_port_range PASSED [ 89%] distributed/tests/test_worker.py::test_worker_task_data PASSED [ 89%] distributed/tests/test_worker.py::test_error_message PASSED [ 89%] distributed/tests/test_worker.py::test_chained_error_message PASSED [ 89%] distributed/tests/test_worker.py::test_plugin_exception PASSED [ 89%] distributed/tests/test_worker.py::test_plugin_multiple_exceptions PASSED [ 89%] distributed/tests/test_worker.py::test_plugin_internal_exception PASSED [ 90%] distributed/tests/test_worker.py::test_gather PASSED [ 90%] distributed/tests/test_worker.py::test_gather_missing_keys PASSED [ 90%] distributed/tests/test_worker.py::test_gather_missing_workers PASSED [ 90%] distributed/tests/test_worker.py::test_io_loop PASSED [ 90%] distributed/tests/test_worker.py::test_access_key PASSED [ 90%] distributed/tests/test_worker.py::test_run_dask_worker PASSED [ 90%] distributed/tests/test_worker.py::test_run_dask_worker_kwonlyarg PASSED [ 90%] distributed/tests/test_worker.py::test_run_coroutine_dask_worker PASSED [ 90%] distributed/tests/test_worker.py::test_Executor PASSED [ 90%] distributed/tests/test_worker.py::test_close_on_disconnect PASSED [ 90%] distributed/tests/test_worker.py::test_memory_limit_auto PASSED [ 90%] distributed/tests/test_worker.py::test_inter_worker_communication PASSED [ 90%] distributed/tests/test_worker.py::test_clean PASSED [ 90%] distributed/tests/test_worker.py::test_message_breakup PASSED [ 90%] distributed/tests/test_worker.py::test_types PASSED [ 90%] distributed/tests/test_worker.py::test_system_monitor PASSED [ 90%] distributed/tests/test_worker.py::test_restrictions PASSED [ 90%] distributed/tests/test_worker.py::test_clean_nbytes PASSED [ 90%] distributed/tests/test_worker.py::test_gather_many_small[True] PASSED [ 90%] distributed/tests/test_worker.py::test_gather_many_small[False] PASSED [ 90%] distributed/tests/test_worker.py::test_multiple_transfers PASSED [ 90%] distributed/tests/test_worker.py::test_share_communication XFAIL (very high flakiness) [ 90%] distributed/tests/test_worker.py::test_dont_overlap_communications_to_same_worker XFAIL [ 90%] distributed/tests/test_worker.py::test_log_exception_on_failed_task PASSED [ 90%] distributed/tests/test_worker.py::test_clean_up_dependencies PASSED [ 90%] distributed/tests/test_worker.py::test_hold_onto_dependents PASSED [ 90%] distributed/tests/test_worker.py::test_worker_death_timeout XPASS [ 90%] distributed/tests/test_worker.py::test_stop_doing_unnecessary_work PASSED [ 90%] distributed/tests/test_worker.py::test_priorities PASSED [ 90%] distributed/tests/test_worker.py::test_heartbeats PASSED [ 90%] distributed/tests/test_worker.py::test_worker_dir[Worker] PASSED [ 90%] distributed/tests/test_worker.py::test_worker_dir[Nanny] PASSED [ 90%] distributed/tests/test_worker.py::test_default_worker_dir PASSED [ 90%] distributed/tests/test_worker.py::test_dataframe_attribute_error PASSED [ 91%] distributed/tests/test_worker.py::test_pid PASSED [ 91%] distributed/tests/test_worker.py::test_get_client PASSED [ 91%] distributed/tests/test_worker.py::test_get_client_sync PASSED [ 91%] distributed/tests/test_worker.py::test_get_client_coroutine PASSED [ 91%] distributed/tests/test_worker.py::test_get_client_coroutine_sync XFAILk/distributed/pull/7339) [ 91%] distributed/tests/test_worker.py::test_global_workers PASSED [ 91%] distributed/tests/test_worker.py::test_worker_fds PASSED [ 91%] distributed/tests/test_worker.py::test_service_hosts_match_worker PASSED [ 91%] distributed/tests/test_worker.py::test_start_services PASSED [ 91%] distributed/tests/test_worker.py::test_scheduler_file PASSED [ 91%] distributed/tests/test_worker.py::test_scheduler_delay PASSED [ 91%] distributed/tests/test_worker.py::test_get_current_task PASSED [ 91%] distributed/tests/test_worker.py::test_deque_handler PASSED [ 91%] distributed/tests/test_worker.py::test_scheduler_address_config PASSED [ 91%] distributed/tests/test_worker.py::test_prefer_gather_from_local_address PASSED [ 91%] distributed/tests/test_worker.py::test_avoid_oversubscription PASSED [ 91%] distributed/tests/test_worker.py::test_custom_metrics PASSED [ 91%] distributed/tests/test_worker.py::test_register_worker_callbacks PASSED [ 91%] distributed/tests/test_worker.py::test_register_worker_callbacks_err PASSED [ 91%] distributed/tests/test_worker.py::test_local_directory PASSED [ 91%] distributed/tests/test_worker.py::test_local_directory_make_new_directory PASSED [ 91%] distributed/tests/test_worker.py::test_host_address PASSED [ 91%] distributed/tests/test_worker.py::test_interface_async[Worker] PASSED [ 91%] distributed/tests/test_worker.py::test_interface_async[Nanny] PASSED [ 91%] distributed/tests/test_worker.py::test_host_uses_scheduler_protocol PASSED [ 91%] distributed/tests/test_worker.py::test_worker_listens_on_same_interface_by_default[Worker] PASSED [ 91%] distributed/tests/test_worker.py::test_worker_listens_on_same_interface_by_default[Nanny] PASSED [ 91%] distributed/tests/test_worker.py::test_close_while_executing[False] PASSED [ 91%] distributed/tests/test_worker.py::test_lifetime_stagger PASSED [ 91%] distributed/tests/test_worker.py::test_bad_metrics PASSED [ 91%] distributed/tests/test_worker.py::test_bad_startup PASSED [ 91%] distributed/tests/test_worker.py::test_update_latency PASSED [ 91%] distributed/tests/test_worker.py::test_shutdown_on_scheduler_comm_closed PASSED [ 91%] distributed/tests/test_worker.py::test_heartbeat_comm_closed PASSED [ 92%] distributed/tests/test_worker.py::test_heartbeat_missing PASSED [ 92%] distributed/tests/test_worker.py::test_heartbeat_missing_real_cluster PASSED [ 92%] distributed/tests/test_worker.py::test_bad_local_directory PASSED [ 92%] distributed/tests/test_worker.py::test_taskstate_metadata PASSED [ 92%] distributed/tests/test_worker.py::test_executor_offload PASSED [ 92%] distributed/tests/test_worker.py::test_story PASSED [ 92%] distributed/tests/test_worker.py::test_stimulus_story PASSED [ 92%] distributed/tests/test_worker.py::test_worker_descopes_data PASSED [ 92%] distributed/tests/test_worker.py::test_gather_dep_one_worker_always_busy PASSED [ 92%] distributed/tests/test_worker.py::test_gather_dep_local_workers_first PASSED [ 92%] distributed/tests/test_worker.py::test_gather_dep_from_remote_workers_if_all_local_workers_are_busy PASSED [ 92%] distributed/tests/test_worker.py::test_worker_client_uses_default_no_close PASSED [ 92%] distributed/tests/test_worker.py::test_worker_client_closes_if_created_on_worker_one_worker PASSED [ 92%] distributed/tests/test_worker.py::test_worker_client_closes_if_created_on_worker_last_worker_alive PASSED [ 92%] distributed/tests/test_worker.py::test_multiple_executors PASSED [ 92%] distributed/tests/test_worker.py::test_executor_inherit_threadname_from_worker PASSED [ 92%] distributed/tests/test_worker.py::test_bad_executor_annotation PASSED [ 92%] distributed/tests/test_worker.py::test_process_executor PASSED [ 92%] distributed/tests/test_worker.py::test_process_executor_kills_process PASSED [ 92%] distributed/tests/test_worker.py::test_process_executor_raise_exception PASSED [ 92%] distributed/tests/test_worker.py::test_worker_state_error_release_error_last PASSED [ 92%] distributed/tests/test_worker.py::test_worker_state_error_release_error_first PASSED [ 92%] distributed/tests/test_worker.py::test_worker_state_error_release_error_int PASSED [ 92%] distributed/tests/test_worker.py::test_worker_state_error_long_chain PASSED [ 92%] distributed/tests/test_worker.py::test_hold_on_to_replicas PASSED [ 92%] distributed/tests/test_worker.py::test_forget_dependents_after_release PASSED [ 92%] distributed/tests/test_worker.py::test_acquire_replicas PASSED [ 92%] distributed/tests/test_worker.py::test_acquire_replicas_same_channel PASSED [ 92%] distributed/tests/test_worker.py::test_acquire_replicas_many PASSED [ 92%] distributed/tests/test_worker.py::test_acquire_replicas_already_in_flight PASSED [ 92%] distributed/tests/test_worker.py::test_remove_replicas_simple PASSED [ 92%] distributed/tests/test_worker.py::test_who_has_consistent_remove_replicas PASSED [ 92%] distributed/tests/test_worker.py::test_acquire_replicas_with_no_priority PASSED [ 92%] distributed/tests/test_worker.py::test_acquire_replicas_large_data PASSED [ 93%] distributed/tests/test_worker.py::test_missing_released_zombie_tasks PASSED [ 93%] distributed/tests/test_worker.py::test_missing_released_zombie_tasks_2 PASSED [ 93%] distributed/tests/test_worker.py::test_worker_status_sync PASSED [ 93%] distributed/tests/test_worker.py::test_log_remove_worker PASSED [ 93%] distributed/tests/test_worker.py::test_task_flight_compute_oserror PASSED [ 93%] distributed/tests/test_worker.py::test_gather_dep_cancelled_rescheduled PASSED [ 93%] distributed/tests/test_worker.py::test_gather_dep_do_not_handle_response_of_not_requested_tasks PASSED [ 93%] distributed/tests/test_worker.py::test_gather_dep_no_longer_in_flight_tasks PASSED [ 93%] distributed/tests/test_worker.py::test_get_data_cancelled_error PASSED [ 93%] distributed/tests/test_worker.py::test_Worker__to_dict PASSED [ 93%] distributed/tests/test_worker.py::test_extension_methods PASSED [ 93%] distributed/tests/test_worker.py::test_benchmark_hardware PASSED [ 93%] distributed/tests/test_worker.py::test_do_not_block_event_loop_during_shutdown PASSED [ 93%] distributed/tests/test_worker.py::test_worker_running_before_running_plugins PASSED [ 93%] distributed/tests/test_worker.py::test_execute_preamble_abort_retirement PASSED [ 93%] distributed/tests/test_worker.py::test_forward_output PASSED [ 93%] distributed/tests/test_worker.py::test_offload_getdata PASSED [ 93%] distributed/tests/test_worker.py::test_startstops PASSED [ 93%] distributed/tests/test_worker.py::test_suppress_keyerror_for_cancelled_tasks[cancelled] PASSED [ 93%] distributed/tests/test_worker.py::test_suppress_keyerror_for_cancelled_tasks[resumed] PASSED [ 93%] distributed/tests/test_worker.py::test_suppress_compute_failure_for_cancelled_tasks PASSED [ 93%] distributed/tests/test_worker_client.py::test_submit_from_worker PASSED [ 93%] distributed/tests/test_worker_client.py::test_submit_from_worker_async PASSED [ 93%] distributed/tests/test_worker_client.py::test_scatter_from_worker PASSED [ 93%] distributed/tests/test_worker_client.py::test_scatter_singleton PASSED [ 93%] distributed/tests/test_worker_client.py::test_gather_multi_machine PASSED [ 93%] distributed/tests/test_worker_client.py::test_same_loop PASSED [ 93%] distributed/tests/test_worker_client.py::test_sync PASSED [ 93%] distributed/tests/test_worker_client.py::test_async PASSED [ 93%] distributed/tests/test_worker_client.py::test_nested_worker_clients PASSED [ 93%] distributed/tests/test_worker_client.py::test_separate_thread_false PASSED [ 93%] distributed/tests/test_worker_client.py::test_client_executor PASSED [ 93%] distributed/tests/test_worker_client.py::test_dont_override_default_get PASSED [ 93%] distributed/tests/test_worker_client.py::test_local_client_warning PASSED [ 94%] distributed/tests/test_worker_client.py::test_closing_worker_doesnt_close_client PASSED [ 94%] distributed/tests/test_worker_client.py::test_timeout PASSED [ 94%] distributed/tests/test_worker_client.py::test_secede_without_stealing_issue_1262 PASSED [ 94%] distributed/tests/test_worker_client.py::test_compute_within_worker_client PASSED [ 94%] distributed/tests/test_worker_client.py::test_worker_client_rejoins PASSED [ 94%] distributed/tests/test_worker_client.py::test_submit_different_names XPASS [ 94%] distributed/tests/test_worker_client.py::test_secede_does_not_claim_worker PASSED [ 94%] distributed/tests/test_worker_client.py::test_log_event PASSED [ 94%] distributed/tests/test_worker_client.py::test_log_event_implicit PASSED [ 94%] distributed/tests/test_worker_memory.py::test_parse_memory_limit_zero PASSED [ 94%] distributed/tests/test_worker_memory.py::test_resource_limit PASSED [ 94%] distributed/tests/test_worker_memory.py::test_parse_memory_limit_worker PASSED [ 94%] distributed/tests/test_worker_memory.py::test_parse_memory_limit_worker_relative PASSED [ 94%] distributed/tests/test_worker_memory.py::test_parse_memory_limit_nanny PASSED [ 94%] distributed/tests/test_worker_memory.py::test_dict_data_if_no_spill_to_disk PASSED [ 94%] distributed/tests/test_worker_memory.py::test_worker_data_callable_local_directory PASSED [ 94%] distributed/tests/test_worker_memory.py::test_worker_data_callable_local_directory_kwargs PASSED [ 94%] distributed/tests/test_worker_memory.py::test_worker_data_callable_kwargs PASSED [ 94%] distributed/tests/test_worker_memory.py::test_workerstate_fail_to_pickle_execute_1[executing] PASSED [ 94%] distributed/tests/test_worker_memory.py::test_workerstate_fail_to_pickle_execute_1[long-running] PASSED [ 94%] distributed/tests/test_worker_memory.py::test_workerstate_fail_to_pickle_flight PASSED [ 94%] distributed/tests/test_worker_memory.py::test_fail_to_pickle_execute_2 PASSED [ 94%] distributed/tests/test_worker_memory.py::test_fail_to_pickle_spill PASSED [ 94%] distributed/tests/test_worker_memory.py::test_spill_target_threshold PASSED [ 94%] distributed/tests/test_worker_memory.py::test_spill_constrained PASSED [ 94%] distributed/tests/test_worker_memory.py::test_spill_spill_threshold PASSED [ 94%] distributed/tests/test_worker_memory.py::test_spill_hysteresis[False-10000000000-1] PASSED [ 94%] distributed/tests/test_worker_memory.py::test_spill_hysteresis[0.7-0-1] PASSED [ 94%] distributed/tests/test_worker_memory.py::test_spill_hysteresis[0.4-0-7] PASSED [ 94%] distributed/tests/test_worker_memory.py::test_pause_executor_manual PASSED [ 94%] distributed/tests/test_worker_memory.py::test_pause_executor_with_memory_monitor PASSED [ 94%] distributed/tests/test_worker_memory.py::test_pause_prevents_deps_fetch PASSED [ 94%] distributed/tests/test_worker_memory.py::test_avoid_memory_monitor_if_zero_limit_worker PASSED [ 94%] distributed/tests/test_worker_memory.py::test_avoid_memory_monitor_if_zero_limit_nanny PASSED [ 95%] distributed/tests/test_worker_memory.py::test_override_data_worker PASSED [ 95%] distributed/tests/test_worker_memory.py::test_override_data_nanny PASSED [ 95%] distributed/tests/test_worker_memory.py::test_override_data_vs_memory_monitor PASSED [ 95%] distributed/tests/test_worker_memory.py::test_manual_evict_proto PASSED [ 95%] distributed/tests/test_worker_memory.py::test_pause_while_spilling PASSED [ 95%] distributed/tests/test_worker_memory.py::test_pause_while_idle PASSED [ 95%] distributed/tests/test_worker_memory.py::test_pause_while_saturated PASSED [ 95%] distributed/tests/test_worker_memory.py::test_worker_log_memory_limit_too_high PASSED [ 95%] distributed/tests/test_worker_memory.py::test_high_unmanaged_memory_warning PASSED [ 95%] distributed/tests/test_worker_memory.py::test_delete_spilled_keys PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_task_lifecycle PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_async_task PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_custom_executor PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_offload PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_execute_failed PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_cancelled_execute PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_cancelled_flight PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_gather_dep_busy PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_gather_dep_no_task PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_gather_dep_failed PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_gather_dep_network_error PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_memory_monitor PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_user_metrics_sync PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_user_metrics_async PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_user_metrics_fail PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_user_metrics_weird PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_do_not_leak_metrics PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_reschedule PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_send_metrics_to_scheduler PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_no_spans_extension PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_new_metrics_during_heartbeat PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_delayed_ledger_is_not_reentrant PASSED [ 95%] distributed/tests/test_worker_metrics.py::test_int_metrics PASSED [ 95%] distributed/tests/test_worker_state_machine.py::test_instruction_match PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_TaskState_tracking PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_TaskState_get_nbytes PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_TaskState_eq PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_TaskState__to_dict PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_TaskState_repr PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_WorkerState__to_dict PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_WorkerState_pickle PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_pickle_exceptions[False-InvalidTransition-kwargs0] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_pickle_exceptions[False-TransitionCounterMaxExceeded-kwargs1] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_pickle_exceptions[False-InvalidTaskState-kwargs2] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_pickle_exceptions[True-InvalidTransition-kwargs0] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_pickle_exceptions[True-TransitionCounterMaxExceeded-kwargs1] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_pickle_exceptions[True-InvalidTaskState-kwargs2] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[TaskState] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[Instruction] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[GatherDep] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[Execute] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[RetryBusyWorkerLater] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[SendMessageToScheduler] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[TaskFinishedMsg] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[TaskErredMsg] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[ReleaseWorkerDataMsg] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[RescheduleMsg] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[LongRunningMsg] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[AddKeysMsg] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[RequestRefreshWhoHasMsg] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[StealResponseMsg] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[StateMachineEvent] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[PauseEvent] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[UnpauseEvent] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[RetryBusyWorkerEvent] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[GatherDepDoneEvent] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[GatherDepSuccessEvent] PASSED [ 96%] distributed/tests/test_worker_state_machine.py::test_slots[GatherDepBusyEvent] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_slots[GatherDepNetworkFailureEvent] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_slots[GatherDepFailureEvent] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_slots[RemoveWorkerEvent] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_slots[ComputeTaskEvent] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_slots[ExecuteDoneEvent] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_slots[ExecuteSuccessEvent] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_slots[ExecuteFailureEvent] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_slots[RescheduleEvent] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_slots[CancelComputeEvent] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_slots[FindMissingEvent] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_slots[RefreshWhoHasEvent] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_slots[AcquireReplicasEvent] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_slots[RemoveReplicasEvent] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_slots[FreeKeysEvent] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_slots[StealRequestEvent] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_slots[UpdateDataEvent] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_slots[SecedeEvent] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_sendmsg_to_dict PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_merge_recs_instructions PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_event_to_dict_with_annotations PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_event_to_dict_without_annotations PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_computetask_to_dict PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_computetask_dummy PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_updatedata_to_dict PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_executesuccess_to_dict PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_executesuccess_dummy PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_executefailure_to_dict PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_executefailure_dummy PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_fetch_to_compute PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_fetch_via_amm_to_compute PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_lose_replica_during_fetch[False] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_lose_replica_during_fetch[True] PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_fetch_to_missing_on_busy PASSED [ 97%] distributed/tests/test_worker_state_machine.py::test_new_replica_while_all_workers_in_flight PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_cancelled_while_in_flight PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_in_memory_while_in_flight PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_forget_data_needed PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_missing_handle_compute_dependency PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_missing_to_waiting PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_task_state_instance_are_garbage_collected PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_fetch_to_missing_on_refresh_who_has PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_fetch_to_missing_on_network_failure PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_aggregate_gather_deps[10000000-3-1] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_aggregate_gather_deps[10000000-3-2] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_aggregate_gather_deps[20000000-2-1] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_aggregate_gather_deps[20000000-2-2] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_aggregate_gather_deps[30000000-1-1] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_aggregate_gather_deps[30000000-1-2] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_aggregate_gather_deps[60000000-1-1] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_aggregate_gather_deps[60000000-1-2] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_gather_priority PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_task_acquires_resources[executing] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_task_acquires_resources[long-running] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_task_releases_resources[executing-ExecuteSuccessEvent] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_task_releases_resources[executing-ExecuteFailureEvent] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_task_releases_resources[executing-RescheduleEvent] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_task_releases_resources[long-running-ExecuteSuccessEvent] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_task_releases_resources[long-running-ExecuteFailureEvent] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_task_releases_resources[long-running-RescheduleEvent] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_task_with_dependencies_acquires_resources PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_resumed_task_releases_resources[executing-ExecuteSuccessEvent] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_resumed_task_releases_resources[executing-ExecuteFailureEvent] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_resumed_task_releases_resources[executing-RescheduleEvent] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_resumed_task_releases_resources[long-running-ExecuteSuccessEvent] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_resumed_task_releases_resources[long-running-ExecuteFailureEvent] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_resumed_task_releases_resources[long-running-RescheduleEvent] PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_clean_log PASSED [ 98%] distributed/tests/test_worker_state_machine.py::test_running_task_in_all_running_tasks[executing] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_running_task_in_all_running_tasks[long-running] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_done_task_not_in_all_running_tasks[executing-ExecuteSuccessEvent] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_done_task_not_in_all_running_tasks[executing-ExecuteFailureEvent] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_done_task_not_in_all_running_tasks[executing-RescheduleEvent] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_done_task_not_in_all_running_tasks[long-running-ExecuteSuccessEvent] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_done_task_not_in_all_running_tasks[long-running-ExecuteFailureEvent] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_done_task_not_in_all_running_tasks[long-running-RescheduleEvent] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_done_resumed_task_not_in_all_running_tasks[executing-ExecuteSuccessEvent] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_done_resumed_task_not_in_all_running_tasks[executing-ExecuteFailureEvent] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_done_resumed_task_not_in_all_running_tasks[executing-RescheduleEvent] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_done_resumed_task_not_in_all_running_tasks[long-running-ExecuteSuccessEvent] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_done_resumed_task_not_in_all_running_tasks[long-running-ExecuteFailureEvent] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_done_resumed_task_not_in_all_running_tasks[long-running-RescheduleEvent] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_gather_dep_failure PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_recompute_erred_task PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_transfer_incoming_metrics PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_throttling_does_not_affect_first_transfer PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_message_target_does_not_affect_first_transfer_on_different_worker PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_throttle_incoming_transfers_on_count_limit PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_throttling_incoming_transfer_on_transfer_bytes_same_worker PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_throttling_incoming_transfer_on_transfer_bytes_different_workers PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_do_not_throttle_connections_while_below_threshold PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_throttle_on_transfer_bytes_regardless_of_threshold PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_worker_nbytes[executing] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_worker_nbytes[long-running] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_fetch_count PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_task_counter PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_remove_worker_while_in_flight[fail] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_remove_worker_while_in_flight[success] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_remove_worker_while_in_flight[no-key] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_remove_worker_while_in_flight[busy] PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_remove_worker_while_in_flight_unused_peer PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_remove_worker_while_in_fetch PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_remove_worker_unknown PASSED [100%] =================================== FAILURES =================================== ________________________________ test_defaults _________________________________ self = address = ('fe80::9e69:b4ff:fe64:abd4', 8786, 0, 0), server_hostname = None def connect( self: _IOStreamType, address: Any, server_hostname: Optional[str] = None ) -> "Future[_IOStreamType]": """Connects the socket to a remote address without blocking. May only be called if the socket passed to the constructor was not previously connected. The address parameter is in the same format as for `socket.connect ` for the type of socket passed to the IOStream constructor, e.g. an ``(ip, port)`` tuple. Hostnames are accepted here, but will be resolved synchronously and block the IOLoop. If you have a hostname instead of an IP address, the `.TCPClient` class is recommended instead of calling this method directly. `.TCPClient` will do asynchronous DNS resolution and handle both IPv4 and IPv6. If ``callback`` is specified, it will be called with no arguments when the connection is completed; if not this method returns a `.Future` (whose result after a successful connection will be the stream itself). In SSL mode, the ``server_hostname`` parameter will be used for certificate validation (unless disabled in the ``ssl_options``) and SNI. Note that it is safe to call `IOStream.write ` while the connection is pending, in which case the data will be written as soon as the connection is ready. Calling `IOStream` read methods before the socket is connected works on some platforms but is non-portable. .. versionchanged:: 4.0 If no callback is given, returns a `.Future`. .. versionchanged:: 4.2 SSL certificates are validated by default; pass ``ssl_options=dict(cert_reqs=ssl.CERT_NONE)`` or a suitably-configured `ssl.SSLContext` to the `SSLIOStream` constructor to disable. .. versionchanged:: 6.0 The ``callback`` argument was removed. Use the returned `.Future` instead. """ self._connecting = True future = Future() # type: Future[_IOStreamType] self._connect_future = typing.cast("Future[IOStream]", future) try: > self.socket.connect(address) E OSError: [Errno 22] Invalid argument /usr/lib/python3.14/site-packages/tornado/iostream.py:1177: OSError The above exception was the direct cause of the following exception: addr = 'tcp://[fe80::9e69:b4ff:fe64:abd4]:8786', timeout = 5, deserialize = True handshake_overrides = None, connection_args = {} async def connect( addr, timeout=None, deserialize=True, handshake_overrides=None, **connection_args ): """ Connect to the given address (a URI such as ``tcp://127.0.0.1:1234``) and yield a ``Comm`` object. If the connection attempt fails, it is retried until the *timeout* is expired. """ if timeout is None: timeout = dask.config.get("distributed.comm.timeouts.connect") timeout = parse_timedelta(timeout, default="seconds") scheme, loc = parse_address(addr) backend = registry.get_backend(scheme) connector = backend.get_connector() comm = None start = time() def time_left(): deadline = start + timeout return max(0, deadline - time()) backoff_base = 0.01 attempt = 0 logger.debug("Establishing connection to %s", loc) # Prefer multiple small attempts than one long attempt. This should protect # primarily from DNS race conditions # gh3104, gh4176, gh4167 intermediate_cap = timeout / 5 active_exception = None while time_left() > 0: try: > comm = await wait_for( connector.connect(loc, deserialize=deserialize, **connection_args), timeout=min(intermediate_cap, time_left()), ) distributed/comm/core.py:342: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ distributed/utils.py:1855: in wait_for return await fut ^^^^^^^^^ distributed/comm/tcp.py:560: in connect convert_stream_closed_error(self, e) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ obj = exc = OSError(22, 'Invalid argument') def convert_stream_closed_error(obj, exc): """ Re-raise StreamClosedError or SSLError as CommClosedError. """ if hasattr(exc, "real_error"): # The stream was closed because of an underlying OS error if exc.real_error is None: raise CommClosedError(f"in {obj}: {exc}") from exc exc = exc.real_error if isinstance(exc, ssl.SSLError): if exc.reason and "UNKNOWN_CA" in exc.reason: raise FatalCommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") > raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc E distributed.comm.core.CommClosedError: in : OSError: [Errno 22] Invalid argument distributed/comm/tcp.py:143: CommClosedError The above exception was the direct cause of the following exception: loop = requires_default_ports = None def test_defaults(loop, requires_default_ports): with popen([sys.executable, "-m", "dask", "scheduler"]): async def f(): # Default behaviour is to listen on all addresses await assert_can_connect_from_everywhere_4_6(8786, timeout=5.0) with Client(f"127.0.0.1:{Scheduler.default_port}", loop=loop) as c: > c.sync(f) distributed/cli/tests/test_dask_scheduler.py:41: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ distributed/utils.py:384: in sync return sync( distributed/utils.py:460: in sync raise error distributed/utils.py:434: in f result = yield future ^^^^^^^^^^^^ /usr/lib/python3.14/site-packages/tornado/gen.py:783: in run value = future.result() ^^^^^^^^^^^^^^^ distributed/cli/tests/test_dask_scheduler.py:38: in f await assert_can_connect_from_everywhere_4_6(8786, timeout=5.0) distributed/utils_test.py:1305: in assert_can_connect_from_everywhere_4_6 await asyncio.gather(*futures) distributed/utils_test.py:1276: in assert_can_connect comm = await connect(addr, timeout=timeout, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ addr = 'tcp://[fe80::9e69:b4ff:fe64:abd4]:8786', timeout = 5, deserialize = True handshake_overrides = None, connection_args = {} async def connect( addr, timeout=None, deserialize=True, handshake_overrides=None, **connection_args ): """ Connect to the given address (a URI such as ``tcp://127.0.0.1:1234``) and yield a ``Comm`` object. If the connection attempt fails, it is retried until the *timeout* is expired. """ if timeout is None: timeout = dask.config.get("distributed.comm.timeouts.connect") timeout = parse_timedelta(timeout, default="seconds") scheme, loc = parse_address(addr) backend = registry.get_backend(scheme) connector = backend.get_connector() comm = None start = time() def time_left(): deadline = start + timeout return max(0, deadline - time()) backoff_base = 0.01 attempt = 0 logger.debug("Establishing connection to %s", loc) # Prefer multiple small attempts than one long attempt. This should protect # primarily from DNS race conditions # gh3104, gh4176, gh4167 intermediate_cap = timeout / 5 active_exception = None while time_left() > 0: try: comm = await wait_for( connector.connect(loc, deserialize=deserialize, **connection_args), timeout=min(intermediate_cap, time_left()), ) break except FatalCommClosedError: raise # Note: CommClosed inherits from OSError except (asyncio.TimeoutError, OSError) as exc: active_exception = exc # As described above, the intermediate timeout is used to distribute # initial, bulk connect attempts homogeneously. In particular with # the jitter upon retries we should not be worried about overloading # any more DNS servers intermediate_cap = timeout # FullJitter see https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ upper_cap = min(time_left(), backoff_base * (2**attempt)) backoff = random.uniform(0, upper_cap) attempt += 1 logger.debug( "Could not connect to %s, waiting for %s before retrying", loc, backoff ) await asyncio.sleep(backoff) else: > raise OSError( f"Timed out trying to connect to {addr} after {timeout} s" ) from active_exception E OSError: Timed out trying to connect to tcp://[fe80::9e69:b4ff:fe64:abd4]:8786 after 5 s distributed/comm/core.py:368: OSError ----------------------------- Captured stderr call ----------------------------- 2026-09-08 21:52:36,635 - distributed.scheduler - INFO - ----------------------------------------------- 2026-09-08 21:52:36,987 - distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy 2026-09-08 21:52:37,032 - distributed.scheduler - INFO - State start 2026-09-08 21:52:37,127 - distributed.scheduler - INFO - ----------------------------------------------- 2026-09-08 21:52:37,147 - distributed.scheduler - INFO - Scheduler at: tcp://172.24.9.22:8786 2026-09-08 21:52:37,153 - distributed.scheduler - INFO - dashboard at: http://172.24.9.22:8787/status 2026-09-08 21:52:37,160 - distributed.scheduler - INFO - Registering Worker plugin shuffle 2026-09-08 21:52:44,414 - distributed.scheduler - INFO - Receive client connection: Client-964919a2-abcf-11f1-86b4-8f07734cb81d 2026-09-08 21:52:44,556 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45678 2026-09-08 21:52:49,669 - distributed.scheduler - INFO - Remove client Client-964919a2-abcf-11f1-86b4-8f07734cb81d 2026-09-08 21:52:49,678 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:45678; closing. 2026-09-08 21:52:49,697 - distributed.scheduler - INFO - Remove client Client-964919a2-abcf-11f1-86b4-8f07734cb81d 2026-09-08 21:52:49,709 - distributed.scheduler - INFO - Close client connection: Client-964919a2-abcf-11f1-86b4-8f07734cb81d 2026-09-08 21:52:49,714 - distributed._signals - INFO - Received signal SIGINT (2) 2026-09-08 21:52:49,716 - distributed.scheduler - INFO - Closing scheduler. Reason: signal-2 2026-09-08 21:52:49,727 - distributed.scheduler - INFO - Scheduler closing all comms 2026-09-08 21:52:49,743 - distributed.scheduler - INFO - Stopped scheduler at 'tcp://172.24.9.22:8786' 2026-09-08 21:52:49,753 - distributed.scheduler - INFO - End scheduler ___________________ test_default_client_server_ipv6[tornado] ___________________ self = address = ('fe80::9e69:b4ff:fe64:abd4', 44377, 0, 0), server_hostname = None def connect( self: _IOStreamType, address: Any, server_hostname: Optional[str] = None ) -> "Future[_IOStreamType]": """Connects the socket to a remote address without blocking. May only be called if the socket passed to the constructor was not previously connected. The address parameter is in the same format as for `socket.connect ` for the type of socket passed to the IOStream constructor, e.g. an ``(ip, port)`` tuple. Hostnames are accepted here, but will be resolved synchronously and block the IOLoop. If you have a hostname instead of an IP address, the `.TCPClient` class is recommended instead of calling this method directly. `.TCPClient` will do asynchronous DNS resolution and handle both IPv4 and IPv6. If ``callback`` is specified, it will be called with no arguments when the connection is completed; if not this method returns a `.Future` (whose result after a successful connection will be the stream itself). In SSL mode, the ``server_hostname`` parameter will be used for certificate validation (unless disabled in the ``ssl_options``) and SNI. Note that it is safe to call `IOStream.write ` while the connection is pending, in which case the data will be written as soon as the connection is ready. Calling `IOStream` read methods before the socket is connected works on some platforms but is non-portable. .. versionchanged:: 4.0 If no callback is given, returns a `.Future`. .. versionchanged:: 4.2 SSL certificates are validated by default; pass ``ssl_options=dict(cert_reqs=ssl.CERT_NONE)`` or a suitably-configured `ssl.SSLContext` to the `SSLIOStream` constructor to disable. .. versionchanged:: 6.0 The ``callback`` argument was removed. Use the returned `.Future` instead. """ self._connecting = True future = Future() # type: Future[_IOStreamType] self._connect_future = typing.cast("Future[IOStream]", future) try: > self.socket.connect(address) E OSError: [Errno 22] Invalid argument /usr/lib/python3.14/site-packages/tornado/iostream.py:1177: OSError The above exception was the direct cause of the following exception: addr = 'tcp://[fe80::9e69:b4ff:fe64:abd4]:44377', timeout = 30 deserialize = True, handshake_overrides = None, connection_args = {'xxx': 'foo'} async def connect( addr, timeout=None, deserialize=True, handshake_overrides=None, **connection_args ): """ Connect to the given address (a URI such as ``tcp://127.0.0.1:1234``) and yield a ``Comm`` object. If the connection attempt fails, it is retried until the *timeout* is expired. """ if timeout is None: timeout = dask.config.get("distributed.comm.timeouts.connect") timeout = parse_timedelta(timeout, default="seconds") scheme, loc = parse_address(addr) backend = registry.get_backend(scheme) connector = backend.get_connector() comm = None start = time() def time_left(): deadline = start + timeout return max(0, deadline - time()) backoff_base = 0.01 attempt = 0 logger.debug("Establishing connection to %s", loc) # Prefer multiple small attempts than one long attempt. This should protect # primarily from DNS race conditions # gh3104, gh4176, gh4167 intermediate_cap = timeout / 5 active_exception = None while time_left() > 0: try: > comm = await wait_for( connector.connect(loc, deserialize=deserialize, **connection_args), timeout=min(intermediate_cap, time_left()), ) distributed/comm/core.py:342: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ distributed/utils.py:1855: in wait_for return await fut ^^^^^^^^^ distributed/comm/tcp.py:560: in connect convert_stream_closed_error(self, e) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ obj = exc = OSError(22, 'Invalid argument') def convert_stream_closed_error(obj, exc): """ Re-raise StreamClosedError or SSLError as CommClosedError. """ if hasattr(exc, "real_error"): # The stream was closed because of an underlying OS error if exc.real_error is None: raise CommClosedError(f"in {obj}: {exc}") from exc exc = exc.real_error if isinstance(exc, ssl.SSLError): if exc.reason and "UNKNOWN_CA" in exc.reason: raise FatalCommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") > raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc E distributed.comm.core.CommClosedError: in : OSError: [Errno 22] Invalid argument distributed/comm/tcp.py:143: CommClosedError The above exception was the direct cause of the following exception: tcp = @requires_ipv6 @gen_test() async def test_default_client_server_ipv6(tcp): external_ip6 = get_ipv6() await check_client_server("[::1]", tcp_eq("::1")) await check_client_server("[::1]:3211", tcp_eq("::1", 3211)) > await check_client_server("[::]", tcp_eq("::"), tcp_eq(external_ip6)) distributed/comm/tests/test_comms.py:622: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ distributed/comm/tests/test_comms.py:563: in check_client_server await client_communicate(key=1234) distributed/comm/tests/test_comms.py:549: in client_communicate comm = await connect(listener.contact_address, **connect_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ addr = 'tcp://[fe80::9e69:b4ff:fe64:abd4]:44377', timeout = 30 deserialize = True, handshake_overrides = None, connection_args = {'xxx': 'foo'} async def connect( addr, timeout=None, deserialize=True, handshake_overrides=None, **connection_args ): """ Connect to the given address (a URI such as ``tcp://127.0.0.1:1234``) and yield a ``Comm`` object. If the connection attempt fails, it is retried until the *timeout* is expired. """ if timeout is None: timeout = dask.config.get("distributed.comm.timeouts.connect") timeout = parse_timedelta(timeout, default="seconds") scheme, loc = parse_address(addr) backend = registry.get_backend(scheme) connector = backend.get_connector() comm = None start = time() def time_left(): deadline = start + timeout return max(0, deadline - time()) backoff_base = 0.01 attempt = 0 logger.debug("Establishing connection to %s", loc) # Prefer multiple small attempts than one long attempt. This should protect # primarily from DNS race conditions # gh3104, gh4176, gh4167 intermediate_cap = timeout / 5 active_exception = None while time_left() > 0: try: comm = await wait_for( connector.connect(loc, deserialize=deserialize, **connection_args), timeout=min(intermediate_cap, time_left()), ) break except FatalCommClosedError: raise # Note: CommClosed inherits from OSError except (asyncio.TimeoutError, OSError) as exc: active_exception = exc # As described above, the intermediate timeout is used to distribute # initial, bulk connect attempts homogeneously. In particular with # the jitter upon retries we should not be worried about overloading # any more DNS servers intermediate_cap = timeout # FullJitter see https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ upper_cap = min(time_left(), backoff_base * (2**attempt)) backoff = random.uniform(0, upper_cap) attempt += 1 logger.debug( "Could not connect to %s, waiting for %s before retrying", loc, backoff ) await asyncio.sleep(backoff) else: > raise OSError( f"Timed out trying to connect to {addr} after {timeout} s" ) from active_exception E OSError: Timed out trying to connect to tcp://[fe80::9e69:b4ff:fe64:abd4]:44377 after 30 s distributed/comm/core.py:368: OSError _____________________ test_tcp_client_server_ipv6[tornado] _____________________ self = address = ('fe80::9e69:b4ff:fe64:abd4', 33375, 0, 0), server_hostname = None def connect( self: _IOStreamType, address: Any, server_hostname: Optional[str] = None ) -> "Future[_IOStreamType]": """Connects the socket to a remote address without blocking. May only be called if the socket passed to the constructor was not previously connected. The address parameter is in the same format as for `socket.connect ` for the type of socket passed to the IOStream constructor, e.g. an ``(ip, port)`` tuple. Hostnames are accepted here, but will be resolved synchronously and block the IOLoop. If you have a hostname instead of an IP address, the `.TCPClient` class is recommended instead of calling this method directly. `.TCPClient` will do asynchronous DNS resolution and handle both IPv4 and IPv6. If ``callback`` is specified, it will be called with no arguments when the connection is completed; if not this method returns a `.Future` (whose result after a successful connection will be the stream itself). In SSL mode, the ``server_hostname`` parameter will be used for certificate validation (unless disabled in the ``ssl_options``) and SNI. Note that it is safe to call `IOStream.write ` while the connection is pending, in which case the data will be written as soon as the connection is ready. Calling `IOStream` read methods before the socket is connected works on some platforms but is non-portable. .. versionchanged:: 4.0 If no callback is given, returns a `.Future`. .. versionchanged:: 4.2 SSL certificates are validated by default; pass ``ssl_options=dict(cert_reqs=ssl.CERT_NONE)`` or a suitably-configured `ssl.SSLContext` to the `SSLIOStream` constructor to disable. .. versionchanged:: 6.0 The ``callback`` argument was removed. Use the returned `.Future` instead. """ self._connecting = True future = Future() # type: Future[_IOStreamType] self._connect_future = typing.cast("Future[IOStream]", future) try: > self.socket.connect(address) E OSError: [Errno 22] Invalid argument /usr/lib/python3.14/site-packages/tornado/iostream.py:1177: OSError The above exception was the direct cause of the following exception: addr = 'tcp://[fe80::9e69:b4ff:fe64:abd4]:33375', timeout = 30 deserialize = True, handshake_overrides = None, connection_args = {'xxx': 'foo'} async def connect( addr, timeout=None, deserialize=True, handshake_overrides=None, **connection_args ): """ Connect to the given address (a URI such as ``tcp://127.0.0.1:1234``) and yield a ``Comm`` object. If the connection attempt fails, it is retried until the *timeout* is expired. """ if timeout is None: timeout = dask.config.get("distributed.comm.timeouts.connect") timeout = parse_timedelta(timeout, default="seconds") scheme, loc = parse_address(addr) backend = registry.get_backend(scheme) connector = backend.get_connector() comm = None start = time() def time_left(): deadline = start + timeout return max(0, deadline - time()) backoff_base = 0.01 attempt = 0 logger.debug("Establishing connection to %s", loc) # Prefer multiple small attempts than one long attempt. This should protect # primarily from DNS race conditions # gh3104, gh4176, gh4167 intermediate_cap = timeout / 5 active_exception = None while time_left() > 0: try: > comm = await wait_for( connector.connect(loc, deserialize=deserialize, **connection_args), timeout=min(intermediate_cap, time_left()), ) distributed/comm/core.py:342: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ distributed/utils.py:1855: in wait_for return await fut ^^^^^^^^^ distributed/comm/tcp.py:560: in connect convert_stream_closed_error(self, e) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ obj = exc = OSError(22, 'Invalid argument') def convert_stream_closed_error(obj, exc): """ Re-raise StreamClosedError or SSLError as CommClosedError. """ if hasattr(exc, "real_error"): # The stream was closed because of an underlying OS error if exc.real_error is None: raise CommClosedError(f"in {obj}: {exc}") from exc exc = exc.real_error if isinstance(exc, ssl.SSLError): if exc.reason and "UNKNOWN_CA" in exc.reason: raise FatalCommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") > raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc E distributed.comm.core.CommClosedError: in : OSError: [Errno 22] Invalid argument distributed/comm/tcp.py:143: CommClosedError The above exception was the direct cause of the following exception: tcp = @requires_ipv6 @gen_test() async def test_tcp_client_server_ipv6(tcp): external_ip6 = get_ipv6() await check_client_server("tcp://[::1]", tcp_eq("::1")) await check_client_server("tcp://[::1]:3231", tcp_eq("::1", 3231)) > await check_client_server("tcp://[::]", tcp_eq("::"), tcp_eq(external_ip6)) distributed/comm/tests/test_comms.py:648: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ distributed/comm/tests/test_comms.py:563: in check_client_server await client_communicate(key=1234) distributed/comm/tests/test_comms.py:549: in client_communicate comm = await connect(listener.contact_address, **connect_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ addr = 'tcp://[fe80::9e69:b4ff:fe64:abd4]:33375', timeout = 30 deserialize = True, handshake_overrides = None, connection_args = {'xxx': 'foo'} async def connect( addr, timeout=None, deserialize=True, handshake_overrides=None, **connection_args ): """ Connect to the given address (a URI such as ``tcp://127.0.0.1:1234``) and yield a ``Comm`` object. If the connection attempt fails, it is retried until the *timeout* is expired. """ if timeout is None: timeout = dask.config.get("distributed.comm.timeouts.connect") timeout = parse_timedelta(timeout, default="seconds") scheme, loc = parse_address(addr) backend = registry.get_backend(scheme) connector = backend.get_connector() comm = None start = time() def time_left(): deadline = start + timeout return max(0, deadline - time()) backoff_base = 0.01 attempt = 0 logger.debug("Establishing connection to %s", loc) # Prefer multiple small attempts than one long attempt. This should protect # primarily from DNS race conditions # gh3104, gh4176, gh4167 intermediate_cap = timeout / 5 active_exception = None while time_left() > 0: try: comm = await wait_for( connector.connect(loc, deserialize=deserialize, **connection_args), timeout=min(intermediate_cap, time_left()), ) break except FatalCommClosedError: raise # Note: CommClosed inherits from OSError except (asyncio.TimeoutError, OSError) as exc: active_exception = exc # As described above, the intermediate timeout is used to distribute # initial, bulk connect attempts homogeneously. In particular with # the jitter upon retries we should not be worried about overloading # any more DNS servers intermediate_cap = timeout # FullJitter see https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ upper_cap = min(time_left(), backoff_base * (2**attempt)) backoff = random.uniform(0, upper_cap) attempt += 1 logger.debug( "Could not connect to %s, waiting for %s before retrying", loc, backoff ) await asyncio.sleep(backoff) else: > raise OSError( f"Timed out trying to connect to {addr} after {timeout} s" ) from active_exception E OSError: Timed out trying to connect to tcp://[fe80::9e69:b4ff:fe64:abd4]:33375 after 30 s distributed/comm/core.py:368: OSError _____________________________ test_silent_startup ______________________________ def test_silent_startup(): code = """if 1: from time import sleep from distributed import LocalCluster if __name__ == "__main__": with LocalCluster(n_workers=1, dashboard_address=":0"): sleep(.1) """ out = subprocess.check_output( [sys.executable, "-Wi", "-c", code], stderr=subprocess.STDOUT ) out = out.decode() try: > assert not out E assert not '2026-09-08 22:08:14,793 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat.\nTraceback (most recent call last):\n File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 226, in read\n frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\ntornado.iostream.StreamClosedError: Stream is closed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/build/python-distributed/src/distributed-2026.8.0/distributed/worker.py", line 1165, in heartbeat\n response = await retry_operation(\n ^^^^^^^^^^^^^^^^^^^^^^\n ...<14 lines>...\n )\n ^\n File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 420, in retry_operation\n return await retry(\n ^^^^^^^^^^^^\n ...<5 lines>...\n )\n ^\n File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 399, in retry\n return await coro()\n ^^^^^^^^^^^^\n File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1212, in send_recv_from_rpc\n return await send_recv(comm=comm, op=key, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 982, in send_recv\n response = await comm.read(deserializers=deserializers)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 237, in read\n convert_stream_closed_error(self, e)\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^\n File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 137, in convert_stream_closed_error\n raise CommClosedError(f"in {obj}: {exc}") from exc\ndistributed.comm.core.CommClosedError: in : Stream is closed\n' distributed/deploy/tests/test_local.py:541: AssertionError ----------------------------- Captured stdout call ----------------------------- === Cluster stdout / stderr === 2026-09-08 22:08:14,793 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 226, in read frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/worker.py", line 1165, in heartbeat response = await retry_operation( ^^^^^^^^^^^^^^^^^^^^^^ ...<14 lines>... ) ^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 420, in retry_operation return await retry( ^^^^^^^^^^^^ ...<5 lines>... ) ^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 399, in retry return await coro() ^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1212, in send_recv_from_rpc return await send_recv(comm=comm, op=key, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 982, in send_recv response = await comm.read(deserializers=deserializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 237, in read convert_stream_closed_error(self, e) ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 137, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in : Stream is closed ______________________________ test_remote_access ______________________________ self = address = ('fe80::9e69:b4ff:fe64:abd4', 43077, 0, 0), server_hostname = None def connect( self: _IOStreamType, address: Any, server_hostname: Optional[str] = None ) -> "Future[_IOStreamType]": """Connects the socket to a remote address without blocking. May only be called if the socket passed to the constructor was not previously connected. The address parameter is in the same format as for `socket.connect ` for the type of socket passed to the IOStream constructor, e.g. an ``(ip, port)`` tuple. Hostnames are accepted here, but will be resolved synchronously and block the IOLoop. If you have a hostname instead of an IP address, the `.TCPClient` class is recommended instead of calling this method directly. `.TCPClient` will do asynchronous DNS resolution and handle both IPv4 and IPv6. If ``callback`` is specified, it will be called with no arguments when the connection is completed; if not this method returns a `.Future` (whose result after a successful connection will be the stream itself). In SSL mode, the ``server_hostname`` parameter will be used for certificate validation (unless disabled in the ``ssl_options``) and SNI. Note that it is safe to call `IOStream.write ` while the connection is pending, in which case the data will be written as soon as the connection is ready. Calling `IOStream` read methods before the socket is connected works on some platforms but is non-portable. .. versionchanged:: 4.0 If no callback is given, returns a `.Future`. .. versionchanged:: 4.2 SSL certificates are validated by default; pass ``ssl_options=dict(cert_reqs=ssl.CERT_NONE)`` or a suitably-configured `ssl.SSLContext` to the `SSLIOStream` constructor to disable. .. versionchanged:: 6.0 The ``callback`` argument was removed. Use the returned `.Future` instead. """ self._connecting = True future = Future() # type: Future[_IOStreamType] self._connect_future = typing.cast("Future[IOStream]", future) try: > self.socket.connect(address) E OSError: [Errno 22] Invalid argument /usr/lib/python3.14/site-packages/tornado/iostream.py:1177: OSError The above exception was the direct cause of the following exception: addr = 'tcp://[fe80::9e69:b4ff:fe64:abd4]:43077', timeout = 0.5 deserialize = True, handshake_overrides = None, connection_args = {} async def connect( addr, timeout=None, deserialize=True, handshake_overrides=None, **connection_args ): """ Connect to the given address (a URI such as ``tcp://127.0.0.1:1234``) and yield a ``Comm`` object. If the connection attempt fails, it is retried until the *timeout* is expired. """ if timeout is None: timeout = dask.config.get("distributed.comm.timeouts.connect") timeout = parse_timedelta(timeout, default="seconds") scheme, loc = parse_address(addr) backend = registry.get_backend(scheme) connector = backend.get_connector() comm = None start = time() def time_left(): deadline = start + timeout return max(0, deadline - time()) backoff_base = 0.01 attempt = 0 logger.debug("Establishing connection to %s", loc) # Prefer multiple small attempts than one long attempt. This should protect # primarily from DNS race conditions # gh3104, gh4176, gh4167 intermediate_cap = timeout / 5 active_exception = None while time_left() > 0: try: > comm = await wait_for( connector.connect(loc, deserialize=deserialize, **connection_args), timeout=min(intermediate_cap, time_left()), ) distributed/comm/core.py:342: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ distributed/utils.py:1855: in wait_for return await fut ^^^^^^^^^ distributed/comm/tcp.py:560: in connect convert_stream_closed_error(self, e) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ obj = exc = OSError(22, 'Invalid argument') def convert_stream_closed_error(obj, exc): """ Re-raise StreamClosedError or SSLError as CommClosedError. """ if hasattr(exc, "real_error"): # The stream was closed because of an underlying OS error if exc.real_error is None: raise CommClosedError(f"in {obj}: {exc}") from exc exc = exc.real_error if isinstance(exc, ssl.SSLError): if exc.reason and "UNKNOWN_CA" in exc.reason: raise FatalCommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") > raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc E distributed.comm.core.CommClosedError: in : OSError: [Errno 22] Invalid argument distributed/comm/tcp.py:143: CommClosedError The above exception was the direct cause of the following exception: loop = def test_remote_access(loop): with LocalCluster( n_workers=0, silence_logs=False, dashboard_address=":0", host="", loop=loop, ) as c: > sync(loop, assert_can_connect_from_everywhere_4_6, c.scheduler.port) distributed/deploy/tests/test_local.py:563: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ distributed/utils.py:460: in sync raise error distributed/utils.py:434: in f result = yield future ^^^^^^^^^^^^ /usr/lib/python3.14/site-packages/tornado/gen.py:783: in run value = future.result() ^^^^^^^^^^^^^^^ distributed/utils_test.py:1305: in assert_can_connect_from_everywhere_4_6 await asyncio.gather(*futures) distributed/utils_test.py:1276: in assert_can_connect comm = await connect(addr, timeout=timeout, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ addr = 'tcp://[fe80::9e69:b4ff:fe64:abd4]:43077', timeout = 0.5 deserialize = True, handshake_overrides = None, connection_args = {} async def connect( addr, timeout=None, deserialize=True, handshake_overrides=None, **connection_args ): """ Connect to the given address (a URI such as ``tcp://127.0.0.1:1234``) and yield a ``Comm`` object. If the connection attempt fails, it is retried until the *timeout* is expired. """ if timeout is None: timeout = dask.config.get("distributed.comm.timeouts.connect") timeout = parse_timedelta(timeout, default="seconds") scheme, loc = parse_address(addr) backend = registry.get_backend(scheme) connector = backend.get_connector() comm = None start = time() def time_left(): deadline = start + timeout return max(0, deadline - time()) backoff_base = 0.01 attempt = 0 logger.debug("Establishing connection to %s", loc) # Prefer multiple small attempts than one long attempt. This should protect # primarily from DNS race conditions # gh3104, gh4176, gh4167 intermediate_cap = timeout / 5 active_exception = None while time_left() > 0: try: comm = await wait_for( connector.connect(loc, deserialize=deserialize, **connection_args), timeout=min(intermediate_cap, time_left()), ) break except FatalCommClosedError: raise # Note: CommClosed inherits from OSError except (asyncio.TimeoutError, OSError) as exc: active_exception = exc # As described above, the intermediate timeout is used to distribute # initial, bulk connect attempts homogeneously. In particular with # the jitter upon retries we should not be worried about overloading # any more DNS servers intermediate_cap = timeout # FullJitter see https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ upper_cap = min(time_left(), backoff_base * (2**attempt)) backoff = random.uniform(0, upper_cap) attempt += 1 logger.debug( "Could not connect to %s, waiting for %s before retrying", loc, backoff ) await asyncio.sleep(backoff) else: > raise OSError( f"Timed out trying to connect to {addr} after {timeout} s" ) from active_exception E OSError: Timed out trying to connect to tcp://[fe80::9e69:b4ff:fe64:abd4]:43077 after 0.5 s distributed/comm/core.py:368: OSError ----------------------------- Captured stderr call ----------------------------- 2026-09-08 22:08:18,155 - distributed.scheduler - INFO - State start 2026-09-08 22:08:18,232 - distributed.scheduler - INFO - Scheduler at: tcp://172.24.9.22:43077 2026-09-08 22:08:18,235 - distributed.scheduler - INFO - dashboard at: http://172.24.9.22:35195/status 2026-09-08 22:08:18,238 - distributed.scheduler - INFO - Registering Worker plugin shuffle 2026-09-08 22:08:18,845 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown 2026-09-08 22:08:18,850 - distributed.scheduler - INFO - Scheduler closing all comms ------------------------------ Captured log call ------------------------------- INFO distributed.scheduler:scheduler.py:1735 State start INFO distributed.scheduler:scheduler.py:4436 Scheduler at: tcp://172.24.9.22:43077 INFO distributed.scheduler:scheduler.py:4451 dashboard at: http://172.24.9.22:35195/status INFO distributed.scheduler:scheduler.py:8137 Registering Worker plugin shuffle INFO distributed.scheduler:scheduler.py:4493 Closing scheduler. Reason: unknown INFO distributed.scheduler:scheduler.py:4521 Scheduler closing all comms ___________________________ test_quiet_client_close ____________________________ loop = def test_quiet_client_close(loop): with captured_logger("distributed") as logger: with Client( loop=loop, processes=False, dashboard_address=":0", threads_per_worker=4, ) as c: futures = c.map(slowinc, range(1000), delay=0.01) # Stop part-way s = c.cluster.scheduler while sum(ts.state == "memory" for ts in list(s.tasks.values())) < 20: sleep(0.01) sleep(0.1) # let things settle out = logger.getvalue() lines = out.strip().split("\n") unexpected_lines = [ line for line in lines if line and "heartbeat from unregistered worker" not in line and "unaware of this worker" not in line and "garbage" not in line and "ended with CancelledError" not in line and set(line) != {"-"} ] > assert not unexpected_lines, lines E AssertionError: ['Async instruction for > ended with CancelledError', 'Async instruction for > ended with CancelledError', 'Failed to communicate with scheduler during heartbeat.', 'Traceback (most recent call last):', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1422, in connect', ' return await self._connect(addr=addr, timeout=timeout)', ...] E assert not ['Failed to communicate with scheduler during heartbeat.', 'Traceback (most recent call last):', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1422, in connect', ' return await self._connect(addr=addr, timeout=timeout)', ' ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1366, in _connect', ' comm = await connect(', ' ^^^^^^^^^^^^^^', ' ...<4 lines>...', ' )', ' ^', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/core.py", line 377, in connect', ' handshake = await comm.read()', ' ^^^^^^^^^^^^^^^^^', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/inproc.py", line 219, in read', ' msg = await self._read_q.get()', ' ^^^^^^^^^^^^^^^^^^^^^^^^', 'asyncio.exceptions.CancelledError', 'The above exception was the direct cause of the following exception:', 'Traceback (most recent call last):', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1413, in connect', ' async with asyncio.timeout(math.inf) as scope:', ' ~~~~~~~~~~~~~~~^^^^^^^^^^', ' File "/usr/lib/python3.14/asyncio/timeouts.py", line 115, in __aexit__', ' raise TimeoutError from exc_val', 'TimeoutError', 'During handling of the above exception, another exception occurred:', 'Traceback (most recent call last):', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/worker.py", line 1165, in heartbeat', ' response = await retry_operation(', ' ^^^^^^^^^^^^^^^^^^^^^^', ' ...<14 lines>...', ' )', ' ^', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 420, in retry_operation', ' return await retry(', ' ^^^^^^^^^^^^', ' ...<5 lines>...', ' )', ' ^', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 399, in retry', ' return await coro()', ' ^^^^^^^^^^^^', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1209, in send_recv_from_rpc', ' comm = await self.pool.connect(self.addr)', ' ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1434, in connect', ' raise CommClosedError(reason)', 'distributed.comm.core.CommClosedError: ConnectionPool closing.'] distributed/tests/test_client.py:5071: AssertionError _________________________________ test_restart _________________________________ c = s = a = , b = @gen_cluster(Worker=Nanny, client=True, timeout=60) async def test_restart(c, s, a, b): x = c.submit(inc, 1) y = c.submit(inc, x) z = c.submit(div, 1, 0) await y assert s.tasks[x.key].state == "memory" assert s.tasks[y.key].state == "memory" assert s.tasks[z.key].state != "memory" > await c.restart() distributed/tests/test_failed_workers.py:160: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ distributed/client.py:3879: in _restart await self.scheduler.restart( distributed/core.py:1212: in send_recv_from_rpc return await send_recv(comm=comm, op=key, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ distributed/core.py:1007: in send_recv raise exc.with_traceback(tb) distributed/core.py:798: in _handle_comm result = await result ^^^^^^^^^^^^^^^^^ distributed/utils.py:826: in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ distributed/scheduler.py:6467: in restart await self.restart_workers( ^^^^^^^^^^^^^^^^^ distributed/utils.py:826: in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > raise TimeoutError(msg) ^^^^^^^^^^^^^^^^^ E TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. distributed/scheduler.py:6650: TimeoutError During handling of the above exception, another exception occurred: fut = ._..test_func..async_fn at 0x3f52f1c7c0> timeout = 120 async def wait_for(fut: Awaitable[T], timeout: float) -> T: async with asyncio.timeout(timeout): > return await fut ^^^^^^^^^ distributed/utils.py:1855: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ distributed/utils_test.py:999: in async_fn await asyncio.sleep(0.01) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ delay = 0.01, result = None async def sleep(delay, result=None): """Coroutine that completes after a given time (in seconds).""" if delay <= 0: await __sleep0() return result if math.isnan(delay): raise ValueError("Invalid delay: NaN (not a number)") loop = events.get_running_loop() future = loop.create_future() h = loop.call_later(delay, futures._set_result_unless_cancelled, future, result) try: > return await future ^^^^^^^^^^^^ E asyncio.exceptions.CancelledError /usr/lib/python3.14/asyncio/tasks.py:705: CancelledError The above exception was the direct cause of the following exception: > lambda: ihook(item=item, **kwds), when=when, reraise=reraise ^^^^^^^^^^^^^^^^^^^^^^^^ ) /usr/lib/python3.14/site-packages/flaky/flaky_pytest_plugin.py:146: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python3.14/contextlib.py:85: in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/contextlib.py:85: in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ distributed/utils_test.py:1058: in test_func return _run_and_close_tornado(async_fn_outer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ distributed/utils_test.py:380: in _run_and_close_tornado return asyncio_run(inner_fn(), loop_factory=get_loop_factory()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/runners.py:205: in run return runner.run(main) ^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/runners.py:128: in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/base_events.py:720: in run_until_complete return future.result() ^^^^^^^^^^^^^^^ distributed/utils_test.py:377: in inner_fn return await async_fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ distributed/utils_test.py:1055: in async_fn_outer return await utils_wait_for(async_fn(), timeout=timeout * 2) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ distributed/utils.py:1854: in wait_for async with asyncio.timeout(timeout): ^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = exc_type = exc_val = CancelledError(), exc_tb = async def __aexit__( self, exc_type: type[BaseException] | None, exc_val: BaseException | None, exc_tb: TracebackType | None, ) -> bool | None: assert self._state in (_State.ENTERED, _State.EXPIRING) if self._timeout_handler is not None: self._timeout_handler.cancel() self._timeout_handler = None if self._state is _State.EXPIRING: self._state = _State.EXPIRED if self._task.uncancel() <= self._cancelling and exc_type is not None: # Since there are no new cancel requests, we're # handling this. if issubclass(exc_type, exceptions.CancelledError): > raise TimeoutError from exc_val E TimeoutError /usr/lib/python3.14/asyncio/timeouts.py:115: TimeoutError ----------------------------- Captured stderr call ----------------------------- 2026-09-09 00:16:00,213 - distributed.scheduler - INFO - State start 2026-09-09 00:16:00,251 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:42905 2026-09-09 00:16:00,259 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:37191/status 2026-09-09 00:16:00,267 - distributed.scheduler - INFO - Registering Worker plugin shuffle 2026-09-09 00:16:00,382 - distributed.nanny - INFO - Start Nanny at: 'tcp://127.0.0.1:38323' 2026-09-09 00:16:00,403 - distributed.nanny - INFO - Start Nanny at: 'tcp://127.0.0.1:46279' 2026-09-09 00:16:09,559 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:36267 2026-09-09 00:16:09,562 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:36267 2026-09-09 00:16:09,564 - distributed.worker - INFO - Worker name: 0 2026-09-09 00:16:09,567 - distributed.worker - INFO - dashboard at: 127.0.0.1:34209 2026-09-09 00:16:09,568 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:42905 2026-09-09 00:16:09,568 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:16:09,569 - distributed.worker - INFO - Threads: 1 2026-09-09 00:16:09,572 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:16:09,574 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-phbktsdr 2026-09-09 00:16:09,576 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:16:10,409 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:45809 2026-09-09 00:16:10,416 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:45809 2026-09-09 00:16:10,417 - distributed.worker - INFO - Worker name: 1 2026-09-09 00:16:10,419 - distributed.worker - INFO - dashboard at: 127.0.0.1:38029 2026-09-09 00:16:10,420 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:42905 2026-09-09 00:16:10,420 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:16:10,420 - distributed.worker - INFO - Threads: 2 2026-09-09 00:16:10,421 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:16:10,422 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-zk8du03e 2026-09-09 00:16:10,424 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:16:24,327 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:45809 name: 1 2026-09-09 00:16:24,338 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:45809 2026-09-09 00:16:24,346 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:33196 2026-09-09 00:16:24,395 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:16:24,430 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:42905 2026-09-09 00:16:24,432 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:16:24,458 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:42905 2026-09-09 00:16:25,419 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:36267 name: 0 2026-09-09 00:16:25,429 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:36267 2026-09-09 00:16:25,436 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:33190 2026-09-09 00:16:25,440 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:16:25,450 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:42905 2026-09-09 00:16:25,451 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:16:25,459 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:42905 2026-09-09 00:16:25,549 - distributed.scheduler - INFO - Receive client connection: Client-b1db1d38-abe3-11f1-86b4-8f07734cb81d 2026-09-09 00:16:25,558 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:39982 2026-09-09 00:16:36,179 - distributed.core - INFO - Event loop was unresponsive in Worker for 10.73s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. 2026-09-09 00:16:36,291 - distributed.scheduler - INFO - Restarting workers and releasing all keys (stimulus_id='client-restart-1788912996.2876732') 2026-09-09 00:16:36,302 - distributed.scheduler - INFO - Restarting all workers (stimulus_id='client-restart-1788912996.2876732' 2026-09-09 00:16:36,344 - distributed.nanny - INFO - Nanny asking worker to close. Reason: client-restart-1788912996.2876732 2026-09-09 00:16:36,350 - distributed.nanny - INFO - Nanny asking worker to close. Reason: client-restart-1788912996.2876732 2026-09-09 00:16:36,356 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:36267. Reason: client-restart-1788912996.2876732 2026-09-09 00:16:36,360 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:16:36,373 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:33190; closing. 2026-09-09 00:16:36,376 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:36267 name: 0 (stimulus_id='handle-worker-cleanup-1788912996.3760545') 2026-09-09 00:16:36,375 - distributed.core - INFO - Connection to tcp://127.0.0.1:42905 has been closed. 2026-09-09 00:16:36,398 - distributed.nanny - INFO - Worker closed 2026-09-09 00:16:37,314 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:45809. Reason: client-restart-1788912996.2876732 2026-09-09 00:16:37,376 - distributed.core - INFO - Event loop was unresponsive in Worker for 11.90s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. 2026-09-09 00:16:37,394 - distributed.worker.state_machine - WARNING - Async instruction for > ended with CancelledError 2026-09-09 00:16:37,409 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:16:37,433 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:33196; closing. 2026-09-09 00:16:37,436 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:45809 name: 1 (stimulus_id='handle-worker-cleanup-1788912997.4356394') 2026-09-09 00:16:37,440 - distributed.core - INFO - Connection to tcp://127.0.0.1:42905 has been closed. 2026-09-09 00:16:37,444 - distributed.scheduler - INFO - Lost all workers 2026-09-09 00:16:37,571 - distributed.nanny - INFO - Unregistering worker (status=Status.running) 2026-09-09 00:16:37,599 - distributed.nanny - WARNING - Restarting worker (status=Status.running) 2026-09-09 00:16:37,668 - distributed.nanny - INFO - Worker closed 2026-09-09 00:16:39,742 - distributed.nanny - INFO - Unregistering worker (status=Status.running) 2026-09-09 00:16:39,770 - distributed.nanny - WARNING - Restarting worker (status=Status.running) 2026-09-09 00:16:46,921 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:37735 2026-09-09 00:16:46,926 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:37735 2026-09-09 00:16:46,927 - distributed.worker - INFO - Worker name: 0 2026-09-09 00:16:46,929 - distributed.worker - INFO - dashboard at: 127.0.0.1:32991 2026-09-09 00:16:46,930 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:42905 2026-09-09 00:16:46,930 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:16:46,931 - distributed.worker - INFO - Threads: 1 2026-09-09 00:16:46,933 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:16:46,934 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-9uk8ukvf 2026-09-09 00:16:46,935 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:16:47,325 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:33685 2026-09-09 00:16:47,327 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:33685 2026-09-09 00:16:47,328 - distributed.worker - INFO - Worker name: 1 2026-09-09 00:16:47,329 - distributed.worker - INFO - dashboard at: 127.0.0.1:34897 2026-09-09 00:16:47,329 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:42905 2026-09-09 00:16:47,329 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:16:47,330 - distributed.worker - INFO - Threads: 2 2026-09-09 00:16:47,330 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:16:47,330 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-gx2mqqme 2026-09-09 00:16:47,331 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:16:56,404 - distributed.scheduler - ERROR - Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. 2026-09-09 00:16:56,420 - distributed.scheduler - ERROR - Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. 2026-09-09 00:16:56,433 - distributed.core - ERROR - Exception while handling op restart Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 798, in _handle_comm result = await result ^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. 2026-09-09 00:16:59,799 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:33685 name: 1 2026-09-09 00:16:59,811 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:33685 2026-09-09 00:16:59,819 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:57442 2026-09-09 00:16:59,818 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:16:59,825 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:42905 2026-09-09 00:16:59,826 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:16:59,831 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:42905 2026-09-09 00:17:03,730 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:37735 name: 0 2026-09-09 00:17:03,742 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:37735 2026-09-09 00:17:03,750 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:57434 2026-09-09 00:17:03,751 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:17:03,757 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:42905 2026-09-09 00:17:03,758 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:17:03,765 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:42905 2026-09-09 00:18:00,219 - distributed.scheduler - INFO - Remove client Client-b1db1d38-abe3-11f1-86b4-8f07734cb81d 2026-09-09 00:18:00,231 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:39982; closing. 2026-09-09 00:18:00,235 - distributed.scheduler - INFO - Remove client Client-b1db1d38-abe3-11f1-86b4-8f07734cb81d 2026-09-09 00:18:00,247 - distributed.scheduler - INFO - Close client connection: Client-b1db1d38-abe3-11f1-86b4-8f07734cb81d 2026-09-09 00:18:00,261 - distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:38323'. Reason: nanny-close 2026-09-09 00:18:00,263 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close 2026-09-09 00:18:00,265 - distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:46279'. Reason: nanny-close 2026-09-09 00:18:00,268 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close 2026-09-09 00:18:00,279 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:33685. Reason: nanny-close 2026-09-09 00:18:00,283 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:18:00,284 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:37735. Reason: nanny-close 2026-09-09 00:18:00,289 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:18:00,300 - distributed.core - INFO - Connection to tcp://127.0.0.1:42905 has been closed. 2026-09-09 00:18:00,302 - distributed.core - INFO - Connection to tcp://127.0.0.1:42905 has been closed. 2026-09-09 00:18:00,306 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:57442; closing. 2026-09-09 00:18:00,309 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:57434; closing. 2026-09-09 00:18:00,314 - distributed.nanny - INFO - Worker closed 2026-09-09 00:18:00,314 - distributed.nanny - INFO - Worker closed 2026-09-09 00:18:00,314 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:33685 name: 1 (stimulus_id='handle-worker-cleanup-1788913080.3132825') 2026-09-09 00:18:00,325 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:37735 name: 0 (stimulus_id='handle-worker-cleanup-1788913080.3253162') 2026-09-09 00:18:00,333 - distributed.scheduler - INFO - Lost all workers 2026-09-09 00:18:00,924 - distributed.nanny - INFO - Nanny at 'tcp://127.0.0.1:46279' closed. 2026-09-09 00:18:01,070 - distributed.nanny - INFO - Nanny at 'tcp://127.0.0.1:38323' closed. 2026-09-09 00:18:01,073 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown 2026-09-09 00:18:01,083 - distributed.scheduler - INFO - Scheduler closing all comms ------------------------------ Captured log call ------------------------------- INFO distributed.scheduler:scheduler.py:1735 State start INFO distributed.scheduler:scheduler.py:4436 Scheduler at: tcp://127.0.0.1:42905 INFO distributed.scheduler:scheduler.py:4451 dashboard at: http://127.0.0.1:37191/status INFO distributed.scheduler:scheduler.py:8137 Registering Worker plugin shuffle INFO distributed.nanny:nanny.py:350 Start Nanny at: 'tcp://127.0.0.1:38323' INFO distributed.nanny:nanny.py:350 Start Nanny at: 'tcp://127.0.0.1:46279' INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:45809 name: 1 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:45809 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:33196 INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:36267 name: 0 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:36267 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:33190 INFO distributed.scheduler:scheduler.py:5916 Receive client connection: Client-b1db1d38-abe3-11f1-86b4-8f07734cb81d INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:39982 INFO distributed.scheduler:scheduler.py:6449 Restarting workers and releasing all keys (stimulus_id='client-restart-1788912996.2876732') INFO distributed.scheduler:scheduler.py:6526 Restarting all workers (stimulus_id='client-restart-1788912996.2876732' INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: client-restart-1788912996.2876732 INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: client-restart-1788912996.2876732 INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:33190; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:36267 name: 0 (stimulus_id='handle-worker-cleanup-1788912996.3760545') INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:33196; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:45809 name: 1 (stimulus_id='handle-worker-cleanup-1788912997.4356394') INFO distributed.scheduler:scheduler.py:5729 Lost all workers INFO distributed.nanny:nanny.py:528 Unregistering worker (status=Status.running) WARNING distributed.nanny:nanny.py:546 Restarting worker (status=Status.running) INFO distributed.nanny:nanny.py:528 Unregistering worker (status=Status.running) WARNING distributed.nanny:nanny.py:546 Restarting worker (status=Status.running) ERROR distributed.scheduler:utils.py:851 Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. ERROR distributed.scheduler:utils.py:851 Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. ERROR distributed.core:core.py:808 Exception while handling op restart Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 798, in _handle_comm result = await result ^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:33685 name: 1 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:33685 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:57442 INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:37735 name: 0 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:37735 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:57434 INFO distributed.scheduler:scheduler.py:5961 Remove client Client-b1db1d38-abe3-11f1-86b4-8f07734cb81d INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:39982; closing. INFO distributed.scheduler:scheduler.py:5961 Remove client Client-b1db1d38-abe3-11f1-86b4-8f07734cb81d INFO distributed.scheduler:scheduler.py:5953 Close client connection: Client-b1db1d38-abe3-11f1-86b4-8f07734cb81d INFO distributed.nanny:nanny.py:587 Closing Nanny at 'tcp://127.0.0.1:38323'. Reason: nanny-close INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: nanny-close INFO distributed.nanny:nanny.py:587 Closing Nanny at 'tcp://127.0.0.1:46279'. Reason: nanny-close INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: nanny-close INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:57442; closing. INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:57434; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:33685 name: 1 (stimulus_id='handle-worker-cleanup-1788913080.3132825') INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:37735 name: 0 (stimulus_id='handle-worker-cleanup-1788913080.3253162') INFO distributed.scheduler:scheduler.py:5729 Lost all workers INFO distributed.nanny:nanny.py:606 Nanny at 'tcp://127.0.0.1:46279' closed. INFO distributed.nanny:nanny.py:606 Nanny at 'tcp://127.0.0.1:38323' closed. INFO distributed.scheduler:scheduler.py:4493 Closing scheduler. Reason: unknown INFO distributed.scheduler:scheduler.py:4521 Scheduler closing all comms _____________________________ test_restart_cleared _____________________________ c = s = a = , b = @gen_cluster(Worker=Nanny, client=True, timeout=60) async def test_restart_cleared(c, s, a, b): x = 2 * delayed(1) + 1 f = c.compute(x) await wait([f]) > await c.restart() distributed/tests/test_failed_workers.py:181: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ distributed/client.py:3879: in _restart await self.scheduler.restart( distributed/core.py:1212: in send_recv_from_rpc return await send_recv(comm=comm, op=key, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ distributed/core.py:1007: in send_recv raise exc.with_traceback(tb) distributed/core.py:798: in _handle_comm result = await result ^^^^^^^^^^^^^^^^^ distributed/utils.py:826: in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ distributed/scheduler.py:6467: in restart await self.restart_workers( ^^^^^^^^^^^^^^^^^ distributed/utils.py:826: in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > raise TimeoutError(msg) ^^^^^^^^^^^^^^^^^ E TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 1 have not returned. Consider a longer timeout, or `wait_for_workers=False`. distributed/scheduler.py:6650: TimeoutError During handling of the above exception, another exception occurred: fut = ._..test_func..async_fn at 0x3f52f1fe20> timeout = 120 async def wait_for(fut: Awaitable[T], timeout: float) -> T: async with asyncio.timeout(timeout): > return await fut ^^^^^^^^^ distributed/utils.py:1855: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ distributed/utils_test.py:999: in async_fn await asyncio.sleep(0.01) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ delay = 0.01, result = None async def sleep(delay, result=None): """Coroutine that completes after a given time (in seconds).""" if delay <= 0: await __sleep0() return result if math.isnan(delay): raise ValueError("Invalid delay: NaN (not a number)") loop = events.get_running_loop() future = loop.create_future() h = loop.call_later(delay, futures._set_result_unless_cancelled, future, result) try: > return await future ^^^^^^^^^^^^ E asyncio.exceptions.CancelledError /usr/lib/python3.14/asyncio/tasks.py:705: CancelledError The above exception was the direct cause of the following exception: > lambda: ihook(item=item, **kwds), when=when, reraise=reraise ^^^^^^^^^^^^^^^^^^^^^^^^ ) /usr/lib/python3.14/site-packages/flaky/flaky_pytest_plugin.py:146: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python3.14/contextlib.py:85: in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/contextlib.py:85: in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ distributed/utils_test.py:1058: in test_func return _run_and_close_tornado(async_fn_outer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ distributed/utils_test.py:380: in _run_and_close_tornado return asyncio_run(inner_fn(), loop_factory=get_loop_factory()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/runners.py:205: in run return runner.run(main) ^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/runners.py:128: in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/base_events.py:720: in run_until_complete return future.result() ^^^^^^^^^^^^^^^ distributed/utils_test.py:377: in inner_fn return await async_fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ distributed/utils_test.py:1055: in async_fn_outer return await utils_wait_for(async_fn(), timeout=timeout * 2) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ distributed/utils.py:1854: in wait_for async with asyncio.timeout(timeout): ^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = exc_type = exc_val = CancelledError(), exc_tb = async def __aexit__( self, exc_type: type[BaseException] | None, exc_val: BaseException | None, exc_tb: TracebackType | None, ) -> bool | None: assert self._state in (_State.ENTERED, _State.EXPIRING) if self._timeout_handler is not None: self._timeout_handler.cancel() self._timeout_handler = None if self._state is _State.EXPIRING: self._state = _State.EXPIRED if self._task.uncancel() <= self._cancelling and exc_type is not None: # Since there are no new cancel requests, we're # handling this. if issubclass(exc_type, exceptions.CancelledError): > raise TimeoutError from exc_val E TimeoutError /usr/lib/python3.14/asyncio/timeouts.py:115: TimeoutError ----------------------------- Captured stderr call ----------------------------- 2026-09-09 00:18:04,719 - distributed.scheduler - INFO - State start 2026-09-09 00:18:04,765 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:37011 2026-09-09 00:18:04,773 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:36939/status 2026-09-09 00:18:04,781 - distributed.scheduler - INFO - Registering Worker plugin shuffle 2026-09-09 00:18:04,872 - distributed.nanny - INFO - Start Nanny at: 'tcp://127.0.0.1:40427' 2026-09-09 00:18:04,902 - distributed.nanny - INFO - Start Nanny at: 'tcp://127.0.0.1:46823' 2026-09-09 00:18:13,414 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:40721 2026-09-09 00:18:13,418 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:40721 2026-09-09 00:18:13,419 - distributed.worker - INFO - Worker name: 1 2026-09-09 00:18:13,421 - distributed.worker - INFO - dashboard at: 127.0.0.1:40321 2026-09-09 00:18:13,422 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:37011 2026-09-09 00:18:13,423 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:18:13,424 - distributed.worker - INFO - Threads: 2 2026-09-09 00:18:13,425 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:18:13,426 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-lmuxyuff 2026-09-09 00:18:13,427 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:18:13,556 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:36859 2026-09-09 00:18:13,559 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:36859 2026-09-09 00:18:13,560 - distributed.worker - INFO - Worker name: 0 2026-09-09 00:18:13,562 - distributed.worker - INFO - dashboard at: 127.0.0.1:38529 2026-09-09 00:18:13,563 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:37011 2026-09-09 00:18:13,563 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:18:13,564 - distributed.worker - INFO - Threads: 1 2026-09-09 00:18:13,564 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:18:13,565 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-roj0u5y9 2026-09-09 00:18:13,566 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:18:27,469 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:36859 name: 0 2026-09-09 00:18:27,484 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:36859 2026-09-09 00:18:27,492 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:34960 2026-09-09 00:18:27,524 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:18:27,544 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:37011 2026-09-09 00:18:27,545 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:18:27,557 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:37011 2026-09-09 00:18:30,266 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:40721 name: 1 2026-09-09 00:18:30,276 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:40721 2026-09-09 00:18:30,284 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:34950 2026-09-09 00:18:30,284 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:18:30,291 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:37011 2026-09-09 00:18:30,292 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:18:30,301 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:37011 2026-09-09 00:18:30,398 - distributed.scheduler - INFO - Receive client connection: Client-fc448712-abe3-11f1-86b4-8f07734cb81d 2026-09-09 00:18:30,408 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:35626 2026-09-09 00:18:30,606 - distributed.scheduler - INFO - Restarting workers and releasing all keys (stimulus_id='client-restart-1788913110.6003568') 2026-09-09 00:18:30,638 - distributed.scheduler - INFO - Restarting all workers (stimulus_id='client-restart-1788913110.6003568' 2026-09-09 00:18:30,690 - distributed.nanny - INFO - Nanny asking worker to close. Reason: client-restart-1788913110.6003568 2026-09-09 00:18:30,700 - distributed.nanny - INFO - Nanny asking worker to close. Reason: client-restart-1788913110.6003568 2026-09-09 00:18:30,704 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:36859. Reason: client-restart-1788913110.6003568 2026-09-09 00:18:30,706 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:18:30,707 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:40721. Reason: client-restart-1788913110.6003568 2026-09-09 00:18:30,709 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:18:30,721 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:34950; closing. 2026-09-09 00:18:30,721 - distributed.core - INFO - Connection to tcp://127.0.0.1:37011 has been closed. 2026-09-09 00:18:30,725 - distributed.core - INFO - Connection to tcp://127.0.0.1:37011 has been closed. 2026-09-09 00:18:30,724 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:40721 name: 1 (stimulus_id='handle-worker-cleanup-1788913110.7240438') 2026-09-09 00:18:30,732 - distributed.nanny - INFO - Worker closed 2026-09-09 00:18:30,738 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:34960; closing. 2026-09-09 00:18:30,742 - distributed.nanny - INFO - Worker closed 2026-09-09 00:18:30,742 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:36859 name: 0 (stimulus_id='handle-worker-cleanup-1788913110.7423713') 2026-09-09 00:18:30,750 - distributed.scheduler - INFO - Lost all workers 2026-09-09 00:18:31,329 - distributed.nanny - INFO - Unregistering worker (status=Status.running) 2026-09-09 00:18:31,370 - distributed.nanny - WARNING - Restarting worker (status=Status.running) 2026-09-09 00:18:31,818 - distributed.nanny - INFO - Unregistering worker (status=Status.running) 2026-09-09 00:18:31,857 - distributed.nanny - WARNING - Restarting worker (status=Status.running) 2026-09-09 00:18:39,527 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:43549 2026-09-09 00:18:39,530 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:43549 2026-09-09 00:18:39,531 - distributed.worker - INFO - Worker name: 0 2026-09-09 00:18:39,533 - distributed.worker - INFO - dashboard at: 127.0.0.1:45009 2026-09-09 00:18:39,534 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:37011 2026-09-09 00:18:39,535 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:18:39,535 - distributed.worker - INFO - Threads: 1 2026-09-09 00:18:39,536 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:18:39,537 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-g7t_djdw 2026-09-09 00:18:39,538 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:18:40,045 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:43475 2026-09-09 00:18:40,048 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:43475 2026-09-09 00:18:40,049 - distributed.worker - INFO - Worker name: 1 2026-09-09 00:18:40,050 - distributed.worker - INFO - dashboard at: 127.0.0.1:37345 2026-09-09 00:18:40,050 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:37011 2026-09-09 00:18:40,050 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:18:40,050 - distributed.worker - INFO - Threads: 2 2026-09-09 00:18:40,051 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:18:40,052 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-cahza2i3 2026-09-09 00:18:40,052 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:18:49,021 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:43549 name: 0 2026-09-09 00:18:49,033 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:43549 2026-09-09 00:18:49,041 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:18:49,041 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:47568 2026-09-09 00:18:49,048 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:37011 2026-09-09 00:18:49,049 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:18:49,055 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:37011 2026-09-09 00:18:50,697 - distributed.scheduler - ERROR - Waited for 2 worker(s) to reconnect after restarting but, after 20s, 1 have not returned. Consider a longer timeout, or `wait_for_workers=False`. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 1 have not returned. Consider a longer timeout, or `wait_for_workers=False`. 2026-09-09 00:18:50,731 - distributed.scheduler - ERROR - Waited for 2 worker(s) to reconnect after restarting but, after 20s, 1 have not returned. Consider a longer timeout, or `wait_for_workers=False`. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 1 have not returned. Consider a longer timeout, or `wait_for_workers=False`. 2026-09-09 00:18:50,751 - distributed.core - ERROR - Exception while handling op restart Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 798, in _handle_comm result = await result ^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 1 have not returned. Consider a longer timeout, or `wait_for_workers=False`. 2026-09-09 00:18:51,607 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:43475 name: 1 2026-09-09 00:18:51,617 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:43475 2026-09-09 00:18:51,625 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:47576 2026-09-09 00:18:51,625 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:18:51,631 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:37011 2026-09-09 00:18:51,632 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:18:51,637 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:37011 2026-09-09 00:20:04,708 - distributed.scheduler - INFO - Remove client Client-fc448712-abe3-11f1-86b4-8f07734cb81d 2026-09-09 00:20:04,718 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:35626; closing. 2026-09-09 00:20:04,721 - distributed.scheduler - INFO - Remove client Client-fc448712-abe3-11f1-86b4-8f07734cb81d 2026-09-09 00:20:04,731 - distributed.scheduler - INFO - Close client connection: Client-fc448712-abe3-11f1-86b4-8f07734cb81d 2026-09-09 00:20:04,742 - distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:40427'. Reason: nanny-close 2026-09-09 00:20:04,744 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close 2026-09-09 00:20:04,746 - distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:46823'. Reason: nanny-close 2026-09-09 00:20:04,749 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close 2026-09-09 00:20:04,751 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:43549. Reason: nanny-close 2026-09-09 00:20:04,756 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:20:04,768 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:43475. Reason: nanny-close 2026-09-09 00:20:04,771 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:20:04,771 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:47568; closing. 2026-09-09 00:20:04,773 - distributed.core - INFO - Connection to tcp://127.0.0.1:37011 has been closed. 2026-09-09 00:20:04,774 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:43549 name: 0 (stimulus_id='handle-worker-cleanup-1788913204.7743165') 2026-09-09 00:20:04,783 - distributed.nanny - INFO - Worker closed 2026-09-09 00:20:04,784 - distributed.core - INFO - Connection to tcp://127.0.0.1:37011 has been closed. 2026-09-09 00:20:04,794 - distributed.nanny - INFO - Worker closed 2026-09-09 00:20:04,802 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:47576; closing. 2026-09-09 00:20:04,805 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:43475 name: 1 (stimulus_id='handle-worker-cleanup-1788913204.804961') 2026-09-09 00:20:04,813 - distributed.scheduler - INFO - Lost all workers 2026-09-09 00:20:05,316 - distributed.nanny - INFO - Nanny at 'tcp://127.0.0.1:40427' closed. 2026-09-09 00:20:05,375 - distributed.nanny - INFO - Nanny at 'tcp://127.0.0.1:46823' closed. 2026-09-09 00:20:05,378 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown 2026-09-09 00:20:05,388 - distributed.scheduler - INFO - Scheduler closing all comms ------------------------------ Captured log call ------------------------------- INFO distributed.scheduler:scheduler.py:1735 State start INFO distributed.scheduler:scheduler.py:4436 Scheduler at: tcp://127.0.0.1:37011 INFO distributed.scheduler:scheduler.py:4451 dashboard at: http://127.0.0.1:36939/status INFO distributed.scheduler:scheduler.py:8137 Registering Worker plugin shuffle INFO distributed.nanny:nanny.py:350 Start Nanny at: 'tcp://127.0.0.1:40427' INFO distributed.nanny:nanny.py:350 Start Nanny at: 'tcp://127.0.0.1:46823' INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:36859 name: 0 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:36859 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:34960 INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:40721 name: 1 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:40721 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:34950 INFO distributed.scheduler:scheduler.py:5916 Receive client connection: Client-fc448712-abe3-11f1-86b4-8f07734cb81d INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:35626 INFO distributed.scheduler:scheduler.py:6449 Restarting workers and releasing all keys (stimulus_id='client-restart-1788913110.6003568') INFO distributed.scheduler:scheduler.py:6526 Restarting all workers (stimulus_id='client-restart-1788913110.6003568' INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: client-restart-1788913110.6003568 INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: client-restart-1788913110.6003568 INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:34950; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:40721 name: 1 (stimulus_id='handle-worker-cleanup-1788913110.7240438') INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:34960; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:36859 name: 0 (stimulus_id='handle-worker-cleanup-1788913110.7423713') INFO distributed.scheduler:scheduler.py:5729 Lost all workers INFO distributed.nanny:nanny.py:528 Unregistering worker (status=Status.running) WARNING distributed.nanny:nanny.py:546 Restarting worker (status=Status.running) INFO distributed.nanny:nanny.py:528 Unregistering worker (status=Status.running) WARNING distributed.nanny:nanny.py:546 Restarting worker (status=Status.running) INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:43549 name: 0 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:43549 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:47568 ERROR distributed.scheduler:utils.py:851 Waited for 2 worker(s) to reconnect after restarting but, after 20s, 1 have not returned. Consider a longer timeout, or `wait_for_workers=False`. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 1 have not returned. Consider a longer timeout, or `wait_for_workers=False`. ERROR distributed.scheduler:utils.py:851 Waited for 2 worker(s) to reconnect after restarting but, after 20s, 1 have not returned. Consider a longer timeout, or `wait_for_workers=False`. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 1 have not returned. Consider a longer timeout, or `wait_for_workers=False`. ERROR distributed.core:core.py:808 Exception while handling op restart Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 798, in _handle_comm result = await result ^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 1 have not returned. Consider a longer timeout, or `wait_for_workers=False`. INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:43475 name: 1 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:43475 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:47576 INFO distributed.scheduler:scheduler.py:5961 Remove client Client-fc448712-abe3-11f1-86b4-8f07734cb81d INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:35626; closing. INFO distributed.scheduler:scheduler.py:5961 Remove client Client-fc448712-abe3-11f1-86b4-8f07734cb81d INFO distributed.scheduler:scheduler.py:5953 Close client connection: Client-fc448712-abe3-11f1-86b4-8f07734cb81d INFO distributed.nanny:nanny.py:587 Closing Nanny at 'tcp://127.0.0.1:40427'. Reason: nanny-close INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: nanny-close INFO distributed.nanny:nanny.py:587 Closing Nanny at 'tcp://127.0.0.1:46823'. Reason: nanny-close INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: nanny-close INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:47568; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:43549 name: 0 (stimulus_id='handle-worker-cleanup-1788913204.7743165') INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:47576; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:43475 name: 1 (stimulus_id='handle-worker-cleanup-1788913204.804961') INFO distributed.scheduler:scheduler.py:5729 Lost all workers INFO distributed.nanny:nanny.py:606 Nanny at 'tcp://127.0.0.1:40427' closed. INFO distributed.nanny:nanny.py:606 Nanny at 'tcp://127.0.0.1:46823' closed. INFO distributed.scheduler:scheduler.py:4493 Closing scheduler. Reason: unknown INFO distributed.scheduler:scheduler.py:4521 Scheduler closing all comms ________________________ test_multiple_clients_restart _________________________ > lambda: ihook(item=item, **kwds), when=when, reraise=reraise ^^^^^^^^^^^^^^^^^^^^^^^^ ) /usr/lib/python3.14/site-packages/flaky/flaky_pytest_plugin.py:146: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python3.14/contextlib.py:85: in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/contextlib.py:85: in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ distributed/utils_test.py:1058: in test_func return _run_and_close_tornado(async_fn_outer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ distributed/utils_test.py:380: in _run_and_close_tornado return asyncio_run(inner_fn(), loop_factory=get_loop_factory()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/runners.py:205: in run return runner.run(main) ^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/runners.py:128: in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/base_events.py:720: in run_until_complete return future.result() ^^^^^^^^^^^^^^^ distributed/utils_test.py:377: in inner_fn return await async_fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ distributed/utils_test.py:1055: in async_fn_outer return await utils_wait_for(async_fn(), timeout=timeout * 2) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ distributed/utils.py:1855: in wait_for return await fut ^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ async def async_fn(): result = None with dask.config.set(config): async with ( _cluster_factory() as (s, workers), _client_factory(s) as c, ): args = [s] + workers if c is not None: args = [c] + args try: coro = func(*args, *outer_args, **kwargs) task = asyncio.create_task(coro) coro2 = utils_wait_for( asyncio.shield(task), timeout=deadline.remaining ) result = await coro2 validate_state(s, *workers) except asyncio.TimeoutError: assert task elapsed = deadline.elapsed buffer = io.StringIO() # This stack indicates where the coro/test is suspended task.print_stack(file=buffer) task.cancel() while not task.cancelled(): await asyncio.sleep(0.01) # Hopefully, the hang has been caused by inconsistent # state, which should be much more meaningful than the # timeout validate_state(s, *workers) # Remove as much of the traceback as possible; it's # uninteresting boilerplate from utils_test and asyncio # and not from the code being tested. > raise asyncio.TimeoutError( f"Test timeout ({timeout}) hit after {elapsed}s.\n" "========== Test stack trace starts here ==========\n" f"{buffer.getvalue()}" ) from None E TimeoutError: Test timeout (60) hit after 60.020716443999845s. E ========== Test stack trace starts here ========== E Stack for wait_for= cb=[shield.._clear_awaited_by_callback() at /usr/lib/python3.14/asyncio/tasks.py:978, _log_on_exception() at /usr/lib/python3.14/asyncio/tasks.py:914]> (most recent call last): E File "/build/python-distributed/src/distributed-2026.8.0/distributed/tests/test_failed_workers.py", line 230, in test_multiple_clients_restart E await c1.restart() distributed/utils_test.py:1009: TimeoutError ----------------------------- Captured stderr call ----------------------------- 2026-09-09 00:22:50,088 - distributed.scheduler - INFO - State start 2026-09-09 00:22:50,128 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-scratch-space/scheduler-zc0vamuw', purging 2026-09-09 00:22:50,180 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:37867 2026-09-09 00:22:50,191 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:36841/status 2026-09-09 00:22:50,202 - distributed.scheduler - INFO - Registering Worker plugin shuffle 2026-09-09 00:22:50,348 - distributed.nanny - INFO - Start Nanny at: 'tcp://127.0.0.1:44137' 2026-09-09 00:22:50,376 - distributed.nanny - INFO - Start Nanny at: 'tcp://127.0.0.1:45835' 2026-09-09 00:22:59,475 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:36807 2026-09-09 00:22:59,478 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:36807 2026-09-09 00:22:59,479 - distributed.worker - INFO - Worker name: 1 2026-09-09 00:22:59,480 - distributed.worker - INFO - dashboard at: 127.0.0.1:42617 2026-09-09 00:22:59,480 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:37867 2026-09-09 00:22:59,480 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:22:59,481 - distributed.worker - INFO - Threads: 2 2026-09-09 00:22:59,482 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:22:59,482 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-qhg_o52k 2026-09-09 00:22:59,483 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:23:02,726 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:46867 2026-09-09 00:23:02,730 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:46867 2026-09-09 00:23:02,731 - distributed.worker - INFO - Worker name: 0 2026-09-09 00:23:02,734 - distributed.worker - INFO - dashboard at: 127.0.0.1:36975 2026-09-09 00:23:02,736 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:37867 2026-09-09 00:23:02,737 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:23:02,738 - distributed.worker - INFO - Threads: 1 2026-09-09 00:23:02,740 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:23:02,741 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-x4oxe8qx 2026-09-09 00:23:02,743 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:23:20,193 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:36807 name: 1 2026-09-09 00:23:20,205 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:36807 2026-09-09 00:23:20,214 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:56514 2026-09-09 00:23:20,219 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:23:20,230 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:37867 2026-09-09 00:23:20,232 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:23:20,245 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:37867 2026-09-09 00:23:20,970 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:46867 name: 0 2026-09-09 00:23:20,992 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:23:20,985 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:46867 2026-09-09 00:23:20,996 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:57592 2026-09-09 00:23:20,999 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:37867 2026-09-09 00:23:20,999 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:23:21,006 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:37867 2026-09-09 00:23:21,161 - distributed.scheduler - INFO - Receive client connection: Client-a98f0814-abe4-11f1-86b4-8f07734cb81d 2026-09-09 00:23:21,174 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:40034 2026-09-09 00:23:21,348 - distributed.scheduler - INFO - Receive client connection: Client-a9a08465-abe4-11f1-86b4-8f07734cb81d 2026-09-09 00:23:21,362 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:40048 2026-09-09 00:23:32,166 - distributed.core - INFO - Event loop was unresponsive in Worker for 10.90s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. 2026-09-09 00:23:35,302 - distributed.core - INFO - Event loop was unresponsive in Worker for 14.30s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. 2026-09-09 00:23:35,385 - distributed.scheduler - INFO - Restarting workers and releasing all keys (stimulus_id='client-restart-1788913415.3803601') 2026-09-09 00:23:35,419 - distributed.scheduler - INFO - Restarting all workers (stimulus_id='client-restart-1788913415.3803601' 2026-09-09 00:23:35,480 - distributed.nanny - INFO - Nanny asking worker to close. Reason: client-restart-1788913415.3803601 2026-09-09 00:23:35,486 - distributed.nanny - INFO - Nanny asking worker to close. Reason: client-restart-1788913415.3803601 2026-09-09 00:23:35,489 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:36807. Reason: client-restart-1788913415.3803601 2026-09-09 00:23:35,494 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:23:35,496 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:46867. Reason: client-restart-1788913415.3803601 2026-09-09 00:23:35,512 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:56514; closing. 2026-09-09 00:23:35,515 - distributed.core - INFO - Connection to tcp://127.0.0.1:37867 has been closed. 2026-09-09 00:23:35,515 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:36807 name: 1 (stimulus_id='handle-worker-cleanup-1788913415.515394') 2026-09-09 00:23:35,522 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:23:35,537 - distributed.nanny - INFO - Worker closed 2026-09-09 00:23:35,552 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:57592; closing. 2026-09-09 00:23:35,555 - distributed.core - INFO - Connection to tcp://127.0.0.1:37867 has been closed. 2026-09-09 00:23:35,554 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:46867 name: 0 (stimulus_id='handle-worker-cleanup-1788913415.554493') 2026-09-09 00:23:35,563 - distributed.scheduler - INFO - Lost all workers 2026-09-09 00:23:35,582 - distributed.nanny - INFO - Worker closed 2026-09-09 00:23:36,989 - distributed.nanny - INFO - Unregistering worker (status=Status.running) 2026-09-09 00:23:37,030 - distributed.nanny - WARNING - Restarting worker (status=Status.running) 2026-09-09 00:23:37,104 - distributed.nanny - INFO - Unregistering worker (status=Status.running) 2026-09-09 00:23:37,140 - distributed.nanny - WARNING - Restarting worker (status=Status.running) 2026-09-09 00:23:48,815 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:43235 2026-09-09 00:23:48,817 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:43235 2026-09-09 00:23:48,818 - distributed.worker - INFO - Worker name: 1 2026-09-09 00:23:48,820 - distributed.worker - INFO - dashboard at: 127.0.0.1:41583 2026-09-09 00:23:48,821 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:37867 2026-09-09 00:23:48,821 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:23:48,822 - distributed.worker - INFO - Threads: 2 2026-09-09 00:23:48,824 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:23:48,824 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-qw806e9h 2026-09-09 00:23:48,826 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:23:50,120 - distributed.scheduler - INFO - Remove client Client-a9a08465-abe4-11f1-86b4-8f07734cb81d 2026-09-09 00:23:50,133 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:40048; closing. 2026-09-09 00:23:50,137 - distributed.scheduler - INFO - Remove client Client-a9a08465-abe4-11f1-86b4-8f07734cb81d 2026-09-09 00:23:50,164 - distributed.scheduler - INFO - Close client connection: Client-a9a08465-abe4-11f1-86b4-8f07734cb81d 2026-09-09 00:23:50,194 - distributed.scheduler - INFO - Remove client Client-a98f0814-abe4-11f1-86b4-8f07734cb81d 2026-09-09 00:23:50,204 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:40034; closing. 2026-09-09 00:23:50,210 - distributed.scheduler - INFO - Remove client Client-a98f0814-abe4-11f1-86b4-8f07734cb81d 2026-09-09 00:23:50,243 - distributed.scheduler - INFO - Close client connection: Client-a98f0814-abe4-11f1-86b4-8f07734cb81d 2026-09-09 00:23:50,260 - distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:44137'. Reason: nanny-close 2026-09-09 00:23:50,264 - distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:45835'. Reason: nanny-close 2026-09-09 00:23:50,923 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:42627 2026-09-09 00:23:50,927 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:42627 2026-09-09 00:23:50,929 - distributed.worker - INFO - Worker name: 0 2026-09-09 00:23:50,932 - distributed.worker - INFO - dashboard at: 127.0.0.1:40771 2026-09-09 00:23:50,933 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:37867 2026-09-09 00:23:50,935 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:23:50,936 - distributed.worker - INFO - Threads: 1 2026-09-09 00:23:50,937 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:23:50,939 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-4qyvlxek 2026-09-09 00:23:50,940 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:23:55,476 - distributed.scheduler - ERROR - Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. 2026-09-09 00:23:55,495 - distributed.scheduler - ERROR - Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. 2026-09-09 00:23:55,539 - distributed.core - ERROR - Exception while handling op restart Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 798, in _handle_comm result = await result ^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. 2026-09-09 00:24:11,950 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:42627 name: 0 2026-09-09 00:24:11,961 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:42627 2026-09-09 00:24:11,969 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:44078 2026-09-09 00:24:11,978 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:24:11,990 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:37867 2026-09-09 00:24:11,992 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:24:12,004 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:37867 2026-09-09 00:24:12,045 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close 2026-09-09 00:24:12,055 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:42627. Reason: nanny-close 2026-09-09 00:24:12,060 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:24:12,081 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:44078; closing. 2026-09-09 00:24:12,084 - distributed.core - INFO - Connection to tcp://127.0.0.1:37867 has been closed. 2026-09-09 00:24:12,085 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:42627 name: 0 (stimulus_id='handle-worker-cleanup-1788913452.0846894') 2026-09-09 00:24:12,094 - distributed.scheduler - INFO - Lost all workers 2026-09-09 00:24:12,104 - distributed.nanny - INFO - Worker closed 2026-09-09 00:24:13,315 - distributed.nanny - INFO - Nanny at 'tcp://127.0.0.1:44137' closed. 2026-09-09 00:24:17,736 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:43235 name: 1 2026-09-09 00:24:17,748 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:43235 2026-09-09 00:24:17,756 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:44062 2026-09-09 00:24:17,762 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:24:17,772 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:37867 2026-09-09 00:24:17,773 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:24:17,783 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:37867 2026-09-09 00:24:17,820 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close 2026-09-09 00:24:17,828 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:43235. Reason: nanny-close 2026-09-09 00:24:17,830 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:24:17,844 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:44062; closing. 2026-09-09 00:24:17,845 - distributed.core - INFO - Connection to tcp://127.0.0.1:37867 has been closed. 2026-09-09 00:24:17,848 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:43235 name: 1 (stimulus_id='handle-worker-cleanup-1788913457.84824') 2026-09-09 00:24:17,860 - distributed.nanny - INFO - Worker closed 2026-09-09 00:24:17,858 - distributed.scheduler - INFO - Lost all workers 2026-09-09 00:24:19,761 - distributed.nanny - INFO - Nanny at 'tcp://127.0.0.1:45835' closed. 2026-09-09 00:24:19,765 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown 2026-09-09 00:24:19,802 - distributed.scheduler - INFO - Scheduler closing all comms ------------------------------ Captured log call ------------------------------- INFO distributed.scheduler:scheduler.py:1735 State start INFO distributed.diskutils:diskutils.py:252 Found stale lock file and directory '/tmp/dask-scratch-space/scheduler-zc0vamuw', purging INFO distributed.scheduler:scheduler.py:4436 Scheduler at: tcp://127.0.0.1:37867 INFO distributed.scheduler:scheduler.py:4451 dashboard at: http://127.0.0.1:36841/status INFO distributed.scheduler:scheduler.py:8137 Registering Worker plugin shuffle INFO distributed.nanny:nanny.py:350 Start Nanny at: 'tcp://127.0.0.1:44137' INFO distributed.nanny:nanny.py:350 Start Nanny at: 'tcp://127.0.0.1:45835' INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:36807 name: 1 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:36807 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:56514 INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:46867 name: 0 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:46867 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:57592 INFO distributed.scheduler:scheduler.py:5916 Receive client connection: Client-a98f0814-abe4-11f1-86b4-8f07734cb81d INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:40034 INFO distributed.scheduler:scheduler.py:5916 Receive client connection: Client-a9a08465-abe4-11f1-86b4-8f07734cb81d INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:40048 INFO distributed.scheduler:scheduler.py:6449 Restarting workers and releasing all keys (stimulus_id='client-restart-1788913415.3803601') INFO distributed.scheduler:scheduler.py:6526 Restarting all workers (stimulus_id='client-restart-1788913415.3803601' INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: client-restart-1788913415.3803601 INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: client-restart-1788913415.3803601 INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:56514; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:36807 name: 1 (stimulus_id='handle-worker-cleanup-1788913415.515394') INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:57592; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:46867 name: 0 (stimulus_id='handle-worker-cleanup-1788913415.554493') INFO distributed.scheduler:scheduler.py:5729 Lost all workers INFO distributed.nanny:nanny.py:528 Unregistering worker (status=Status.running) WARNING distributed.nanny:nanny.py:546 Restarting worker (status=Status.running) INFO distributed.nanny:nanny.py:528 Unregistering worker (status=Status.running) WARNING distributed.nanny:nanny.py:546 Restarting worker (status=Status.running) INFO distributed.scheduler:scheduler.py:5961 Remove client Client-a9a08465-abe4-11f1-86b4-8f07734cb81d INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:40048; closing. INFO distributed.scheduler:scheduler.py:5961 Remove client Client-a9a08465-abe4-11f1-86b4-8f07734cb81d INFO distributed.scheduler:scheduler.py:5953 Close client connection: Client-a9a08465-abe4-11f1-86b4-8f07734cb81d INFO distributed.scheduler:scheduler.py:5961 Remove client Client-a98f0814-abe4-11f1-86b4-8f07734cb81d INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:40034; closing. INFO distributed.scheduler:scheduler.py:5961 Remove client Client-a98f0814-abe4-11f1-86b4-8f07734cb81d INFO distributed.scheduler:scheduler.py:5953 Close client connection: Client-a98f0814-abe4-11f1-86b4-8f07734cb81d INFO distributed.nanny:nanny.py:587 Closing Nanny at 'tcp://127.0.0.1:44137'. Reason: nanny-close INFO distributed.nanny:nanny.py:587 Closing Nanny at 'tcp://127.0.0.1:45835'. Reason: nanny-close ERROR distributed.scheduler:utils.py:851 Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. ERROR distributed.scheduler:utils.py:851 Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. ERROR distributed.core:core.py:808 Exception while handling op restart Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 798, in _handle_comm result = await result ^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. ERROR asyncio:base_events.py:1886 Task exception was never retrieved future: exception=TimeoutError('Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`.')> Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 798, in _handle_comm result = await result ^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:42627 name: 0 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:42627 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:44078 INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: nanny-close INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:44078; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:42627 name: 0 (stimulus_id='handle-worker-cleanup-1788913452.0846894') INFO distributed.scheduler:scheduler.py:5729 Lost all workers INFO distributed.nanny:nanny.py:606 Nanny at 'tcp://127.0.0.1:44137' closed. INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:43235 name: 1 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:43235 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:44062 INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: nanny-close INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:44062; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:43235 name: 1 (stimulus_id='handle-worker-cleanup-1788913457.84824') INFO distributed.scheduler:scheduler.py:5729 Lost all workers INFO distributed.nanny:nanny.py:606 Nanny at 'tcp://127.0.0.1:45835' closed. INFO distributed.scheduler:scheduler.py:4493 Closing scheduler. Reason: unknown INFO distributed.scheduler:scheduler.py:4521 Scheduler closing all comms ____________________________ test_restart_scheduler ____________________________ > lambda: ihook(item=item, **kwds), when=when, reraise=reraise ^^^^^^^^^^^^^^^^^^^^^^^^ ) /usr/lib/python3.14/site-packages/flaky/flaky_pytest_plugin.py:146: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python3.14/contextlib.py:85: in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/contextlib.py:85: in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ distributed/utils_test.py:1058: in test_func return _run_and_close_tornado(async_fn_outer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ distributed/utils_test.py:380: in _run_and_close_tornado return asyncio_run(inner_fn(), loop_factory=get_loop_factory()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/runners.py:205: in run return runner.run(main) ^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/runners.py:128: in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/base_events.py:720: in run_until_complete return future.result() ^^^^^^^^^^^^^^^ distributed/utils_test.py:377: in inner_fn return await async_fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ distributed/utils_test.py:1055: in async_fn_outer return await utils_wait_for(async_fn(), timeout=timeout * 2) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ distributed/utils.py:1855: in wait_for return await fut ^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ async def async_fn(): result = None with dask.config.set(config): async with ( _cluster_factory() as (s, workers), _client_factory(s) as c, ): args = [s] + workers if c is not None: args = [c] + args try: coro = func(*args, *outer_args, **kwargs) task = asyncio.create_task(coro) coro2 = utils_wait_for( asyncio.shield(task), timeout=deadline.remaining ) result = await coro2 validate_state(s, *workers) except asyncio.TimeoutError: assert task elapsed = deadline.elapsed buffer = io.StringIO() # This stack indicates where the coro/test is suspended task.print_stack(file=buffer) task.cancel() while not task.cancelled(): await asyncio.sleep(0.01) # Hopefully, the hang has been caused by inconsistent # state, which should be much more meaningful than the # timeout validate_state(s, *workers) # Remove as much of the traceback as possible; it's # uninteresting boilerplate from utils_test and asyncio # and not from the code being tested. > raise asyncio.TimeoutError( f"Test timeout ({timeout}) hit after {elapsed}s.\n" "========== Test stack trace starts here ==========\n" f"{buffer.getvalue()}" ) from None E TimeoutError: Test timeout (60) hit after 60.006166047998704s. E ========== Test stack trace starts here ========== E Stack for wait_for= cb=[shield.._clear_awaited_by_callback() at /usr/lib/python3.14/asyncio/tasks.py:978, _log_on_exception() at /usr/lib/python3.14/asyncio/tasks.py:914]> (most recent call last): E File "/build/python-distributed/src/distributed-2026.8.0/distributed/tests/test_failed_workers.py", line 263, in test_restart_scheduler E await s.restart(stimulus_id="test") distributed/utils_test.py:1009: TimeoutError ----------------------------- Captured stderr call ----------------------------- 2026-09-09 00:24:21,410 - distributed.scheduler - INFO - State start 2026-09-09 00:24:21,458 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:43115 2026-09-09 00:24:21,466 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:35999/status 2026-09-09 00:24:21,476 - distributed.scheduler - INFO - Registering Worker plugin shuffle 2026-09-09 00:24:21,586 - distributed.nanny - INFO - Start Nanny at: 'tcp://127.0.0.1:40875' 2026-09-09 00:24:21,611 - distributed.nanny - INFO - Start Nanny at: 'tcp://127.0.0.1:38743' 2026-09-09 00:24:32,843 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:46085 2026-09-09 00:24:32,846 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:46085 2026-09-09 00:24:32,847 - distributed.worker - INFO - Worker name: 1 2026-09-09 00:24:32,849 - distributed.worker - INFO - dashboard at: 127.0.0.1:44283 2026-09-09 00:24:32,849 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:43115 2026-09-09 00:24:32,850 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:24:32,851 - distributed.worker - INFO - Threads: 2 2026-09-09 00:24:32,852 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:24:32,852 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-4crk7fyg 2026-09-09 00:24:32,854 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:24:35,609 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:33389 2026-09-09 00:24:35,617 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:33389 2026-09-09 00:24:35,618 - distributed.worker - INFO - Worker name: 0 2026-09-09 00:24:35,620 - distributed.worker - INFO - dashboard at: 127.0.0.1:35607 2026-09-09 00:24:35,621 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:43115 2026-09-09 00:24:35,621 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:24:35,622 - distributed.worker - INFO - Threads: 1 2026-09-09 00:24:35,622 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:24:35,623 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-0n49z8f6 2026-09-09 00:24:35,624 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:24:58,452 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:46085 name: 1 2026-09-09 00:24:58,469 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:46085 2026-09-09 00:24:58,479 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:34346 2026-09-09 00:24:58,481 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:24:58,489 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:43115 2026-09-09 00:24:58,491 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:24:58,501 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:43115 2026-09-09 00:25:04,956 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:33389 name: 0 2026-09-09 00:25:04,967 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:33389 2026-09-09 00:25:04,975 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:34360 2026-09-09 00:25:04,989 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:25:05,009 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:43115 2026-09-09 00:25:05,010 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:25:05,032 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:43115 2026-09-09 00:25:05,043 - distributed.scheduler - INFO - Restarting workers and releasing all keys (stimulus_id='test') 2026-09-09 00:25:05,051 - distributed.scheduler - INFO - Restarting all workers (stimulus_id='test' 2026-09-09 00:25:05,105 - distributed.nanny - INFO - Nanny asking worker to close. Reason: test 2026-09-09 00:25:05,109 - distributed.nanny - INFO - Nanny asking worker to close. Reason: test 2026-09-09 00:25:05,118 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:46085. Reason: test 2026-09-09 00:25:05,123 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:25:05,125 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:33389. Reason: test 2026-09-09 00:25:05,130 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:25:05,140 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:34346; closing. 2026-09-09 00:25:05,143 - distributed.core - INFO - Connection to tcp://127.0.0.1:43115 has been closed. 2026-09-09 00:25:05,144 - distributed.core - INFO - Connection to tcp://127.0.0.1:43115 has been closed. 2026-09-09 00:25:05,144 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:46085 name: 1 (stimulus_id='handle-worker-cleanup-1788913505.1437945') 2026-09-09 00:25:05,158 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:34360; closing. 2026-09-09 00:25:05,160 - distributed.nanny - INFO - Worker closed 2026-09-09 00:25:05,161 - distributed.nanny - INFO - Worker closed 2026-09-09 00:25:05,162 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:33389 name: 0 (stimulus_id='handle-worker-cleanup-1788913505.1616445') 2026-09-09 00:25:05,170 - distributed.scheduler - INFO - Lost all workers 2026-09-09 00:25:05,944 - distributed.nanny - INFO - Unregistering worker (status=Status.running) 2026-09-09 00:25:05,970 - distributed.nanny - WARNING - Restarting worker (status=Status.running) 2026-09-09 00:25:06,102 - distributed.nanny - INFO - Unregistering worker (status=Status.running) 2026-09-09 00:25:06,144 - distributed.nanny - WARNING - Restarting worker (status=Status.running) 2026-09-09 00:25:15,555 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:36449 2026-09-09 00:25:15,560 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:36449 2026-09-09 00:25:15,561 - distributed.worker - INFO - Worker name: 0 2026-09-09 00:25:15,563 - distributed.worker - INFO - dashboard at: 127.0.0.1:39265 2026-09-09 00:25:15,563 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:43115 2026-09-09 00:25:15,564 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:25:15,564 - distributed.worker - INFO - Threads: 1 2026-09-09 00:25:15,565 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:25:15,565 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-i88lopnj 2026-09-09 00:25:15,566 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:25:18,043 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:34059 2026-09-09 00:25:18,046 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:34059 2026-09-09 00:25:18,048 - distributed.worker - INFO - Worker name: 1 2026-09-09 00:25:18,049 - distributed.worker - INFO - dashboard at: 127.0.0.1:45025 2026-09-09 00:25:18,050 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:43115 2026-09-09 00:25:18,051 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:25:18,051 - distributed.worker - INFO - Threads: 2 2026-09-09 00:25:18,052 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:25:18,053 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-fnd1m08b 2026-09-09 00:25:18,054 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:25:21,412 - distributed.scheduler - ERROR - Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6629, in restart_workers await asyncio.sleep(0.2) File "/usr/lib/python3.14/asyncio/tasks.py", line 705, in sleep return await future ^^^^^^^^^^^^ asyncio.exceptions.CancelledError 2026-09-09 00:25:21,438 - distributed.scheduler - ERROR - Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6629, in restart_workers await asyncio.sleep(0.2) File "/usr/lib/python3.14/asyncio/tasks.py", line 705, in sleep return await future ^^^^^^^^^^^^ asyncio.exceptions.CancelledError 2026-09-09 00:25:21,467 - distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:40875'. Reason: nanny-close 2026-09-09 00:25:21,471 - distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:38743'. Reason: nanny-close 2026-09-09 00:25:32,872 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:36449 name: 0 2026-09-09 00:25:32,889 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:36449 2026-09-09 00:25:32,898 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:43484 2026-09-09 00:25:32,900 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:25:32,912 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:43115 2026-09-09 00:25:32,913 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:25:32,924 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:43115 2026-09-09 00:25:32,947 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close 2026-09-09 00:25:32,959 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:36449. Reason: nanny-close 2026-09-09 00:25:32,963 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:25:32,982 - distributed.core - INFO - Connection to tcp://127.0.0.1:43115 has been closed. 2026-09-09 00:25:32,982 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:43484; closing. 2026-09-09 00:25:33,008 - distributed.nanny - INFO - Worker closed 2026-09-09 00:25:33,004 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:36449 name: 0 (stimulus_id='handle-worker-cleanup-1788913533.003798') 2026-09-09 00:25:33,015 - distributed.scheduler - INFO - Lost all workers 2026-09-09 00:25:36,945 - distributed.nanny - WARNING - Worker process still alive after 4.0 seconds, killing 2026-09-09 00:25:37,073 - distributed.nanny - INFO - Worker process 13607 was killed by signal 9 2026-09-09 00:25:37,090 - distributed.nanny - INFO - Nanny at 'tcp://127.0.0.1:40875' closed. 2026-09-09 00:25:39,395 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:34059 name: 1 2026-09-09 00:25:39,408 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:34059 2026-09-09 00:25:39,417 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:25:39,417 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:43492 2026-09-09 00:25:39,425 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:43115 2026-09-09 00:25:39,426 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:25:39,434 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:43115 2026-09-09 00:25:39,475 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close 2026-09-09 00:25:39,487 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:34059. Reason: nanny-close 2026-09-09 00:25:39,491 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:25:39,505 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:43492; closing. 2026-09-09 00:25:39,506 - distributed.core - INFO - Connection to tcp://127.0.0.1:43115 has been closed. 2026-09-09 00:25:39,510 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:34059 name: 1 (stimulus_id='handle-worker-cleanup-1788913539.509513') 2026-09-09 00:25:39,520 - distributed.nanny - INFO - Worker closed 2026-09-09 00:25:39,521 - distributed.scheduler - INFO - Lost all workers 2026-09-09 00:25:40,666 - distributed.nanny - INFO - Nanny at 'tcp://127.0.0.1:38743' closed. 2026-09-09 00:25:40,671 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown 2026-09-09 00:25:40,686 - distributed.scheduler - INFO - Scheduler closing all comms ------------------------------ Captured log call ------------------------------- INFO distributed.scheduler:scheduler.py:1735 State start INFO distributed.scheduler:scheduler.py:4436 Scheduler at: tcp://127.0.0.1:43115 INFO distributed.scheduler:scheduler.py:4451 dashboard at: http://127.0.0.1:35999/status INFO distributed.scheduler:scheduler.py:8137 Registering Worker plugin shuffle INFO distributed.nanny:nanny.py:350 Start Nanny at: 'tcp://127.0.0.1:40875' INFO distributed.nanny:nanny.py:350 Start Nanny at: 'tcp://127.0.0.1:38743' INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:46085 name: 1 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:46085 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:34346 INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:33389 name: 0 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:33389 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:34360 INFO distributed.scheduler:scheduler.py:6449 Restarting workers and releasing all keys (stimulus_id='test') INFO distributed.scheduler:scheduler.py:6526 Restarting all workers (stimulus_id='test' INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: test INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: test INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:34346; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:46085 name: 1 (stimulus_id='handle-worker-cleanup-1788913505.1437945') INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:34360; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:33389 name: 0 (stimulus_id='handle-worker-cleanup-1788913505.1616445') INFO distributed.scheduler:scheduler.py:5729 Lost all workers INFO distributed.nanny:nanny.py:528 Unregistering worker (status=Status.running) WARNING distributed.nanny:nanny.py:546 Restarting worker (status=Status.running) INFO distributed.nanny:nanny.py:528 Unregistering worker (status=Status.running) WARNING distributed.nanny:nanny.py:546 Restarting worker (status=Status.running) ERROR distributed.scheduler:utils.py:851 Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6629, in restart_workers await asyncio.sleep(0.2) File "/usr/lib/python3.14/asyncio/tasks.py", line 705, in sleep return await future ^^^^^^^^^^^^ asyncio.exceptions.CancelledError ERROR distributed.scheduler:utils.py:851 Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6629, in restart_workers await asyncio.sleep(0.2) File "/usr/lib/python3.14/asyncio/tasks.py", line 705, in sleep return await future ^^^^^^^^^^^^ asyncio.exceptions.CancelledError INFO distributed.nanny:nanny.py:587 Closing Nanny at 'tcp://127.0.0.1:40875'. Reason: nanny-close INFO distributed.nanny:nanny.py:587 Closing Nanny at 'tcp://127.0.0.1:38743'. Reason: nanny-close INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:36449 name: 0 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:36449 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:43484 INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: nanny-close INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:43484; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:36449 name: 0 (stimulus_id='handle-worker-cleanup-1788913533.003798') INFO distributed.scheduler:scheduler.py:5729 Lost all workers WARNING distributed.nanny:nanny.py:880 Worker process still alive after 4.0 seconds, killing INFO distributed.nanny:nanny.py:807 Worker process 13607 was killed by signal 9 INFO distributed.nanny:nanny.py:606 Nanny at 'tcp://127.0.0.1:40875' closed. INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:34059 name: 1 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:34059 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:43492 INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: nanny-close INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:43492; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:34059 name: 1 (stimulus_id='handle-worker-cleanup-1788913539.509513') INFO distributed.scheduler:scheduler.py:5729 Lost all workers INFO distributed.nanny:nanny.py:606 Nanny at 'tcp://127.0.0.1:38743' closed. INFO distributed.scheduler:scheduler.py:4493 Closing scheduler. Reason: unknown INFO distributed.scheduler:scheduler.py:4521 Scheduler closing all comms _______________ test_forgotten_futures_dont_clean_up_new_futures _______________ c = s = a = , b = @gen_cluster(Worker=Nanny, client=True, timeout=60) async def test_forgotten_futures_dont_clean_up_new_futures(c, s, a, b): x = c.submit(inc, 1) > await c.restart() distributed/tests/test_failed_workers.py:275: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ distributed/client.py:3879: in _restart await self.scheduler.restart( distributed/core.py:1212: in send_recv_from_rpc return await send_recv(comm=comm, op=key, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ distributed/core.py:1007: in send_recv raise exc.with_traceback(tb) distributed/core.py:798: in _handle_comm result = await result ^^^^^^^^^^^^^^^^^ distributed/utils.py:826: in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ distributed/scheduler.py:6467: in restart await self.restart_workers( ^^^^^^^^^^^^^^^^^ distributed/utils.py:826: in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > raise TimeoutError(msg) ^^^^^^^^^^^^^^^^^ E TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. distributed/scheduler.py:6650: TimeoutError During handling of the above exception, another exception occurred: fut = ._..test_func..async_fn at 0x3eafd8c5e0> timeout = 120 async def wait_for(fut: Awaitable[T], timeout: float) -> T: async with asyncio.timeout(timeout): > return await fut ^^^^^^^^^ distributed/utils.py:1855: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ distributed/utils_test.py:999: in async_fn await asyncio.sleep(0.01) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ delay = 0.01, result = None async def sleep(delay, result=None): """Coroutine that completes after a given time (in seconds).""" if delay <= 0: await __sleep0() return result if math.isnan(delay): raise ValueError("Invalid delay: NaN (not a number)") loop = events.get_running_loop() future = loop.create_future() h = loop.call_later(delay, futures._set_result_unless_cancelled, future, result) try: > return await future ^^^^^^^^^^^^ E asyncio.exceptions.CancelledError /usr/lib/python3.14/asyncio/tasks.py:705: CancelledError The above exception was the direct cause of the following exception: > lambda: ihook(item=item, **kwds), when=when, reraise=reraise ^^^^^^^^^^^^^^^^^^^^^^^^ ) /usr/lib/python3.14/site-packages/flaky/flaky_pytest_plugin.py:146: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python3.14/contextlib.py:85: in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/contextlib.py:85: in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ distributed/utils_test.py:1058: in test_func return _run_and_close_tornado(async_fn_outer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ distributed/utils_test.py:380: in _run_and_close_tornado return asyncio_run(inner_fn(), loop_factory=get_loop_factory()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/runners.py:205: in run return runner.run(main) ^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/runners.py:128: in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/base_events.py:720: in run_until_complete return future.result() ^^^^^^^^^^^^^^^ distributed/utils_test.py:377: in inner_fn return await async_fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ distributed/utils_test.py:1055: in async_fn_outer return await utils_wait_for(async_fn(), timeout=timeout * 2) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ distributed/utils.py:1854: in wait_for async with asyncio.timeout(timeout): ^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = exc_type = exc_val = CancelledError(), exc_tb = async def __aexit__( self, exc_type: type[BaseException] | None, exc_val: BaseException | None, exc_tb: TracebackType | None, ) -> bool | None: assert self._state in (_State.ENTERED, _State.EXPIRING) if self._timeout_handler is not None: self._timeout_handler.cancel() self._timeout_handler = None if self._state is _State.EXPIRING: self._state = _State.EXPIRED if self._task.uncancel() <= self._cancelling and exc_type is not None: # Since there are no new cancel requests, we're # handling this. if issubclass(exc_type, exceptions.CancelledError): > raise TimeoutError from exc_val E TimeoutError /usr/lib/python3.14/asyncio/timeouts.py:115: TimeoutError ----------------------------- Captured stderr call ----------------------------- 2026-09-09 00:25:42,674 - distributed.scheduler - INFO - State start 2026-09-09 00:25:42,735 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:33761 2026-09-09 00:25:42,745 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:42777/status 2026-09-09 00:25:42,755 - distributed.scheduler - INFO - Registering Worker plugin shuffle 2026-09-09 00:25:42,885 - distributed.nanny - INFO - Start Nanny at: 'tcp://127.0.0.1:41631' 2026-09-09 00:25:42,919 - distributed.nanny - INFO - Start Nanny at: 'tcp://127.0.0.1:39139' 2026-09-09 00:25:51,529 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:40245 2026-09-09 00:25:51,533 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:40245 2026-09-09 00:25:51,534 - distributed.worker - INFO - Worker name: 0 2026-09-09 00:25:51,536 - distributed.worker - INFO - dashboard at: 127.0.0.1:39273 2026-09-09 00:25:51,536 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:33761 2026-09-09 00:25:51,537 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:25:51,537 - distributed.worker - INFO - Threads: 1 2026-09-09 00:25:51,538 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:25:51,539 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-1lkbpf0p 2026-09-09 00:25:51,540 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:25:51,818 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:46235 2026-09-09 00:25:51,820 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:46235 2026-09-09 00:25:51,821 - distributed.worker - INFO - Worker name: 1 2026-09-09 00:25:51,822 - distributed.worker - INFO - dashboard at: 127.0.0.1:42301 2026-09-09 00:25:51,822 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:33761 2026-09-09 00:25:51,823 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:25:51,823 - distributed.worker - INFO - Threads: 2 2026-09-09 00:25:51,824 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:25:51,824 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-2f7kubdi 2026-09-09 00:25:51,825 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:26:07,240 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:40245 name: 0 2026-09-09 00:26:07,267 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:26:07,259 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:40245 2026-09-09 00:26:07,270 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:33482 2026-09-09 00:26:07,275 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:33761 2026-09-09 00:26:07,275 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:26:07,282 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:33761 2026-09-09 00:26:15,575 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:46235 name: 1 2026-09-09 00:26:15,591 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:46235 2026-09-09 00:26:15,599 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:33486 2026-09-09 00:26:15,607 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:26:15,618 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:33761 2026-09-09 00:26:15,619 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:26:15,628 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:33761 2026-09-09 00:26:15,735 - distributed.scheduler - INFO - Receive client connection: Client-119d13cf-abe5-11f1-86b4-8f07734cb81d 2026-09-09 00:26:15,746 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:52190 2026-09-09 00:26:15,810 - distributed.scheduler - INFO - Restarting workers and releasing all keys (stimulus_id='client-restart-1788913575.79458') 2026-09-09 00:26:15,820 - distributed.scheduler - INFO - Restarting all workers (stimulus_id='client-restart-1788913575.79458' 2026-09-09 00:26:15,941 - distributed.nanny - INFO - Nanny asking worker to close. Reason: client-restart-1788913575.79458 2026-09-09 00:26:15,949 - distributed.nanny - INFO - Nanny asking worker to close. Reason: client-restart-1788913575.79458 2026-09-09 00:26:15,959 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:46235. Reason: client-restart-1788913575.79458 2026-09-09 00:26:15,965 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:26:15,982 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:33486; closing. 2026-09-09 00:26:15,984 - distributed.core - INFO - Connection to tcp://127.0.0.1:33761 has been closed. 2026-09-09 00:26:15,986 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:46235 name: 1 (stimulus_id='handle-worker-cleanup-1788913575.986165') 2026-09-09 00:26:16,003 - distributed.nanny - INFO - Worker closed 2026-09-09 00:26:17,061 - distributed.nanny - INFO - Unregistering worker (status=Status.running) 2026-09-09 00:26:17,097 - distributed.nanny - WARNING - Restarting worker (status=Status.running) 2026-09-09 00:26:19,947 - distributed.nanny - WARNING - Worker process still alive after 4.0 seconds, killing 2026-09-09 00:26:20,020 - distributed.core - INFO - Connection to tcp://127.0.0.1:33482 has been closed. 2026-09-09 00:26:20,022 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:40245 name: 0 (stimulus_id='handle-worker-cleanup-1788913580.0219257') 2026-09-09 00:26:20,036 - distributed.scheduler - INFO - Lost all workers 2026-09-09 00:26:20,078 - distributed.nanny - INFO - Worker process 13624 was killed by signal 9 2026-09-09 00:26:20,096 - distributed.nanny - INFO - Unregistering worker (status=Status.running) 2026-09-09 00:26:20,151 - distributed.nanny - WARNING - Restarting worker (status=Status.running) 2026-09-09 00:26:27,488 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:44323 2026-09-09 00:26:27,491 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:44323 2026-09-09 00:26:27,492 - distributed.worker - INFO - Worker name: 1 2026-09-09 00:26:27,495 - distributed.worker - INFO - dashboard at: 127.0.0.1:35907 2026-09-09 00:26:27,496 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:33761 2026-09-09 00:26:27,497 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:26:27,498 - distributed.worker - INFO - Threads: 2 2026-09-09 00:26:27,499 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:26:27,500 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-barnz8fl 2026-09-09 00:26:27,502 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:26:35,359 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:39943 2026-09-09 00:26:35,366 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:39943 2026-09-09 00:26:35,367 - distributed.worker - INFO - Worker name: 0 2026-09-09 00:26:35,370 - distributed.worker - INFO - dashboard at: 127.0.0.1:43617 2026-09-09 00:26:35,371 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:33761 2026-09-09 00:26:35,371 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:26:35,372 - distributed.worker - INFO - Threads: 1 2026-09-09 00:26:35,374 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 00:26:35,375 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-3wl1_iwg 2026-09-09 00:26:35,376 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:26:35,906 - distributed.scheduler - ERROR - Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. 2026-09-09 00:26:35,931 - distributed.scheduler - ERROR - Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. 2026-09-09 00:26:35,954 - distributed.core - ERROR - Exception while handling op restart Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 798, in _handle_comm result = await result ^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. 2026-09-09 00:26:44,232 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:44323 name: 1 2026-09-09 00:26:44,249 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:44323 2026-09-09 00:26:44,259 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:41172 2026-09-09 00:26:44,344 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:26:44,402 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:33761 2026-09-09 00:26:44,404 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:26:44,448 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:33761 2026-09-09 00:27:00,175 - distributed.core - INFO - Event loop was unresponsive in Worker for 15.77s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. 2026-09-09 00:27:04,701 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:39943 name: 0 2026-09-09 00:27:04,718 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:39943 2026-09-09 00:27:04,727 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:34722 2026-09-09 00:27:04,745 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 00:27:04,761 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:33761 2026-09-09 00:27:04,763 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 00:27:04,779 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:33761 2026-09-09 00:27:42,678 - distributed.scheduler - INFO - Remove client Client-119d13cf-abe5-11f1-86b4-8f07734cb81d 2026-09-09 00:27:42,694 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:52190; closing. 2026-09-09 00:27:42,700 - distributed.scheduler - INFO - Remove client Client-119d13cf-abe5-11f1-86b4-8f07734cb81d 2026-09-09 00:27:42,718 - distributed.scheduler - INFO - Close client connection: Client-119d13cf-abe5-11f1-86b4-8f07734cb81d 2026-09-09 00:27:42,755 - distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:41631'. Reason: nanny-close 2026-09-09 00:27:42,758 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close 2026-09-09 00:27:42,759 - distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:39139'. Reason: nanny-close 2026-09-09 00:27:42,764 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close 2026-09-09 00:27:42,773 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:39943. Reason: nanny-close 2026-09-09 00:27:42,783 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:44323. Reason: nanny-close 2026-09-09 00:27:42,798 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:27:42,801 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 00:27:42,827 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:34722; closing. 2026-09-09 00:27:42,829 - distributed.core - INFO - Connection to tcp://127.0.0.1:33761 has been closed. 2026-09-09 00:27:42,830 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:39943 name: 0 (stimulus_id='handle-worker-cleanup-1788913662.8303766') 2026-09-09 00:27:42,834 - distributed.core - INFO - Connection to tcp://127.0.0.1:33761 has been closed. 2026-09-09 00:27:42,843 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:41172; closing. 2026-09-09 00:27:42,851 - distributed.nanny - INFO - Worker closed 2026-09-09 00:27:42,849 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:44323 name: 1 (stimulus_id='handle-worker-cleanup-1788913662.848527') 2026-09-09 00:27:42,862 - distributed.nanny - INFO - Worker closed 2026-09-09 00:27:42,857 - distributed.scheduler - INFO - Lost all workers 2026-09-09 00:27:43,866 - distributed.nanny - INFO - Nanny at 'tcp://127.0.0.1:41631' closed. 2026-09-09 00:27:44,183 - distributed.nanny - INFO - Nanny at 'tcp://127.0.0.1:39139' closed. 2026-09-09 00:27:44,187 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown 2026-09-09 00:27:44,198 - distributed.scheduler - INFO - Scheduler closing all comms ------------------------------ Captured log call ------------------------------- INFO distributed.scheduler:scheduler.py:1735 State start INFO distributed.scheduler:scheduler.py:4436 Scheduler at: tcp://127.0.0.1:33761 INFO distributed.scheduler:scheduler.py:4451 dashboard at: http://127.0.0.1:42777/status INFO distributed.scheduler:scheduler.py:8137 Registering Worker plugin shuffle INFO distributed.nanny:nanny.py:350 Start Nanny at: 'tcp://127.0.0.1:41631' INFO distributed.nanny:nanny.py:350 Start Nanny at: 'tcp://127.0.0.1:39139' INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:40245 name: 0 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:40245 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:33482 INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:46235 name: 1 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:46235 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:33486 INFO distributed.scheduler:scheduler.py:5916 Receive client connection: Client-119d13cf-abe5-11f1-86b4-8f07734cb81d INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:52190 INFO distributed.scheduler:scheduler.py:6449 Restarting workers and releasing all keys (stimulus_id='client-restart-1788913575.79458') INFO distributed.scheduler:scheduler.py:6526 Restarting all workers (stimulus_id='client-restart-1788913575.79458' INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: client-restart-1788913575.79458 INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: client-restart-1788913575.79458 INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:33486; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:46235 name: 1 (stimulus_id='handle-worker-cleanup-1788913575.986165') INFO distributed.nanny:nanny.py:528 Unregistering worker (status=Status.running) WARNING distributed.nanny:nanny.py:546 Restarting worker (status=Status.running) WARNING distributed.nanny:nanny.py:880 Worker process still alive after 4.0 seconds, killing INFO distributed.core:core.py:857 Connection to tcp://127.0.0.1:33482 has been closed. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:40245 name: 0 (stimulus_id='handle-worker-cleanup-1788913580.0219257') INFO distributed.scheduler:scheduler.py:5729 Lost all workers INFO distributed.nanny:nanny.py:807 Worker process 13624 was killed by signal 9 INFO distributed.nanny:nanny.py:528 Unregistering worker (status=Status.running) WARNING distributed.nanny:nanny.py:546 Restarting worker (status=Status.running) ERROR distributed.scheduler:utils.py:851 Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. ERROR distributed.scheduler:utils.py:851 Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. ERROR distributed.core:core.py:808 Exception while handling op restart Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 798, in _handle_comm result = await result ^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6467, in restart await self.restart_workers( ...<4 lines>... ) File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils.py", line 826, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/scheduler.py", line 6650, in restart_workers raise TimeoutError(msg) TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 20s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:44323 name: 1 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:44323 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:41172 INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:39943 name: 0 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:39943 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:34722 INFO distributed.scheduler:scheduler.py:5961 Remove client Client-119d13cf-abe5-11f1-86b4-8f07734cb81d INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:52190; closing. INFO distributed.scheduler:scheduler.py:5961 Remove client Client-119d13cf-abe5-11f1-86b4-8f07734cb81d INFO distributed.scheduler:scheduler.py:5953 Close client connection: Client-119d13cf-abe5-11f1-86b4-8f07734cb81d INFO distributed.nanny:nanny.py:587 Closing Nanny at 'tcp://127.0.0.1:41631'. Reason: nanny-close INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: nanny-close INFO distributed.nanny:nanny.py:587 Closing Nanny at 'tcp://127.0.0.1:39139'. Reason: nanny-close INFO distributed.nanny:nanny.py:859 Nanny asking worker to close. Reason: nanny-close INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:34722; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:39943 name: 0 (stimulus_id='handle-worker-cleanup-1788913662.8303766') INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:41172; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:44323 name: 1 (stimulus_id='handle-worker-cleanup-1788913662.848527') INFO distributed.scheduler:scheduler.py:5729 Lost all workers INFO distributed.nanny:nanny.py:606 Nanny at 'tcp://127.0.0.1:41631' closed. INFO distributed.nanny:nanny.py:606 Nanny at 'tcp://127.0.0.1:39139' closed. INFO distributed.scheduler:scheduler.py:4493 Closing scheduler. Reason: unknown INFO distributed.scheduler:scheduler.py:4521 Scheduler closing all comms ___________________ test_dont_steal_fast_tasks_compute_time ____________________ c = s = workers = (, , ...) @gen_cluster( client=True, nthreads=[("127.0.0.1", 1)] * 10, config={"distributed.scheduler.work-stealing-interval": "100ms"}, ) async def test_dont_steal_fast_tasks_compute_time(c, s, *workers): def do_nothing(x, y=None): pass xs = c.map(do_nothing, range(10), workers=workers[0].address) await wait(xs) futures = c.map(do_nothing, range(100), y=xs) await wait(futures) > assert len(set.union(*(s.tasks[x.key].who_has for x in xs))) == 1 E AssertionError: assert 10 == 1 E + where 10 = len({, , , , , , , , , }) E + where {, , , , , , , , , } = (*. at 0x3e8e11e420>) E + where = set.union distributed/tests/test_steal.py:338: AssertionError ----------------------------- Captured stderr call ----------------------------- 2026-09-09 01:13:57,311 - distributed.scheduler - INFO - State start 2026-09-09 01:13:57,366 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:44073 2026-09-09 01:13:57,378 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:37547/status 2026-09-09 01:13:57,390 - distributed.scheduler - INFO - Registering Worker plugin shuffle 2026-09-09 01:13:57,843 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:45555 2026-09-09 01:13:57,853 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:45555 2026-09-09 01:13:57,863 - distributed.worker - INFO - Worker name: 0 2026-09-09 01:13:57,873 - distributed.worker - INFO - dashboard at: 127.0.0.1:38615 2026-09-09 01:13:57,882 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:44073 2026-09-09 01:13:57,892 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:57,902 - distributed.worker - INFO - Threads: 1 2026-09-09 01:13:57,912 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 01:13:57,921 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-_w4lm_g6 2026-09-09 01:13:57,931 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:57,952 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:45173 2026-09-09 01:13:57,965 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:45173 2026-09-09 01:13:57,979 - distributed.worker - INFO - Worker name: 1 2026-09-09 01:13:57,991 - distributed.worker - INFO - dashboard at: 127.0.0.1:37913 2026-09-09 01:13:58,004 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:44073 2026-09-09 01:13:58,018 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:58,032 - distributed.worker - INFO - Threads: 1 2026-09-09 01:13:58,044 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 01:13:58,057 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-mq5jvv5b 2026-09-09 01:13:58,070 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:58,098 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:33753 2026-09-09 01:13:58,111 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:33753 2026-09-09 01:13:58,124 - distributed.worker - INFO - Worker name: 2 2026-09-09 01:13:58,136 - distributed.worker - INFO - dashboard at: 127.0.0.1:39363 2026-09-09 01:13:58,148 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:44073 2026-09-09 01:13:58,160 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:58,172 - distributed.worker - INFO - Threads: 1 2026-09-09 01:13:58,184 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 01:13:58,196 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-8w3fh_os 2026-09-09 01:13:58,209 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:58,233 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:35721 2026-09-09 01:13:58,243 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:35721 2026-09-09 01:13:58,253 - distributed.worker - INFO - Worker name: 3 2026-09-09 01:13:58,264 - distributed.worker - INFO - dashboard at: 127.0.0.1:44749 2026-09-09 01:13:58,274 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:44073 2026-09-09 01:13:58,284 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:58,294 - distributed.worker - INFO - Threads: 1 2026-09-09 01:13:58,303 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 01:13:58,313 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-d6sp_ask 2026-09-09 01:13:58,322 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:58,341 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:42799 2026-09-09 01:13:58,351 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:42799 2026-09-09 01:13:58,361 - distributed.worker - INFO - Worker name: 4 2026-09-09 01:13:58,371 - distributed.worker - INFO - dashboard at: 127.0.0.1:40433 2026-09-09 01:13:58,381 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:44073 2026-09-09 01:13:58,391 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:58,400 - distributed.worker - INFO - Threads: 1 2026-09-09 01:13:58,410 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 01:13:58,421 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-r39zymjt 2026-09-09 01:13:58,431 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:58,451 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:46045 2026-09-09 01:13:58,462 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:46045 2026-09-09 01:13:58,473 - distributed.worker - INFO - Worker name: 5 2026-09-09 01:13:58,484 - distributed.worker - INFO - dashboard at: 127.0.0.1:41005 2026-09-09 01:13:58,496 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:44073 2026-09-09 01:13:58,506 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:58,516 - distributed.worker - INFO - Threads: 1 2026-09-09 01:13:58,527 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 01:13:58,537 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-nct5g18s 2026-09-09 01:13:58,548 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:58,570 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:34311 2026-09-09 01:13:58,582 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:34311 2026-09-09 01:13:58,593 - distributed.worker - INFO - Worker name: 6 2026-09-09 01:13:58,603 - distributed.worker - INFO - dashboard at: 127.0.0.1:38277 2026-09-09 01:13:58,613 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:44073 2026-09-09 01:13:58,651 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:58,662 - distributed.worker - INFO - Threads: 1 2026-09-09 01:13:58,673 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 01:13:58,683 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-ejtq3c4f 2026-09-09 01:13:58,694 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:58,720 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:46433 2026-09-09 01:13:58,731 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:46433 2026-09-09 01:13:58,741 - distributed.worker - INFO - Worker name: 7 2026-09-09 01:13:58,752 - distributed.worker - INFO - dashboard at: 127.0.0.1:42767 2026-09-09 01:13:58,762 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:44073 2026-09-09 01:13:58,772 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:58,782 - distributed.worker - INFO - Threads: 1 2026-09-09 01:13:58,791 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 01:13:58,800 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-ozv0fdsg 2026-09-09 01:13:58,811 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:58,835 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:46525 2026-09-09 01:13:58,847 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:46525 2026-09-09 01:13:58,859 - distributed.worker - INFO - Worker name: 8 2026-09-09 01:13:58,871 - distributed.worker - INFO - dashboard at: 127.0.0.1:44957 2026-09-09 01:13:58,883 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:44073 2026-09-09 01:13:58,894 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:58,906 - distributed.worker - INFO - Threads: 1 2026-09-09 01:13:58,920 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 01:13:58,932 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-r29p27f9 2026-09-09 01:13:58,946 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:58,970 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:41159 2026-09-09 01:13:58,982 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:41159 2026-09-09 01:13:58,994 - distributed.worker - INFO - Worker name: 9 2026-09-09 01:13:59,007 - distributed.worker - INFO - dashboard at: 127.0.0.1:34107 2026-09-09 01:13:59,021 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:44073 2026-09-09 01:13:59,037 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:59,052 - distributed.worker - INFO - Threads: 1 2026-09-09 01:13:59,068 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 01:13:59,082 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-uzbp6x0r 2026-09-09 01:13:59,095 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:13:59,319 - distributed.comm.tcp - INFO - Connection from tcp://127.0.0.1:45738 closed before handshake completed 2026-09-09 01:13:59,321 - distributed.comm.tcp - INFO - Connection from tcp://127.0.0.1:45746 closed before handshake completed 2026-09-09 01:13:59,365 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:33753 name: 2 2026-09-09 01:13:59,381 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:33753 2026-09-09 01:13:59,393 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45750 2026-09-09 01:13:59,401 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:35721 name: 3 2026-09-09 01:13:59,416 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:35721 2026-09-09 01:13:59,428 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45764 2026-09-09 01:13:59,436 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:42799 name: 4 2026-09-09 01:13:59,451 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:42799 2026-09-09 01:13:59,463 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45768 2026-09-09 01:13:59,470 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:46045 name: 5 2026-09-09 01:13:59,485 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:46045 2026-09-09 01:13:59,497 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45774 2026-09-09 01:13:59,578 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:34311 name: 6 2026-09-09 01:13:59,623 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:34311 2026-09-09 01:13:59,635 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45788 2026-09-09 01:13:59,647 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:46433 name: 7 2026-09-09 01:13:59,662 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:46433 2026-09-09 01:13:59,675 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45800 2026-09-09 01:13:59,683 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:46525 name: 8 2026-09-09 01:13:59,699 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:46525 2026-09-09 01:13:59,710 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45816 2026-09-09 01:13:59,718 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:41159 name: 9 2026-09-09 01:13:59,732 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:41159 2026-09-09 01:13:59,744 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45822 2026-09-09 01:13:59,859 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 01:13:59,879 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 01:13:59,895 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 01:13:59,911 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 01:13:59,926 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 01:13:59,940 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 01:13:59,953 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 01:13:59,968 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 01:14:00,009 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:45173 name: 1 2026-09-09 01:14:00,024 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:45173 2026-09-09 01:14:00,036 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45834 2026-09-09 01:14:00,044 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:45555 name: 0 2026-09-09 01:14:00,059 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:45555 2026-09-09 01:14:00,070 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45838 2026-09-09 01:14:00,076 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:44073 2026-09-09 01:14:00,087 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:00,102 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:44073 2026-09-09 01:14:00,114 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:00,126 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:44073 2026-09-09 01:14:00,136 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:00,148 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:44073 2026-09-09 01:14:00,158 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:00,170 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:44073 2026-09-09 01:14:00,179 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:00,191 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:44073 2026-09-09 01:14:00,200 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:00,212 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:44073 2026-09-09 01:14:00,222 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:00,234 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:44073 2026-09-09 01:14:00,244 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:00,311 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:44073 2026-09-09 01:14:00,313 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:44073 2026-09-09 01:14:00,315 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:44073 2026-09-09 01:14:00,317 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:44073 2026-09-09 01:14:00,319 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:44073 2026-09-09 01:14:00,321 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:44073 2026-09-09 01:14:00,323 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:44073 2026-09-09 01:14:00,325 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:44073 2026-09-09 01:14:00,330 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 01:14:00,344 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 01:14:00,382 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:44073 2026-09-09 01:14:00,391 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:00,401 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:44073 2026-09-09 01:14:00,410 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:00,445 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:44073 2026-09-09 01:14:00,448 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:44073 2026-09-09 01:14:00,601 - distributed.scheduler - INFO - Receive client connection: Client-bd2a41e7-abeb-11f1-86b4-8f07734cb81d 2026-09-09 01:14:00,618 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45846 2026-09-09 01:14:17,782 - distributed.scheduler - INFO - Remove client Client-bd2a41e7-abeb-11f1-86b4-8f07734cb81d 2026-09-09 01:14:17,884 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:45846; closing. 2026-09-09 01:14:18,126 - distributed.scheduler - INFO - Remove client Client-bd2a41e7-abeb-11f1-86b4-8f07734cb81d 2026-09-09 01:14:18,225 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:45555. Reason: worker-close 2026-09-09 01:14:18,239 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:45173. Reason: worker-close 2026-09-09 01:14:18,252 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:33753. Reason: worker-close 2026-09-09 01:14:18,265 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:35721. Reason: worker-close 2026-09-09 01:14:18,278 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:42799. Reason: worker-close 2026-09-09 01:14:18,290 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:46045. Reason: worker-close 2026-09-09 01:14:18,303 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:34311. Reason: worker-close 2026-09-09 01:14:18,316 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:46433. Reason: worker-close 2026-09-09 01:14:18,329 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:46525. Reason: worker-close 2026-09-09 01:14:18,342 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:41159. Reason: worker-close 2026-09-09 01:14:18,514 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 01:14:18,527 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 01:14:18,538 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 01:14:18,550 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 01:14:18,561 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 01:14:18,572 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 01:14:18,583 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 01:14:18,594 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 01:14:18,605 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 01:14:18,616 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 01:14:18,757 - distributed.scheduler - INFO - Close client connection: Client-bd2a41e7-abeb-11f1-86b4-8f07734cb81d 2026-09-09 01:14:19,217 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 226, in read frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/worker.py", line 1165, in heartbeat response = await retry_operation( ^^^^^^^^^^^^^^^^^^^^^^ ...<14 lines>... ) ^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 420, in retry_operation return await retry( ^^^^^^^^^^^^ ...<5 lines>... ) ^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 399, in retry return await coro() ^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1212, in send_recv_from_rpc return await send_recv(comm=comm, op=key, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 982, in send_recv response = await comm.read(deserializers=deserializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 237, in read convert_stream_closed_error(self, e) ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 137, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in : Stream is closed 2026-09-09 01:14:19,260 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 226, in read frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/worker.py", line 1165, in heartbeat response = await retry_operation( ^^^^^^^^^^^^^^^^^^^^^^ ...<14 lines>... ) ^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 420, in retry_operation return await retry( ^^^^^^^^^^^^ ...<5 lines>... ) ^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 399, in retry return await coro() ^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1212, in send_recv_from_rpc return await send_recv(comm=comm, op=key, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 982, in send_recv response = await comm.read(deserializers=deserializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 237, in read convert_stream_closed_error(self, e) ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 137, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in : Stream is closed 2026-09-09 01:14:19,410 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:45822; closing. 2026-09-09 01:14:19,415 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:45774; closing. 2026-09-09 01:14:19,420 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:45788; closing. 2026-09-09 01:14:19,425 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:45800; closing. 2026-09-09 01:14:19,431 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:45768; closing. 2026-09-09 01:14:19,437 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:45838; closing. 2026-09-09 01:14:19,442 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:45816; closing. 2026-09-09 01:14:19,448 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:45764; closing. 2026-09-09 01:14:19,453 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:45750; closing. 2026-09-09 01:14:19,458 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:45834; closing. 2026-09-09 01:14:19,480 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:41159 name: 9 (stimulus_id='handle-worker-cleanup-1788916459.4793055') 2026-09-09 01:14:19,495 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:46045 name: 5 (stimulus_id='handle-worker-cleanup-1788916459.495362') 2026-09-09 01:14:19,510 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:34311 name: 6 (stimulus_id='handle-worker-cleanup-1788916459.509631') 2026-09-09 01:14:19,524 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:46433 name: 7 (stimulus_id='handle-worker-cleanup-1788916459.5235069') 2026-09-09 01:14:19,538 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:42799 name: 4 (stimulus_id='handle-worker-cleanup-1788916459.5379431') 2026-09-09 01:14:19,556 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:45555 name: 0 (stimulus_id='handle-worker-cleanup-1788916459.555504') 2026-09-09 01:14:19,570 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:46525 name: 8 (stimulus_id='handle-worker-cleanup-1788916459.5700839') 2026-09-09 01:14:19,586 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:35721 name: 3 (stimulus_id='handle-worker-cleanup-1788916459.5855424') 2026-09-09 01:14:19,601 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:33753 name: 2 (stimulus_id='handle-worker-cleanup-1788916459.6012957') 2026-09-09 01:14:19,617 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:45173 name: 1 (stimulus_id='handle-worker-cleanup-1788916459.6166422') 2026-09-09 01:14:19,629 - distributed.scheduler - INFO - Lost all workers 2026-09-09 01:14:19,660 - distributed.core - INFO - Connection to tcp://127.0.0.1:44073 has been closed. 2026-09-09 01:14:19,670 - distributed.core - INFO - Connection to tcp://127.0.0.1:44073 has been closed. 2026-09-09 01:14:19,678 - distributed.core - INFO - Connection to tcp://127.0.0.1:44073 has been closed. 2026-09-09 01:14:19,686 - distributed.core - INFO - Connection to tcp://127.0.0.1:44073 has been closed. 2026-09-09 01:14:19,691 - distributed.core - INFO - Connection to tcp://127.0.0.1:44073 has been closed. 2026-09-09 01:14:19,695 - distributed.core - INFO - Connection to tcp://127.0.0.1:44073 has been closed. 2026-09-09 01:14:19,699 - distributed.core - INFO - Connection to tcp://127.0.0.1:44073 has been closed. 2026-09-09 01:14:19,703 - distributed.core - INFO - Connection to tcp://127.0.0.1:44073 has been closed. 2026-09-09 01:14:19,706 - distributed.core - INFO - Connection to tcp://127.0.0.1:44073 has been closed. 2026-09-09 01:14:19,711 - distributed.core - INFO - Connection to tcp://127.0.0.1:44073 has been closed. 2026-09-09 01:14:19,804 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown 2026-09-09 01:14:19,821 - distributed.scheduler - INFO - Scheduler closing all comms ------------------------------ Captured log call ------------------------------- INFO distributed.scheduler:scheduler.py:1735 State start INFO distributed.scheduler:scheduler.py:4436 Scheduler at: tcp://127.0.0.1:44073 INFO distributed.scheduler:scheduler.py:4451 dashboard at: http://127.0.0.1:37547/status INFO distributed.scheduler:scheduler.py:8137 Registering Worker plugin shuffle INFO distributed.worker:worker.py:1355 Start worker at: tcp://127.0.0.1:45555 INFO distributed.worker:worker.py:1356 Listening to: tcp://127.0.0.1:45555 INFO distributed.worker:worker.py:1359 Worker name: 0 INFO distributed.worker:worker.py:1361 dashboard at: 127.0.0.1:38615 INFO distributed.worker:worker.py:1362 Waiting to connect to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1363 ------------------------------------------------- INFO distributed.worker:worker.py:1364 Threads: 1 INFO distributed.worker:worker.py:1366 Memory: 250.90 GiB INFO distributed.worker:worker.py:1370 Local Directory: /tmp/dask-scratch-space/worker-_w4lm_g6 INFO distributed.worker:worker.py:1083 ------------------------------------------------- INFO distributed.worker:worker.py:1355 Start worker at: tcp://127.0.0.1:45173 INFO distributed.worker:worker.py:1356 Listening to: tcp://127.0.0.1:45173 INFO distributed.worker:worker.py:1359 Worker name: 1 INFO distributed.worker:worker.py:1361 dashboard at: 127.0.0.1:37913 INFO distributed.worker:worker.py:1362 Waiting to connect to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1363 ------------------------------------------------- INFO distributed.worker:worker.py:1364 Threads: 1 INFO distributed.worker:worker.py:1366 Memory: 250.90 GiB INFO distributed.worker:worker.py:1370 Local Directory: /tmp/dask-scratch-space/worker-mq5jvv5b INFO distributed.worker:worker.py:1083 ------------------------------------------------- INFO distributed.worker:worker.py:1355 Start worker at: tcp://127.0.0.1:33753 INFO distributed.worker:worker.py:1356 Listening to: tcp://127.0.0.1:33753 INFO distributed.worker:worker.py:1359 Worker name: 2 INFO distributed.worker:worker.py:1361 dashboard at: 127.0.0.1:39363 INFO distributed.worker:worker.py:1362 Waiting to connect to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1363 ------------------------------------------------- INFO distributed.worker:worker.py:1364 Threads: 1 INFO distributed.worker:worker.py:1366 Memory: 250.90 GiB INFO distributed.worker:worker.py:1370 Local Directory: /tmp/dask-scratch-space/worker-8w3fh_os INFO distributed.worker:worker.py:1083 ------------------------------------------------- INFO distributed.worker:worker.py:1355 Start worker at: tcp://127.0.0.1:35721 INFO distributed.worker:worker.py:1356 Listening to: tcp://127.0.0.1:35721 INFO distributed.worker:worker.py:1359 Worker name: 3 INFO distributed.worker:worker.py:1361 dashboard at: 127.0.0.1:44749 INFO distributed.worker:worker.py:1362 Waiting to connect to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1363 ------------------------------------------------- INFO distributed.worker:worker.py:1364 Threads: 1 INFO distributed.worker:worker.py:1366 Memory: 250.90 GiB INFO distributed.worker:worker.py:1370 Local Directory: /tmp/dask-scratch-space/worker-d6sp_ask INFO distributed.worker:worker.py:1083 ------------------------------------------------- INFO distributed.worker:worker.py:1355 Start worker at: tcp://127.0.0.1:42799 INFO distributed.worker:worker.py:1356 Listening to: tcp://127.0.0.1:42799 INFO distributed.worker:worker.py:1359 Worker name: 4 INFO distributed.worker:worker.py:1361 dashboard at: 127.0.0.1:40433 INFO distributed.worker:worker.py:1362 Waiting to connect to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1363 ------------------------------------------------- INFO distributed.worker:worker.py:1364 Threads: 1 INFO distributed.worker:worker.py:1366 Memory: 250.90 GiB INFO distributed.worker:worker.py:1370 Local Directory: /tmp/dask-scratch-space/worker-r39zymjt INFO distributed.worker:worker.py:1083 ------------------------------------------------- INFO distributed.worker:worker.py:1355 Start worker at: tcp://127.0.0.1:46045 INFO distributed.worker:worker.py:1356 Listening to: tcp://127.0.0.1:46045 INFO distributed.worker:worker.py:1359 Worker name: 5 INFO distributed.worker:worker.py:1361 dashboard at: 127.0.0.1:41005 INFO distributed.worker:worker.py:1362 Waiting to connect to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1363 ------------------------------------------------- INFO distributed.worker:worker.py:1364 Threads: 1 INFO distributed.worker:worker.py:1366 Memory: 250.90 GiB INFO distributed.worker:worker.py:1370 Local Directory: /tmp/dask-scratch-space/worker-nct5g18s INFO distributed.worker:worker.py:1083 ------------------------------------------------- INFO distributed.worker:worker.py:1355 Start worker at: tcp://127.0.0.1:34311 INFO distributed.worker:worker.py:1356 Listening to: tcp://127.0.0.1:34311 INFO distributed.worker:worker.py:1359 Worker name: 6 INFO distributed.worker:worker.py:1361 dashboard at: 127.0.0.1:38277 INFO distributed.worker:worker.py:1362 Waiting to connect to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1363 ------------------------------------------------- INFO distributed.worker:worker.py:1364 Threads: 1 INFO distributed.worker:worker.py:1366 Memory: 250.90 GiB INFO distributed.worker:worker.py:1370 Local Directory: /tmp/dask-scratch-space/worker-ejtq3c4f INFO distributed.worker:worker.py:1083 ------------------------------------------------- INFO distributed.worker:worker.py:1355 Start worker at: tcp://127.0.0.1:46433 INFO distributed.worker:worker.py:1356 Listening to: tcp://127.0.0.1:46433 INFO distributed.worker:worker.py:1359 Worker name: 7 INFO distributed.worker:worker.py:1361 dashboard at: 127.0.0.1:42767 INFO distributed.worker:worker.py:1362 Waiting to connect to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1363 ------------------------------------------------- INFO distributed.worker:worker.py:1364 Threads: 1 INFO distributed.worker:worker.py:1366 Memory: 250.90 GiB INFO distributed.worker:worker.py:1370 Local Directory: /tmp/dask-scratch-space/worker-ozv0fdsg INFO distributed.worker:worker.py:1083 ------------------------------------------------- INFO distributed.worker:worker.py:1355 Start worker at: tcp://127.0.0.1:46525 INFO distributed.worker:worker.py:1356 Listening to: tcp://127.0.0.1:46525 INFO distributed.worker:worker.py:1359 Worker name: 8 INFO distributed.worker:worker.py:1361 dashboard at: 127.0.0.1:44957 INFO distributed.worker:worker.py:1362 Waiting to connect to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1363 ------------------------------------------------- INFO distributed.worker:worker.py:1364 Threads: 1 INFO distributed.worker:worker.py:1366 Memory: 250.90 GiB INFO distributed.worker:worker.py:1370 Local Directory: /tmp/dask-scratch-space/worker-r29p27f9 INFO distributed.worker:worker.py:1083 ------------------------------------------------- INFO distributed.worker:worker.py:1355 Start worker at: tcp://127.0.0.1:41159 INFO distributed.worker:worker.py:1356 Listening to: tcp://127.0.0.1:41159 INFO distributed.worker:worker.py:1359 Worker name: 9 INFO distributed.worker:worker.py:1361 dashboard at: 127.0.0.1:34107 INFO distributed.worker:worker.py:1362 Waiting to connect to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1363 ------------------------------------------------- INFO distributed.worker:worker.py:1364 Threads: 1 INFO distributed.worker:worker.py:1366 Memory: 250.90 GiB INFO distributed.worker:worker.py:1370 Local Directory: /tmp/dask-scratch-space/worker-uzbp6x0r INFO distributed.worker:worker.py:1083 ------------------------------------------------- INFO distributed.comm.tcp:tcp.py:677 Connection from tcp://127.0.0.1:45738 closed before handshake completed INFO distributed.comm.tcp:tcp.py:677 Connection from tcp://127.0.0.1:45746 closed before handshake completed INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:33753 name: 2 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:33753 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:45750 INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:35721 name: 3 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:35721 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:45764 INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:42799 name: 4 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:42799 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:45768 INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:46045 name: 5 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:46045 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:45774 INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:34311 name: 6 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:34311 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:45788 INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:46433 name: 7 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:46433 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:45800 INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:46525 name: 8 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:46525 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:45816 INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:41159 name: 9 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:41159 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:45822 INFO distributed.worker:worker.py:1776 Starting Worker plugin shuffle INFO distributed.worker:worker.py:1776 Starting Worker plugin shuffle INFO distributed.worker:worker.py:1776 Starting Worker plugin shuffle INFO distributed.worker:worker.py:1776 Starting Worker plugin shuffle INFO distributed.worker:worker.py:1776 Starting Worker plugin shuffle INFO distributed.worker:worker.py:1776 Starting Worker plugin shuffle INFO distributed.worker:worker.py:1776 Starting Worker plugin shuffle INFO distributed.worker:worker.py:1776 Starting Worker plugin shuffle INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:45173 name: 1 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:45173 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:45834 INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:45555 name: 0 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:45555 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:45838 INFO distributed.worker:worker.py:1150 Registered to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1151 ------------------------------------------------- INFO distributed.worker:worker.py:1150 Registered to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1151 ------------------------------------------------- INFO distributed.worker:worker.py:1150 Registered to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1151 ------------------------------------------------- INFO distributed.worker:worker.py:1150 Registered to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1151 ------------------------------------------------- INFO distributed.worker:worker.py:1150 Registered to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1151 ------------------------------------------------- INFO distributed.worker:worker.py:1150 Registered to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1151 ------------------------------------------------- INFO distributed.worker:worker.py:1150 Registered to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1151 ------------------------------------------------- INFO distributed.worker:worker.py:1150 Registered to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1151 ------------------------------------------------- INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:44073 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:44073 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:44073 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:44073 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:44073 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:44073 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:44073 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1776 Starting Worker plugin shuffle INFO distributed.worker:worker.py:1776 Starting Worker plugin shuffle INFO distributed.worker:worker.py:1150 Registered to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1151 ------------------------------------------------- INFO distributed.worker:worker.py:1150 Registered to: tcp://127.0.0.1:44073 INFO distributed.worker:worker.py:1151 ------------------------------------------------- INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:44073 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:44073 INFO distributed.scheduler:scheduler.py:5916 Receive client connection: Client-bd2a41e7-abeb-11f1-86b4-8f07734cb81d INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:45846 INFO distributed.scheduler:scheduler.py:5961 Remove client Client-bd2a41e7-abeb-11f1-86b4-8f07734cb81d INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:45846; closing. INFO distributed.scheduler:scheduler.py:5961 Remove client Client-bd2a41e7-abeb-11f1-86b4-8f07734cb81d INFO distributed.worker:worker.py:1459 Stopping worker at tcp://127.0.0.1:45555. Reason: worker-close INFO distributed.worker:worker.py:1459 Stopping worker at tcp://127.0.0.1:45173. Reason: worker-close INFO distributed.worker:worker.py:1459 Stopping worker at tcp://127.0.0.1:33753. Reason: worker-close INFO distributed.worker:worker.py:1459 Stopping worker at tcp://127.0.0.1:35721. Reason: worker-close INFO distributed.worker:worker.py:1459 Stopping worker at tcp://127.0.0.1:42799. Reason: worker-close INFO distributed.worker:worker.py:1459 Stopping worker at tcp://127.0.0.1:46045. Reason: worker-close INFO distributed.worker:worker.py:1459 Stopping worker at tcp://127.0.0.1:34311. Reason: worker-close INFO distributed.worker:worker.py:1459 Stopping worker at tcp://127.0.0.1:46433. Reason: worker-close INFO distributed.worker:worker.py:1459 Stopping worker at tcp://127.0.0.1:46525. Reason: worker-close INFO distributed.worker:worker.py:1459 Stopping worker at tcp://127.0.0.1:41159. Reason: worker-close INFO distributed.worker:worker.py:1792 Removing Worker plugin shuffle INFO distributed.worker:worker.py:1792 Removing Worker plugin shuffle INFO distributed.worker:worker.py:1792 Removing Worker plugin shuffle INFO distributed.worker:worker.py:1792 Removing Worker plugin shuffle INFO distributed.worker:worker.py:1792 Removing Worker plugin shuffle INFO distributed.worker:worker.py:1792 Removing Worker plugin shuffle INFO distributed.worker:worker.py:1792 Removing Worker plugin shuffle INFO distributed.worker:worker.py:1792 Removing Worker plugin shuffle INFO distributed.worker:worker.py:1792 Removing Worker plugin shuffle INFO distributed.worker:worker.py:1792 Removing Worker plugin shuffle INFO distributed.scheduler:scheduler.py:5953 Close client connection: Client-bd2a41e7-abeb-11f1-86b4-8f07734cb81d ERROR distributed.worker:worker.py:1206 Failed to communicate with scheduler during heartbeat. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 226, in read frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/worker.py", line 1165, in heartbeat response = await retry_operation( ^^^^^^^^^^^^^^^^^^^^^^ ...<14 lines>... ) ^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 420, in retry_operation return await retry( ^^^^^^^^^^^^ ...<5 lines>... ) ^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 399, in retry return await coro() ^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1212, in send_recv_from_rpc return await send_recv(comm=comm, op=key, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 982, in send_recv response = await comm.read(deserializers=deserializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 237, in read convert_stream_closed_error(self, e) ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 137, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in : Stream is closed ERROR distributed.worker:worker.py:1206 Failed to communicate with scheduler during heartbeat. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 226, in read frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/worker.py", line 1165, in heartbeat response = await retry_operation( ^^^^^^^^^^^^^^^^^^^^^^ ...<14 lines>... ) ^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 420, in retry_operation return await retry( ^^^^^^^^^^^^ ...<5 lines>... ) ^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 399, in retry return await coro() ^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1212, in send_recv_from_rpc return await send_recv(comm=comm, op=key, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 982, in send_recv response = await comm.read(deserializers=deserializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 237, in read convert_stream_closed_error(self, e) ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 137, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in : Stream is closed INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:45822; closing. INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:45774; closing. INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:45788; closing. INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:45800; closing. INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:45768; closing. INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:45838; closing. INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:45816; closing. INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:45764; closing. INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:45750; closing. INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:45834; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:41159 name: 9 (stimulus_id='handle-worker-cleanup-1788916459.4793055') INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:46045 name: 5 (stimulus_id='handle-worker-cleanup-1788916459.495362') INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:34311 name: 6 (stimulus_id='handle-worker-cleanup-1788916459.509631') INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:46433 name: 7 (stimulus_id='handle-worker-cleanup-1788916459.5235069') INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:42799 name: 4 (stimulus_id='handle-worker-cleanup-1788916459.5379431') INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:45555 name: 0 (stimulus_id='handle-worker-cleanup-1788916459.555504') INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:46525 name: 8 (stimulus_id='handle-worker-cleanup-1788916459.5700839') INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:35721 name: 3 (stimulus_id='handle-worker-cleanup-1788916459.5855424') INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:33753 name: 2 (stimulus_id='handle-worker-cleanup-1788916459.6012957') INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:45173 name: 1 (stimulus_id='handle-worker-cleanup-1788916459.6166422') INFO distributed.scheduler:scheduler.py:5729 Lost all workers INFO distributed.core:core.py:857 Connection to tcp://127.0.0.1:44073 has been closed. INFO distributed.core:core.py:857 Connection to tcp://127.0.0.1:44073 has been closed. INFO distributed.core:core.py:857 Connection to tcp://127.0.0.1:44073 has been closed. INFO distributed.core:core.py:857 Connection to tcp://127.0.0.1:44073 has been closed. INFO distributed.core:core.py:857 Connection to tcp://127.0.0.1:44073 has been closed. INFO distributed.core:core.py:857 Connection to tcp://127.0.0.1:44073 has been closed. INFO distributed.core:core.py:857 Connection to tcp://127.0.0.1:44073 has been closed. INFO distributed.core:core.py:857 Connection to tcp://127.0.0.1:44073 has been closed. INFO distributed.core:core.py:857 Connection to tcp://127.0.0.1:44073 has been closed. INFO distributed.core:core.py:857 Connection to tcp://127.0.0.1:44073 has been closed. INFO distributed.scheduler:scheduler.py:4493 Closing scheduler. Reason: unknown INFO distributed.scheduler:scheduler.py:4521 Scheduler closing all comms _________________________ test_steal_host_restrictions _________________________ c = s = wa = wb = @pytest.mark.skipif(not LINUX, reason="Need 127.0.0.2 to mean localhost") @gen_cluster(client=True, nthreads=[("127.0.0.1", 1), ("127.0.0.2", 2)]) async def test_steal_host_restrictions(c, s, wa, wb): futures = c.map(slowinc, range(100), delay=0.05, workers="127.0.0.1") while len(wa.state.tasks) + len(wb.state.tasks) < 100: await asyncio.sleep(0.01) assert len(wa.state.tasks) == 100 assert len(wb.state.tasks) == 0 async with Worker(s.address, nthreads=1) as wc: while s.workers[wc.address].status != Status.running: await asyncio.sleep(0.01) s.extensions["stealing"].balance() await asyncio.sleep(0.1) assert 20 < len(wa.state.tasks) < 95 assert len(wb.state.tasks) == 0 > assert 5 < len(wc.state.tasks) < 80 E AssertionError: assert 5 < 0 E + where 0 = len({}) E + where {} = .tasks E + where = .state distributed/tests/test_steal.py:511: AssertionError ----------------------------- Captured stderr call ----------------------------- 2026-09-09 01:14:38,005 - distributed.scheduler - INFO - State start 2026-09-09 01:14:38,065 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:36653 2026-09-09 01:14:38,076 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:40153/status 2026-09-09 01:14:38,088 - distributed.scheduler - INFO - Registering Worker plugin shuffle 2026-09-09 01:14:38,190 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:42441 2026-09-09 01:14:38,200 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:42441 2026-09-09 01:14:38,238 - distributed.worker - INFO - Worker name: 0 2026-09-09 01:14:38,250 - distributed.worker - INFO - dashboard at: 127.0.0.1:43073 2026-09-09 01:14:38,261 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:36653 2026-09-09 01:14:38,272 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:38,283 - distributed.worker - INFO - Threads: 1 2026-09-09 01:14:38,294 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 01:14:38,305 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-t74ym89v 2026-09-09 01:14:38,317 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:38,346 - distributed.worker - INFO - Start worker at: tcp://127.0.0.2:42067 2026-09-09 01:14:38,357 - distributed.worker - INFO - Listening to: tcp://127.0.0.2:42067 2026-09-09 01:14:38,368 - distributed.worker - INFO - Worker name: 1 2026-09-09 01:14:38,379 - distributed.worker - INFO - dashboard at: 127.0.0.2:38011 2026-09-09 01:14:38,388 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:36653 2026-09-09 01:14:38,398 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:38,407 - distributed.worker - INFO - Threads: 2 2026-09-09 01:14:38,416 - distributed.worker - INFO - Memory: 250.90 GiB 2026-09-09 01:14:38,425 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-675q99jf 2026-09-09 01:14:38,435 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:38,492 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:42441 name: 0 2026-09-09 01:14:38,506 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:42441 2026-09-09 01:14:38,516 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:46766 2026-09-09 01:14:38,522 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.2:42067 name: 1 2026-09-09 01:14:38,534 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.2:42067 2026-09-09 01:14:38,543 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:46778 2026-09-09 01:14:38,552 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 01:14:38,568 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 01:14:38,588 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:36653 2026-09-09 01:14:38,600 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:38,615 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:36653 2026-09-09 01:14:38,626 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:38,665 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:36653 2026-09-09 01:14:38,668 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:36653 2026-09-09 01:14:38,765 - distributed.scheduler - INFO - Receive client connection: Client-d3f24958-abeb-11f1-86b4-8f07734cb81d 2026-09-09 01:14:38,780 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:46792 2026-09-09 01:14:39,684 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:46283 2026-09-09 01:14:39,695 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:46283 2026-09-09 01:14:39,707 - distributed.worker - INFO - dashboard at: 127.0.0.1:35873 2026-09-09 01:14:39,717 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:36653 2026-09-09 01:14:39,728 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:39,738 - distributed.worker - INFO - Threads: 1 2026-09-09 01:14:39,749 - distributed.worker - INFO - Memory: 1.96 GiB 2026-09-09 01:14:39,760 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-0tl_5kyj 2026-09-09 01:14:39,772 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:40,001 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:46283 name: tcp://127.0.0.1:46283 2026-09-09 01:14:40,028 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:46283 2026-09-09 01:14:40,040 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:46804 2026-09-09 01:14:40,095 - distributed.worker - INFO - Starting Worker plugin shuffle 2026-09-09 01:14:40,136 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:36653 2026-09-09 01:14:40,148 - distributed.worker - INFO - ------------------------------------------------- 2026-09-09 01:14:40,222 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:36653 2026-09-09 01:14:40,409 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:46283. Reason: worker-close 2026-09-09 01:14:40,444 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 01:14:40,547 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:46804; closing. 2026-09-09 01:14:40,555 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:46283 name: tcp://127.0.0.1:46283 (stimulus_id='handle-worker-cleanup-1788916480.5543883') 2026-09-09 01:14:40,605 - distributed.core - INFO - Connection to tcp://127.0.0.1:36653 has been closed. 2026-09-09 01:14:40,831 - distributed.scheduler - INFO - Remove client Client-d3f24958-abeb-11f1-86b4-8f07734cb81d 2026-09-09 01:14:40,900 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:46792; closing. 2026-09-09 01:14:40,906 - distributed.scheduler - INFO - Remove client Client-d3f24958-abeb-11f1-86b4-8f07734cb81d 2026-09-09 01:14:40,948 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:42441. Reason: worker-close 2026-09-09 01:14:40,962 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.2:42067. Reason: worker-close 2026-09-09 01:14:40,980 - distributed.scheduler - INFO - Close client connection: Client-d3f24958-abeb-11f1-86b4-8f07734cb81d 2026-09-09 01:14:40,994 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 01:14:41,181 - distributed.worker.state_machine - WARNING - Async instruction for > ended with CancelledError 2026-09-09 01:14:41,271 - distributed.worker - INFO - Removing Worker plugin shuffle 2026-09-09 01:14:41,331 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 226, in read frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/worker.py", line 1165, in heartbeat response = await retry_operation( ^^^^^^^^^^^^^^^^^^^^^^ ...<14 lines>... ) ^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 420, in retry_operation return await retry( ^^^^^^^^^^^^ ...<5 lines>... ) ^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 399, in retry return await coro() ^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1212, in send_recv_from_rpc return await send_recv(comm=comm, op=key, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 982, in send_recv response = await comm.read(deserializers=deserializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 237, in read convert_stream_closed_error(self, e) ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 137, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in : Stream is closed 2026-09-09 01:14:41,395 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:46778; closing. 2026-09-09 01:14:41,410 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.2:42067 name: 1 (stimulus_id='handle-worker-cleanup-1788916481.4097772') 2026-09-09 01:14:41,435 - distributed.core - INFO - Connection to tcp://127.0.0.1:36653 has been closed. 2026-09-09 01:14:41,453 - distributed.core - INFO - Connection to tcp://127.0.0.1:36653 has been closed. 2026-09-09 01:14:41,461 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:46766; closing. 2026-09-09 01:14:41,465 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:42441 name: 0 (stimulus_id='handle-worker-cleanup-1788916481.4651337') 2026-09-09 01:14:41,478 - distributed.scheduler - INFO - Lost all workers 2026-09-09 01:14:41,506 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown 2026-09-09 01:14:41,522 - distributed.scheduler - INFO - Scheduler closing all comms ------------------------------ Captured log call ------------------------------- INFO distributed.scheduler:scheduler.py:1735 State start INFO distributed.scheduler:scheduler.py:4436 Scheduler at: tcp://127.0.0.1:36653 INFO distributed.scheduler:scheduler.py:4451 dashboard at: http://127.0.0.1:40153/status INFO distributed.scheduler:scheduler.py:8137 Registering Worker plugin shuffle INFO distributed.worker:worker.py:1355 Start worker at: tcp://127.0.0.1:42441 INFO distributed.worker:worker.py:1356 Listening to: tcp://127.0.0.1:42441 INFO distributed.worker:worker.py:1359 Worker name: 0 INFO distributed.worker:worker.py:1361 dashboard at: 127.0.0.1:43073 INFO distributed.worker:worker.py:1362 Waiting to connect to: tcp://127.0.0.1:36653 INFO distributed.worker:worker.py:1363 ------------------------------------------------- INFO distributed.worker:worker.py:1364 Threads: 1 INFO distributed.worker:worker.py:1366 Memory: 250.90 GiB INFO distributed.worker:worker.py:1370 Local Directory: /tmp/dask-scratch-space/worker-t74ym89v INFO distributed.worker:worker.py:1083 ------------------------------------------------- INFO distributed.worker:worker.py:1355 Start worker at: tcp://127.0.0.2:42067 INFO distributed.worker:worker.py:1356 Listening to: tcp://127.0.0.2:42067 INFO distributed.worker:worker.py:1359 Worker name: 1 INFO distributed.worker:worker.py:1361 dashboard at: 127.0.0.2:38011 INFO distributed.worker:worker.py:1362 Waiting to connect to: tcp://127.0.0.1:36653 INFO distributed.worker:worker.py:1363 ------------------------------------------------- INFO distributed.worker:worker.py:1364 Threads: 2 INFO distributed.worker:worker.py:1366 Memory: 250.90 GiB INFO distributed.worker:worker.py:1370 Local Directory: /tmp/dask-scratch-space/worker-675q99jf INFO distributed.worker:worker.py:1083 ------------------------------------------------- INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:42441 name: 0 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:42441 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:46766 INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.2:42067 name: 1 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.2:42067 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:46778 INFO distributed.worker:worker.py:1776 Starting Worker plugin shuffle INFO distributed.worker:worker.py:1776 Starting Worker plugin shuffle INFO distributed.worker:worker.py:1150 Registered to: tcp://127.0.0.1:36653 INFO distributed.worker:worker.py:1151 ------------------------------------------------- INFO distributed.worker:worker.py:1150 Registered to: tcp://127.0.0.1:36653 INFO distributed.worker:worker.py:1151 ------------------------------------------------- INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:36653 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:36653 INFO distributed.scheduler:scheduler.py:5916 Receive client connection: Client-d3f24958-abeb-11f1-86b4-8f07734cb81d INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:46792 INFO distributed.worker:worker.py:1355 Start worker at: tcp://127.0.0.1:46283 INFO distributed.worker:worker.py:1356 Listening to: tcp://127.0.0.1:46283 INFO distributed.worker:worker.py:1361 dashboard at: 127.0.0.1:35873 INFO distributed.worker:worker.py:1362 Waiting to connect to: tcp://127.0.0.1:36653 INFO distributed.worker:worker.py:1363 ------------------------------------------------- INFO distributed.worker:worker.py:1364 Threads: 1 INFO distributed.worker:worker.py:1366 Memory: 1.96 GiB INFO distributed.worker:worker.py:1370 Local Directory: /tmp/dask-scratch-space/worker-0tl_5kyj INFO distributed.worker:worker.py:1083 ------------------------------------------------- INFO distributed.scheduler:scheduler.py:4784 Register worker addr: tcp://127.0.0.1:46283 name: tcp://127.0.0.1:46283 INFO distributed.scheduler:scheduler.py:6181 Starting worker compute stream, tcp://127.0.0.1:46283 INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:46804 INFO distributed.worker:worker.py:1776 Starting Worker plugin shuffle INFO distributed.worker:worker.py:1150 Registered to: tcp://127.0.0.1:36653 INFO distributed.worker:worker.py:1151 ------------------------------------------------- INFO distributed.core:core.py:847 Starting established connection to tcp://127.0.0.1:36653 INFO distributed.worker:worker.py:1459 Stopping worker at tcp://127.0.0.1:46283. Reason: worker-close INFO distributed.worker:worker.py:1792 Removing Worker plugin shuffle INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:46804; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:46283 name: tcp://127.0.0.1:46283 (stimulus_id='handle-worker-cleanup-1788916480.5543883') INFO distributed.core:core.py:857 Connection to tcp://127.0.0.1:36653 has been closed. INFO distributed.scheduler:scheduler.py:5961 Remove client Client-d3f24958-abeb-11f1-86b4-8f07734cb81d INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:46792; closing. INFO distributed.scheduler:scheduler.py:5961 Remove client Client-d3f24958-abeb-11f1-86b4-8f07734cb81d INFO distributed.worker:worker.py:1459 Stopping worker at tcp://127.0.0.1:42441. Reason: worker-close INFO distributed.worker:worker.py:1459 Stopping worker at tcp://127.0.0.2:42067. Reason: worker-close INFO distributed.scheduler:scheduler.py:5953 Close client connection: Client-d3f24958-abeb-11f1-86b4-8f07734cb81d INFO distributed.worker:worker.py:1792 Removing Worker plugin shuffle WARNING distributed.worker.state_machine:worker_state_machine.py:3632 Async instruction for > ended with CancelledError INFO distributed.worker:worker.py:1792 Removing Worker plugin shuffle ERROR distributed.worker:worker.py:1206 Failed to communicate with scheduler during heartbeat. Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 226, in read frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/worker.py", line 1165, in heartbeat response = await retry_operation( ^^^^^^^^^^^^^^^^^^^^^^ ...<14 lines>... ) ^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 420, in retry_operation return await retry( ^^^^^^^^^^^^ ...<5 lines>... ) ^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 399, in retry return await coro() ^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1212, in send_recv_from_rpc return await send_recv(comm=comm, op=key, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 982, in send_recv response = await comm.read(deserializers=deserializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 237, in read convert_stream_closed_error(self, e) ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 137, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in : Stream is closed INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:46778; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.2:42067 name: 1 (stimulus_id='handle-worker-cleanup-1788916481.4097772') INFO distributed.core:core.py:857 Connection to tcp://127.0.0.1:36653 has been closed. INFO distributed.core:core.py:857 Connection to tcp://127.0.0.1:36653 has been closed. INFO distributed.core:core.py:872 Received 'close-stream' from tcp://127.0.0.1:46766; closing. INFO distributed.scheduler:scheduler.py:5601 Remove worker addr: tcp://127.0.0.1:42441 name: 0 (stimulus_id='handle-worker-cleanup-1788916481.4651337') INFO distributed.scheduler:scheduler.py:5729 Lost all workers INFO distributed.scheduler:scheduler.py:4493 Closing scheduler. Reason: unknown INFO distributed.scheduler:scheduler.py:4521 Scheduler closing all comms =========================== short test summary info ============================ FAILED distributed/cli/tests/test_dask_scheduler.py::test_defaults - OSError: Timed out trying to connect to tcp://[fe80::9e69:b4ff:fe64:abd4]:8786 after 5 s FAILED distributed/comm/tests/test_comms.py::test_default_client_server_ipv6[tornado] - OSError: Timed out trying to connect to tcp://[fe80::9e69:b4ff:fe64:abd4]:44377 after 30 s FAILED distributed/comm/tests/test_comms.py::test_tcp_client_server_ipv6[tornado] - OSError: Timed out trying to connect to tcp://[fe80::9e69:b4ff:fe64:abd4]:33375 after 30 s FAILED distributed/deploy/tests/test_local.py::test_silent_startup - assert not '2026-09-08 22:08:14,793 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat.\nTraceback (most recent call last):\n File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 226, in read\n frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\ntornado.iostream.StreamClosedError: Stream is closed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/build/python-distributed/src/distributed-2026.8.0/distributed/worker.py", line 1165, in heartbeat\n response = await retry_operation(\n ^^^^^^^^^^^^^^^^^^^^^^\n ...<14 lines>...\n )\n ^\n File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 420, in retry_operation\n return await retry(\n ^^^^^^^^^^^^\n ...<5 lines>...\n )\n ^\n File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 399, in retry\n return await coro()\n ^^^^^^^^^^^^\n File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1212, in send_recv_from_rpc\n return await send_recv(comm=comm, op=key, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 982, in send_recv\n response = await comm.read(deserializers=deserializers)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 237, in read\n convert_stream_closed_error(self, e)\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^\n File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/tcp.py", line 137, in convert_stream_closed_error\n raise CommClosedError(f"in {obj}: {exc}") from exc\ndistributed.comm.core.CommClosedError: in : Stream is closed\n' FAILED distributed/deploy/tests/test_local.py::test_remote_access - OSError: Timed out trying to connect to tcp://[fe80::9e69:b4ff:fe64:abd4]:43077 after 0.5 s FAILED distributed/tests/test_client.py::test_quiet_client_close - AssertionError: ['Async instruction for > ended with CancelledError', 'Async instruction for > ended with CancelledError', 'Failed to communicate with scheduler during heartbeat.', 'Traceback (most recent call last):', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1422, in connect', ' return await self._connect(addr=addr, timeout=timeout)', ...] assert not ['Failed to communicate with scheduler during heartbeat.', 'Traceback (most recent call last):', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1422, in connect', ' return await self._connect(addr=addr, timeout=timeout)', ' ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1366, in _connect', ' comm = await connect(', ' ^^^^^^^^^^^^^^', ' ...<4 lines>...', ' )', ' ^', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/core.py", line 377, in connect', ' handshake = await comm.read()', ' ^^^^^^^^^^^^^^^^^', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/comm/inproc.py", line 219, in read', ' msg = await self._read_q.get()', ' ^^^^^^^^^^^^^^^^^^^^^^^^', 'asyncio.exceptions.CancelledError', 'The above exception was the direct cause of the following exception:', 'Traceback (most recent call last):', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1413, in connect', ' async with asyncio.timeout(math.inf) as scope:', ' ~~~~~~~~~~~~~~~^^^^^^^^^^', ' File "/usr/lib/python3.14/asyncio/timeouts.py", line 115, in __aexit__', ' raise TimeoutError from exc_val', 'TimeoutError', 'During handling of the above exception, another exception occurred:', 'Traceback (most recent call last):', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/worker.py", line 1165, in heartbeat', ' response = await retry_operation(', ' ^^^^^^^^^^^^^^^^^^^^^^', ' ...<14 lines>...', ' )', ' ^', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 420, in retry_operation', ' return await retry(', ' ^^^^^^^^^^^^', ' ...<5 lines>...', ' )', ' ^', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/utils_comm.py", line 399, in retry', ' return await coro()', ' ^^^^^^^^^^^^', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1209, in send_recv_from_rpc', ' comm = await self.pool.connect(self.addr)', ' ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^', ' File "/build/python-distributed/src/distributed-2026.8.0/distributed/core.py", line 1434, in connect', ' raise CommClosedError(reason)', 'distributed.comm.core.CommClosedError: ConnectionPool closing.'] FAILED distributed/tests/test_failed_workers.py::test_restart - TimeoutError FAILED distributed/tests/test_failed_workers.py::test_restart_cleared - TimeoutError FAILED distributed/tests/test_failed_workers.py::test_multiple_clients_restart - TimeoutError: Test timeout (60) hit after 60.020716443999845s. ========== Test stack trace starts here ========== Stack for wait_for= cb=[shield.._clear_awaited_by_callback() at /usr/lib/python3.14/asyncio/tasks.py:978, _log_on_exception() at /usr/lib/python3.14/asyncio/tasks.py:914]> (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/tests/test_failed_workers.py", line 230, in test_multiple_clients_restart await c1.restart() FAILED distributed/tests/test_failed_workers.py::test_restart_scheduler - TimeoutError: Test timeout (60) hit after 60.006166047998704s. ========== Test stack trace starts here ========== Stack for wait_for= cb=[shield.._clear_awaited_by_callback() at /usr/lib/python3.14/asyncio/tasks.py:978, _log_on_exception() at /usr/lib/python3.14/asyncio/tasks.py:914]> (most recent call last): File "/build/python-distributed/src/distributed-2026.8.0/distributed/tests/test_failed_workers.py", line 263, in test_restart_scheduler await s.restart(stimulus_id="test") FAILED distributed/tests/test_failed_workers.py::test_forgotten_futures_dont_clean_up_new_futures - TimeoutError FAILED distributed/tests/test_steal.py::test_dont_steal_fast_tasks_compute_time - AssertionError: assert 10 == 1 + where 10 = len({, , , , , , , , , }) + where {, , , , , , , , , } = (*. at 0x3e8e11e420>) + where = set.union FAILED distributed/tests/test_steal.py::test_steal_host_restrictions - AssertionError: assert 5 < 0 + where 0 = len({}) + where {} = .tasks + where = .state = 13 failed, 3313 passed, 78 skipped, 516 deselected, 15 xfailed, 4 xpassed in 14929.82s (4:08:49) = ==> ERROR:(B A failure occurred in check().(B  Aborting...(B ]3008;end=e71d3dabc47a43cfa0d58f062d2a3c48\[!p]104\[?7h]3008;end=75832fa3e18a4dadbb386d1ec0d91f1a\==> ERROR:(B Build failed, check /var/lib/archbuild/extra-riscv64/felix-3/build(B [?12l[?25h[?12l[?25h]3008;end=5848493057f7442fafba90741c8b156a\[?12l[?25hreceiving incremental file list python-distributed-2026.8.0-1-riscv64-build.log python-distributed-2026.8.0-1-riscv64-check.log python-distributed-2026.8.0-1-riscv64-prepare.log sent 81 bytes received 63,255 bytes 25,334.40 bytes/sec total size is 652,605 speedup is 10.30